As innovation in generative artificial intelligence (AI) continues at a dizzying pace, concerns about safety and risk are becoming more apparent. Some lawmakers have demanded new rules and regulations for artificial intelligence tools, while some technology and business leaders have suggested a pause in the training of AI systems to assess their security.
In an interview, Avivah Litan , vice president of analytics at Gartner, discussed what data and analytics leaders responsible for AI development need to know about managing trust, risk and security in AI.
Given concerns about the security and risk of AI, should organisations continue to explore the use of generative AI, or is a pause warranted?
Avivah Litan: The reality is that the development of generative artificial intelligence is not stopping. Organisations must act now to formulate a corporate-wide AI trust, risk and security management (AI TRiSM) strategy. There is an urgent need for a new class of AI TRiSM tools to manage data and process flows between users and companies that support generative AI models.
Currently, there are no off-the-shelf tools on the market that provide users with a systematic privacy guarantee or effective filtering of the content of their interactions with these models, for example filtering out factual errors, hallucinations, copyrighted material or confidential information.
AI developers urgently need to work with policymakers, including any new regulators that may emerge, to establish principles and practices for generative AI oversight and risk management.
What are some of the most significant risks that generative AI currently poses to businesses?
Avivah Litan: Generative artificial intelligence poses a number of new risks:
“Hallucinations” and fabrications , including factual errors, are some of the most prevalent problems that already arise with generative AI-based chatbot solutions. Training data can lead to biased, misguided or erroneous answers, but these can be difficult to detect, especially as solutions are increasingly reliable and can be relied upon.
Deepfake , when generative AI is used to create content with malicious intent, represents a significant risk of generative AI. These fake images, videos and voice recordings have been used to attack celebrities and politicians, create and disseminate misleading information, and even create fake accounts or take over and hack existing legitimate accounts.
In a recent example, an AI-generated image of Pope Francis in a fashionable white down jacket went viral on social media. While this example seemed seemingly harmless, it provided a glimpse into a future where deepfakes pose serious reputational, fake, fraudulent and political threats to individuals, organisations and governments.
Data privacy : employees can easily disclose confidential and proprietary company data when interacting with generative AI-based chatbot solutions. These applications can indefinitely store information captured through user input and even use the information to train other models – further compromising confidentiality. Such information can also fall into the wrong hands in the event of a security breach.
Copyright issues : generative AI chatbots are trained on a large amount of web data, which may contain copyrighted material. As a result, some outputs may violate copyright or intellectual property (IP) protection. Without references to sources or transparency in generative output, the only way to mitigate this risk is for users to check the output to ensure that it does not infringe copyright or IP rights.
Cybersecurity concerns : in addition to more advanced social engineering and phishing threats, attackers can use these tools to more easily generate malicious code. Vendors offering generative AI models assure customers that they train their models to reject malicious cyber-security requests; however, they do not provide users with the tools to effectively audit all the protections in place.
Vendors also place great emphasis on a ‘red team’ approach. These claims require users to place full confidence in the vendors’ ability to meet security objectives.
What actions can business leaders take now to manage the generative risks of AI?
Avivah Litan: It should be noted that there are two general approaches to using ChatGPT and similar applications. Using an out-of-the-box model uses these services as is, without direct customisation. The rapid engineering approach uses tools to create, tune and evaluate rapid input and output.
When used right out of the box, organisations must implement manual reviews of all model output to detect incorrect, erroneous or biased results. Establish a governance and compliance framework for the use of these solutions in the enterprise, including clear policies that prohibit employees from asking questions that reveal sensitive organisational or personal data.
Organisations should monitor unsanctioned use of ChatGPT and similar solutions through existing security controls and dashboards to detect policy violations. For example, firewalls can block access by corporate users, security information and event management systems can monitor event logs for violations, and secure network gateways can monitor for unauthorised API calls.
All of these risk mitigation measures apply to rapid engineering use. In addition, steps should be taken to protect internal data and other sensitive data used to construct prompts on third-party infrastructure. Create and store the developed prompts as an immutable resource.
These resources can represent proven engineering monites that are safe to use. They can also represent a corpus of finely tuned and highly developed prompts that can be more easily reused, shared or sold.
