In 2023, a quiet drama unfolded in the corridors of tech giant Samsung. It was not a hacking attack from outside, but an internal failure with huge consequences.
Engineers, in pursuit of efficiency, pasted bits of confidential source code directly into the publicly available ChatGPT model. Unknowingly, they handed over the company’s crown jewels to an external party, creating a textbook example of a new kind of insider threat – unintentional but equally destructive.
This incident exposes a fundamental truth of the artificial intelligence era.
While boards and IT departments focus on building technological fortresses, the real AI security battlefield has moved inside organisations.
The biggest risks no longer lie in sophisticated code, but in three key human-machine interactions: the data we feed it, the way we use it and the blind trust we place in it.
Every AI model is a mirror image of the data on which it has been trained. The ‘rubbish in, rubbish out’ principle gains a powerful new force in this context. Algorithms are not objective judges; they are historians who uncritically replicate patterns from the past.
Trained on data reflecting decades of social prejudices, they become a tool for perpetuating them on a massive scale.
The high-profile Apple Card case, investigated in 2019 for gender discrimination in the granting of credit limits, illustrates this perfectly.
The problem lay not in the malicious intention of the programmers, but in the historical financial data that the system took to be the objective truth, leading to legal risk and reputational damage. However, the input data problem also has a second, much darker face – deliberate sabotage.
Through data poisoning, an insider or external actor can discreetly introduce manipulated information into the training set.
Imagine a demand forecasting system in a retail chain whose data has been poisoned by a competitor. The model, damaged from the inside, starts systematically ordering the wrong goods to the wrong locations, paralysing the company’s logistics for months.
Even if the input is clean, companies face another challenge: the employee as unwitting saboteur. Organisations deploy powerful AI tools, often without providing employees with a clear security ‘instruction manual’.
Accustomed to the convenience of search engines, employees treat public language models as omniscient, unobtrusive assistants. The ‘I’ll just ask the AI’ syndrome leads them to paste snippets of code, marketing strategies, financial analysis or customer data into them to improve, summarise or analyse text.
The pursuit of productivity without understanding the technology leads to an immediate and irreversible loss of intellectual property.
Moreover, even internal, supposedly secure models can be fooled. Techniques such as prompt injection, a form of next-generation social engineering, make it possible, through cleverly worded commands, to force a model to ignore its security instructions and reveal sensitive information it has ‘memorised’ from other conversations.
The most insidious risk, however, comes at the end of this chain – the output. It involves abdicating human critical thinking and treating AI results as an infallible oracle.
Models can generate completely false information, known as ‘hallucinations’, with remarkable confidence and in a very convincing style.
Humans have a natural tendency to trust results presented in an authoritative manner, especially if they come from a system perceived as ‘intelligent’. The famous failure of IBM’ s Watson for Oncology system is a tragic example of this.
The system that was supposed to revolutionise cancer treatment recommended unsafe therapies because it was trained on limited, hypothetical data.
In automated systems, such as those managing a supply chain, this dynamic leads to a cascade of errors. A single erroneous output from a model can have a knock-on effect, leading to a series of catastrophic decisions before a human can intervene, as demonstrated by the $500 million loss suffered by Zillow, whose algorithm got caught in a loop of erroneously overestimating property values.
Artificial intelligence security is therefore not just a technical problem for engineers to solve. It is a fundamental organisational and cultural challenge.
Effective defence requires a strategy that puts humans at the centre. It is essential to implement mandatory, cyclical training that builds a culture of ‘healthy scepticism’ towards AI.
There needs to be ironclad rules about what information can be fed into external models – the default rule should be: “if you can’t publish it on the company’s homepage, you can’t paste it into AI”.
Above all, however, in critical business processes, systems must be designed so that the final decision is always up to the human and the interface clearly communicates the level of uncertainty in the model.
In the coming decade, the market leaders will not be those companies that implement AI fastest, but those that integrate it most cleverly with their most important resource: critically thinking people. The ultimate firewall must become the informed employee.
