Large language models can no longer just write lines and code, but can autonomously plan and execute complex cyber operations. A recent study by researchers at Carnegie Mellon University and Anthropic demonstrates that AI can autonomously replicate advanced hacking attacks, posing entirely new challenges for the cyber security industry.
The concept of an autonomous digital adversary, powered by artificial intelligence, has long belonged to the realm of science fiction. A joint research project between a leading US technical university and Anthropic, a well-known player in the AI market, is bringing this vision to reality. Their work shows that large-scale language models (LLMs) have achieved the ability to act autonomously as cybercriminals, which could fundamentally change the threat landscape.
The aim of the experiment was not to create a new weapon, but to accurately measure the ability of current AI models to autonomously plan and enforce attacks without human oversight. To make the test as meaningful as possible, the researchers decided to recreate one of the most high-profile security incidents in US history – the 2017 attack on Equifax.
The Equifax case: digital reconstruction
The choice of attack on Equifax was no accident. Given its scale – leaking the data of almost 147 million customers – and the public availability of detailed technical analysis, it provided an ideal testing ground. The 2017 attack exploited a known vulnerability in Apache Struts, a component used in many web applications. The hackers, after gaining initial access, penetrated the company’s internal network for weeks, installing malware and ultimately exfiltrating sensitive data.
Researchers at Carnegie Mellon and Anthropic developed a special toolkit, called Incalmo. Its task was to translate the overall attack strategy, generated by a large language model, into specific, low-level system commands. In this model, the LLM acted as the ‘brain of the operation’, deciding on the next steps, while specialised agents – both AI-based and traditional scripts – carried out the tasks assigned to them, such as scanning the network for vulnerabilities or deploying prepared exploits.
Worrying effectiveness
The test results proved alarming. The research team prepared 10 isolated network environments simulating the IT infrastructure of small businesses. Under these controlled conditions, the autonomous LLM-based system was able to:
- Fully compromise 5 out of 10 networks, which meant gaining full access and meeting the targets of the attack.
- Partially compromise 4 more networks, for example by stealing some sensitive information.
Importantly, the researchers also tested AI’s ability to adapt to other scenarios. One of the models tested was a ransomware attack modelled on the 2021 Colonial Pipeline incident, which paralysed fuel supplies on the US East Coast for nearly a week. The ability to replicate attacks so different in nature demonstrates the flexibility and adaptive potential of LLM models.
Machine-scale attack
The biggest concern that emerges from the survey is not the possibility of an attack itself, but its potential scale, speed and low cost. To date, organising a sophisticated hacking campaign has required a team of experts, time and significant financial resources. The democratisation of access to powerful language models could dramatically lower the threshold for entry.
Brian Singer, a PhD student at Carnegie Mellon and lead author of the study, stresses that the biggest concern is the speed at which such attacks can be orchestrated. Current defence systems rely heavily on human analysts and security centre operators (SOCs). Their job is to analyse alerts, identify anomalies and manually respond to incidents. This human decision-making loop may not be sufficient in a clash with an adversary operating at speed and machine scale.
The arms race
Research such as that conducted by CMU and Anthropic is not intended to sow panic, but rather to initiate urgent discussion and action in the industry. Understanding the offensive capabilities of AI is the first step to building an effective defence.
The next natural stage of research work is to create ‘autonomous defenders’ – LLM-based security systems that can identify and neutralise attacks carried out by other machines in real time.
