Rogue AI Hugging Face cyber incident heralds new ‘era of agentic autonomous attacks’

Cyber insurers were hit this week by news that an advanced OpenAI-built agent went rogue by escaping test limits to carry out an autonomous, “unprecedented” cyber attack by itself. 

According to the ChatGPT creator, the advanced OpenAI agent – an artificial intelligence (AI) system that can operate alone after human instruction – was able to break out of its testing environment by identifying weaknesses in security and then creating its own cyber attack, which gained access to a number of internal company systems. 

In the initial disclosure of the attack, model testing platform Hugging Face said it was assessing whether any customer or partner data was affected and had closed the vulnerabilities highlighted by the incident. 

What may worry cyber insurers, however, is the firm’s statement on what the incident means for wider cyber security. 

It said: “Autonomous, AI-driven offensive tooling is no longer theoretical. Defending an online platform now means treating the data and model surface as a first-class attack surface and using AI on defence to keep pace.” 

Cyber insurers and brokers distributing this product will now need to highlight the risk of what non-human, AI threat actors can do to their businesses if they do not have the proper protections in place. 

Isabel Simpson, partner at Clyde and Co, explained: “Many organisations are focused on what AI can do for their business, but the more pressing question is whether their governance frameworks and keeping pace with what AI can do on its own.

“As AI systems become more autonomous, organisations will need to treat AI governance, cyber resilience and accountability as core business risks, rather than standalone compliance exercises.” 

Insurer Acrisure added: ”As businesses become increasingly dependent on AI, and AI systems continue to grow in capability and autonomy, the associated exposures are becoming more complex and potentially more severe.

“Organisations are no longer only concerned with threat actors using AI to make attacks more effective – they must also consider the risks arising from the deployment of AI within their own operations, products and supply chains.”

Without supervision

While the model that carried out the autonomous attack on Hugging Face did have its guardrails deliberately lowered for testing, the capability to conduct an attack without human instruction should be a red flag for businesses and cyber insurance providers. 

Richard Ford, vice president of engineering at cyber risk modeller CyberCube, explained: ”The notable evolution here is that this resulted from an otherwise benign AI task fully autonomously and essentially unprompted. 

“We are entering the era of agentic autonomous attacks and it’s very unlikely that will play out well for us.” 

Ford’s colleague William Altman, director of cyber threat intelligence services at CyberCube, added: ”Ransomware used to require a team and the cost of paying that team limited how many attacks were worth running and who was worth attacking. Agentic ransomware could change that. 

”When the cost of running a full attack chain approaches zero, criminals don’t need to be selective. Small and medium-sized businesses that were safe because they weren’t worth a dedicated team’s time become viable targets at scale.

“For insurers, this is an early signal to weigh in pricing – not yet a trend of losses, but models built on the idea that attackers had to select their targets need re-evaluating.”