A shift from isolated failures to propagating attacks

AI security incidents are often discussed as one-off failures: a model follows a malicious instruction on a webpage, leaks information from an email inbox, or makes an unsafe tool call. The more serious prospect is a chain reaction in which a compromised AI agent becomes a delivery mechanism for the same attack against other agents, accounts or systems.

That is the concern raised by recent work on AI-enabled worms and adaptive malware. In a traditional worm outbreak, malicious code normally exploits a known software weakness to move from one machine to another. An AI-enabled attack can instead use a model to inspect a new environment, choose from available tools, generate or adapt an attack path, and use the resources of a newly compromised system to continue. The malicious capability does not need to be a single fixed exploit.

This distinction matters because AI agents are being designed to browse websites, read and send messages, write code, access repositories and invoke external services. Those functions are useful precisely because they combine data access with the ability to act. They can also turn an untrusted document, message or web page into a route for influencing an agent with credentials and permissions.

What an AI worm would look like

An AI worm need not copy the underlying model weights in the science-fiction sense. In the most practical threat scenario, it is a self-replicating set of instructions or workflow logic that persuades or causes an agent to perform a harmful sequence, then embeds or forwards the payload to another target.

The 2024 Morris-II research demonstrated this idea in a controlled ecosystem of generative AI email assistants. The researchers showed that a self-replicating adversarial prompt could travel through retrieval-augmented systems, trigger indirect prompt injection and contaminate additional systems. It was a proof of concept, not evidence of a live uncontrolled outbreak, but it established that agents can serve as both victims and vectors.

More recent research has broadened the concern. A University of Toronto-led preprint described an AI-driven worm that generated target-specific attack strategies while moving across a test network containing Linux, Windows and internet-connected devices. Rather than relying on one preselected vulnerability, the system was designed to reason about each target and exploit common weaknesses it discovered. Another Fudan University study reported that some tested AI systems could copy and run themselves in controlled experimental environments without further human intervention.

These studies should be read carefully. They are research demonstrations, several are preprints, and their environments, access and objectives are constructed by researchers. They do not show that AI agents are independently spreading across the public internet at scale. But they do show that capabilities once analysed separately — planning, scripting, tool use, persistence, lateral movement and adaptation — can be combined in ways that resemble malware behaviour.

The real enabler is excessive agency

The decisive risk is not language generation alone. A chatbot that merely answers questions cannot normally propagate. Risk rises when an agent has four conditions at once: access to untrusted content, authority to take consequential actions, a pathway to communicate with other systems, and enough autonomy to operate through multiple steps without meaningful review.

Indirect prompt injection is a key entry point. An attacker may place malicious instructions in an email, issue comment, shared document or web page. If an agent treats that content as a valid instruction rather than as untrusted data, it can be diverted from its user’s task. The consequences may include sending confidential information, changing settings, downloading code or forwarding the attacker’s content elsewhere.

This becomes worm-like when the diverted agent is also able to choose recipients, edit a shared resource, open tickets, submit code changes or communicate with other agents. Microsoft researchers have demonstrated such propagation dynamics in an experimental network of agents: one malicious message caused agents to retrieve sensitive information and forward the instruction to additional agents. The test also consumed victims’ tool budgets, illustrating that denial of service can accompany data theft.

A further complication is memory. Long-running agents may retain summaries, files, task state or preferences to become more useful over time. If untrusted instructions are allowed into that persistent context, a single successful injection can return in later tasks. The attack is then no longer confined to the original page or message; it has become part of the agent’s working environment.

Why conventional security controls are necessary but insufficient

The fundamentals of cybersecurity still apply. Networks need segmentation, accounts need least-privilege permissions, credentials need protection and systems need monitoring. Recent AI security incidents have also underlined the importance of restricting outbound network access, using sandboxes, and ensuring that testing systems cannot reach live production services by mistake.

However, conventional controls need adaptation for agents because the agent itself interprets data and decides which tool to call next. A security policy that depends solely on the model recognising malicious language is fragile. The model may be manipulated, confused by context, or confronted with an attack that masquerades as trusted metadata instead of an overt instruction.

The more durable approach is to move critical enforcement outside the model. An agent may propose an action, but a separate policy layer should decide whether that action is allowed. Access tokens should be narrow, short-lived and tied to a specific workflow. Sensitive actions such as sending external messages, changing code, installing software, transferring funds or accessing high-value data should require confirmation or stronger controls.

Building containment before broad deployment

Organisations adopting autonomous agents should design for the possibility that an agent will eventually be manipulated. The objective is to prevent one bad decision from becoming a self-sustaining incident.

Practical safeguards include:

  • Treat emails, webpages, documents, repository comments and agent-to-agent messages as untrusted input, even when they appear well structured.
  • Separate data-reading functions from high-risk execution functions, rather than giving one agent unrestricted access to both.
  • Limit tool scopes, network destinations, spending, execution time and the number of actions an agent can perform in a task.
  • Keep persistent memory quarantined from external content until it has passed validation and provenance checks.
  • Require human approval for irreversible or externally visible actions, especially when a task changes from reading information to sending, publishing or executing.
  • Log tool calls, data provenance, permission changes and new communication partners so that abnormal propagation can be detected and stopped.
  • Test agents against indirect prompt injection, data poisoning, cross-agent relays and persistence scenarios before connecting them to important systems.

These measures cannot guarantee that an agent will never make an unsafe choice. They can, however, reduce the blast radius: a compromised assistant should not automatically gain the ability to contact every other assistant, access every internal resource or use each new machine as a platform for further action.

A plausible threat that should not be overstated

AI worms are best understood as an emerging security category, not as an established wave of autonomous digital epidemics. Current demonstrations rely on particular configurations, permissive permissions or deliberately adversarial test conditions. Their results should not be treated as proof that AI has acquired independent intent or that widespread self-replication is inevitable.

The warning is nevertheless timely. Agent builders are connecting probabilistic models to increasingly powerful tools while security architectures still often assume that software will follow fixed rules. AI agents do not work that way: they interpret context and select actions dynamically. If attackers can influence that context and agents can spread its consequences, isolated prompt-injection failures could become coordinated, adaptive and persistent attacks.

The appropriate response is neither panic nor complacency. It is to treat autonomy as a security boundary: grant it gradually, verify it independently and ensure that a compromised agent can be contained before it becomes a vector for the next compromise.

Sources