Darktrace published research on September 24 into a weakness in programming tools with artificial intelligence agents: locally stored conversation history can be modified and then accepted as legitimate context. In controlled tests, the researchers used this history “poisoning” to convince agents that they were already participating in authorized offensive security exercises and get them to perform actions they had initially refused.

The attack changes the past the agent sees

The technique does not alter the model's weights or need to directly defeat its system message. The attacker modifies the records that the agentic harness incorporates into the next request. According to Darktrace, a fabricated sequence of responses, tool calls, and results can present a false history of authorization and compliance. The model makes decisions using that context and may interpret a dangerous request as the continuation of a legitimate task.

The team says it confirmed the problem in Anthropic's Claude Code, OpenAI's Codex, AWS's Kiro-CLI, and the open-source Pi project. Darktrace reported its findings to Anthropic, AWS, and OpenAI on August 18 and published the research after a disclosure period of more than 30 days. The company does not attribute the behavior to a vulnerability unique to one model; it describes it as a common design decision in harnesses that trust client-side histories without verifying their integrity.

In the experiments, the researchers inserted lengthy conversations showing the agent accepting supposed red-team exercises. Darktrace reports that, inside isolated labs, some agents went from refusing offensive requests to enumerating machines, scanning networks, moving laterally, escalating privileges, and demonstrating impact. In one comparison, 78 fabricated turns were enough to change the response to the same request. Resistance varied across models: the article notes that some configurations activated safeguards and refused to continue.

A vector that still requires prior access

The scenario does not mean that a remote conversation can rewrite local history on its own. Darktrace's example starts with malicious code—for example, a package or MCP server installed by the developer—gaining the ability to modify the harness's database. That requirement implies prior access to the environment, but it also shows how a limited intrusion could exploit the broad permissions often granted to an agent with a terminal and tools.

The idea has precedents. 0DIN previously documented that manipulating an agent's memory or history could change the framework through which it interpreted a task. Darktrace's work extends that line of research with tests across several harnesses and a complete chain within controlled environments. Even so, the results come from a laboratory and do not demonstrate an active campaign or that all models respond in the same way.

Darktrace proposes that providers cryptographically sign messages generated by the model and validate those signatures in every exchange. It also recommends integrity checks on local files, limiting privileges, reviewing dependencies and MCP servers, and detecting deviations in agent behavior. Signing would be a provider-side mitigation; teams deploying agents still need to treat history, tools, and local results as potentially hostile inputs.

The finding shifts the security question from “what the model allows” to “what context the system accepts as true.” An agent can maintain reasonable safeguards and still make a dangerous decision if its operational memory has been tampered with. That is why protecting the chain that constructs the context is as important as evaluating the model that responds.

Sources: Darktrace and 0DIN.

— Lía Torres, AI News