Learn / Detecting and responding to AI abuse
Lesson 4 of 610 minReviewed
Responding to an agent incident
Run an incident where an agent or AI coding tool acted on injected instructions: contain the identity, preserve the agent's record, scope through audit logs and remove the trigger.
When an agent follows injected instructions, it acts with real credentials: a repository token, a cloud role, a mailbox grant. The incident is therefore about what that identity did, not about the model. The phases are the familiar ones; NIST SP 800-61 Rev. 3 (April 2025) places incident response inside the CSF 2.0 functions, from preparation through detection, response and recovery [1]. What changes is where the evidence lives and what has to be removed afterwards.
Three incidents worth studying
| Incident | What happened | Response lesson |
|---|---|---|
| GitHub MCP toxic agent flow (May 2025) | A malicious issue in a public repository steered an agent holding a broad GitHub token into copying private repository data into a public pull request. Invariant Labs describes it as an architectural problem, not a bug in the MCP server [2]. | Scope by what the token could reach. A server patch alone does not close it; least-privilege tokens and one repository per session do. |
| s1ngularity, Nx (August 2025) | Malicious nx versions ran a postinstall script that drove installed AI CLIs with approval-bypass flags to inventory secrets, pushed results to public repositories named s1ngularity-repository and added a shutdown line to shell startup files [3, 5]. | The advisory's checklist: GitHub audit log for repository creation, /tmp/inventory.txt, .bashrc and .zshrc, rotate GitHub and npm tokens, purge package caches [3]. |
| Amazon Q Developer for VS Code 1.84.0 (July 2025) | An overly broad GitHub token in the extension's build configuration let an attacker commit malicious code that shipped in a release. AWS says it did not run because of a syntax error [4]. | The AI tool is part of your supply chain: know which versions are installed and remove the affected one everywhere, forks included. |
Runbook
- Contain the identity. Revoke or rotate every credential the agent held (API keys, OAuth grants, cloud roles, MCP server tokens) and disable its high-risk tools or the MCP server involved. Resetting the conversation changes nothing while those tokens stay valid.
- Preserve the agent's record before cleanup. Collect transcripts, tool-call logs, MCP configurations and server versions. Local agents keep their record on the endpoint: Claude Code, for example, stores session transcripts in plaintext under
~/.claude/projects/for 30 days by default [7]. Collect them before they age out, and handle them as sensitive, since they may contain the very secrets that leaked. - Scope through the credentials. The tool-call log shows what the agent tried; cloud, SaaS and identity audit logs for the agent's identity over the exposure window show what actually happened.
- Remove the trigger. Delete or quarantine the injected issue, document or web page, the poisoned memory entry, the modified configuration file or the rogue server. OWASP lists memory and context poisoning as ASI06 because a persisted entry re-hijacks the agent on its next run [6].
- Sweep the fleet when a shared component is involved. For a vulnerable or malicious local MCP server, the MCP project's guidance is to find every instance from configuration inventory, block its egress, rotate every credential passed to it and push a fixed, pinned version through the approved channel [8].
- Recover with a test. Re-run the injection against the fixed agent before restoring its permissions, and add the logging or approval gate that would have caught it earlier.
Mistakes that prolong these incidents
- Filing it as a model bug and waiting for the vendor, while the agent's tokens remain live.
- Reimaging the laptop before copying transcripts and agent configuration.
- Rotating the user's password but not the tokens the agent and its MCP servers used.
- Removing the bad package but not the cached copy: the Nx advisory tells responders to clean npm, yarn and pnpm caches and internal registry mirrors [3].
Try it on ThreatDeck
- Respond to a compromised AI agent Work through the checklist; progress is saved in this browser.
- s1ngularity host artifacts (osquery) Sweep developer machines for the inventory files and bypass-flag runs.
- MCP config modified by an unexpected process Find hosts where a rogue server was written into an AI tool's config.
Self-check
4 questionsPick an answer to see whether it is right and why. Only your score is saved, in this browser.
-
The server worked as designed; the agent held a token that could read private repositories while processing attacker-written issues. Narrowing what the identity can reach is what removes the exfiltration path.
-
Containment is about the identity. Until its tokens are revoked and the tool disabled, the agent can act again. The document is the trigger and must be removed too, but only after it has been preserved as evidence.
-
Transcripts are the best record of what the agent was asked and did, and they age out. Reimaging or waiting loses them; collect them and treat them as sensitive because they may contain leaked secrets.
-
Persisted injected content or memory steers the agent each time it is read. Removing the trigger is a separate step from containing the identity, and OWASP tracks the persistence case as ASI06.
References
- NIST SP 800-61 Rev. 3: Incident Response Recommendations and Considerations for Cybersecurity Risk Management: A CSF 2.0 Community Profile. NIST, 2025-04.
- GitHub MCP Exploited: Accessing private repositories via MCP. Invariant Labs, 2025-05-26.
- Malicious versions of Nx and some supporting plugins were published (GHSA-cxm3-wv7p-598c). Nx (nrwl) security advisory, 2025-08-27.
- Security Update for Amazon Q Developer Extension for Visual Studio Code (Version #1.84), AWS-2025-015. AWS, 2025-07-23.
- s1ngularity: supply chain attack leaks secrets on GitHub: everything you need to know. Wiz Research, 2025-08.
- OWASP Top 10 for Agentic Applications for 2026. OWASP GenAI Security Project, 2025-12.
- Data usage (local caching of session transcripts). Anthropic (Claude Code docs).
- MCP Local Server Security. Model Context Protocol, 2026-07-28.