threatdeck
Live 13:45 UTC ·

Learn / Detecting and responding to AI abuse

Lesson 2 of 69 minReviewed

The telemetry you actually have

Know which cloud, application and agent logs record AI use, which are off until you turn them on, and how to keep prompt logs from becoming a new data leak.

Before writing detections, find out what is actually recorded. For AI services the answer splits in two: audit logs that say who called which API, which are usually on, and content logs that say what was asked and answered, which are usually off.

Model-use evidence by platform (defaults as documented, November 2026)
SourceWhat it recordsDefault
AWS CloudTrail, BedrockCaller, source IP, user agent, model id for InvokeModel, Converse and their streaming forms (management events); agent and knowledge-base calls are data eventsManagement events on; data events need event selectors [2]
Bedrock model invocation loggingFull request and response bodies, token counts, caller ARN, to CloudWatch Logs or S3Disabled [1]
Azure activity logControl-plane operations on the resource, such as listing keys or changing deploymentsCollected automatically [3]
Azure OpenAI resource logsRequest-level diagnostics routed by a diagnostic settingNot stored until you create one [3]
Google Cloud Data Access audit logs, Vertex AIGenerateContent, StreamGenerateContent, Predict and RawPredict calls (DATA_READ)Disabled; must be enabled explicitly [4, 5]

Two details catch teams out. CloudTrail never carries prompt or response text, so without invocation logging you can prove that a principal called a model but not what it generated; LLMjacking operators know this and read or delete the logging configuration first [6]. And the Converse API is logged under its own event name, not as InvokeModel, so a detection that only counts InvokeModel misses traffic attackers adopted within a month of the API's launch [6]. For streaming calls, Bedrock also writes a stream-completion event that carries token counts; count only AwsApiCall events when you count invocations, or you double them [2].

Application and agent telemetry

Cloud logs see the call, not the user behind it. Your own application should log, per request, the end user, the model, token counts, which tools were offered and called with which arguments, and the decision of any guardrail. The OpenTelemetry project maintains semantic conventions for generative AI clients, including MCP, so spans and metrics from different SDKs can share attribute names instead of each team inventing its own [7].

Coding agents run on endpoints, outside the cloud logs entirely. Claude Code, for example, exports OpenTelemetry events such as claude_code.user_prompt, claude_code.tool_decision and claude_code.tool_result once CLAUDE_CODE_ENABLE_TELEMETRY=1 is set; administrators can set the exporter for every user through managed settings, and a repository's own settings cannot enable telemetry or redirect it [8]. Content is opt-in: prompt text is redacted unless OTEL_LOG_USER_PROMPTS=1, and Bash commands and MCP tool names appear only with OTEL_LOG_TOOL_DETAILS=1 [8].

Managed settings: export coding-agent telemetry to your collector (illustrative)
{
  "env": {
    "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
    "OTEL_LOGS_EXPORTER": "otlp",
    "OTEL_EXPORTER_OTLP_PROTOCOL": "grpc",
    "OTEL_EXPORTER_OTLP_ENDPOINT": "https://otel.example.internal:4317",
    "OTEL_LOG_TOOL_DETAILS": "1"
  }
}

Prompt logs are sensitive data

A content log holds whatever users and retrieved documents put in front of the model: source code, customer records, credentials pasted by mistake. NIST's Generative AI Profile lists data privacy, the leakage or unauthorised use of personal and sensitive data, among the risks generative AI creates or worsens [9]. Logging everything for detection can therefore build the most sensitive dataset in the company. Practical rules:

  • Always log metadata (who, which model, tokens, tools, decisions); log content only where you have a stated detection or investigation use for it.
  • Store content logs in a separate destination with narrower access than general SIEM data, and set a retention period.
  • Alert on changes to the logging configuration itself: on Bedrock that means PutModelInvocationLoggingConfiguration and DeleteModelInvocationLoggingConfiguration [1].
  • Remember that enabling Google Cloud Data Access logs can be high-volume and billable, so scope them to the AI projects that need them [4].

Try it on ThreatDeck

Self-check

4 questions

Pick an answer to see whether it is right and why. Only your score is saved, in this browser.

  1. Bedrock model invocation logging was never enabled in an account. What can CloudTrail still tell you about a suspected abuse?
  2. A detection counts InvokeModel events per principal. Which abuse can it miss entirely?
  3. Your Vertex AI hunt returns nothing, ever. What is the most likely reason?
  4. What is the safer default for coding-agent telemetry in an organisation?

References

  1. Monitor model invocation using CloudWatch Logs and Amazon S3. AWS (Amazon Bedrock User Guide).
  2. Monitor Amazon Bedrock API calls using CloudTrail. AWS (Amazon Bedrock User Guide).
  3. Monitor Azure OpenAI in Microsoft Foundry Models (classic). Microsoft Learn.
  4. Enable Data Access audit logs. Google Cloud (Cloud Logging documentation).
  5. Vertex AI audit logging. Google Cloud documentation.
  6. The Growing Dangers of LLMjacking: Evolving Tactics and Evading Sanctions. Sysdig Threat Research Team, 2024-09-18.
  7. Semantic conventions for generative AI (spans, metrics, events, MCP). OpenTelemetry.
  8. Monitoring (OpenTelemetry) for Claude Code. Anthropic (Claude Code docs).
  9. NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST, 2024-07.
Completing the self-check also marks the lesson complete. Saved in this browser only.

Following

Follow the vendors, products, AI tools, techniques and actors you care about. Matching stories, CVEs and exploited bugs are marked, a Following tab appears in the feed, and returning visits show what changed. Saved only in this browser.

Presets

Get updates

A personal RSS feed of new stories on everything you follow. Paste it into any feed reader, Slack or Teams RSS app.