Skip to content

Skill 4.2.8: Monitor agent resource consumption patterns

Task 4.2: Operating, Monitoring, and Securing ML and AI Solutions — Optimize and manage ML and AI infrastructure costs and performance

Core concept: An agent is a cost amplifier, not a single inference call. One user request fans out into a planning call → N tool invocations → optional retrieval → a synthesis call. The unit of optimization is therefore cost and latency per agent invocation (and per session), never the account-level token total.

Hard rule: CloudWatch Bedrock runtime metrics are dimensioned by ModelId, not by agent, session, or invocation. Per-invocation attribution requires tracing (AgentCore Observability / X-Ray), not CloudWatch model metrics.

Agent invocation anatomy — what actually consumes resources

Consumption source Driver Monitoring signal
Model calls (input tokens) System prompt + tool schemas + conversation history + retrieved memory, re-sent every turn InputTokenCount per span
Model calls (output tokens) Reasoning/plan steps and final answer; uncapped max_tokens OutputTokenCount per span
Tool invocations Fan-out, parallel calls, and retry loops Tool-call span count + duration
Memory operations Short-term (session) writes; long-term memory reads inject tokens on every invocation Memory span count, retrieval token size
Orchestration latency Multi-agent coordination, sequential step chaining Trace critical path
Infrastructure Runtime compute held during the agent's idle waits Session duration / idle timeout

Instrumentation decision matrix

Question you are answering Right tool
Which component made this invocation slow? X-Ray trace (segments → subsegments) or AgentCore Observability trace
How many tokens per invocation? AgentCore Observability spans (session/trace-level)
How many tokens per model, account-wide? CloudWatch AWS/Bedrock metrics
Is a rising bill more traffic or heavier prompts? Token usage beside invocation count on one dashboard
Which team/model/environment caused the increase? Cost Explorer grouped by cost allocation tag
Alert before the threshold is crossed AWS Budgets (forecasted alert)
Idle/underused agent runtime capacity Trusted Advisor cost checks, Compute Optimizer

Tag prerequisite: Cost allocation tags must be activated in the Billing console before they appear as Cost Explorer groupings. Without them, a cost report shows a service total and no attribution — the single most common reason agent cost investigations stall.

Metrics and traces that matter

  • CloudWatch AWS/Bedrock: Invocations, InvocationLatency, InputTokenCount, OutputTokenCount, InvocationThrottles, InvocationClientErrors, InvocationServerErrors.
  • CloudWatch Logs: Bedrock model invocation logging is off by default; enable it (to CloudWatch Logs or S3) only with a retention policy, because full prompt/response bodies are the fastest-growing log volume in an agent workload.
  • AgentCore Observability: captures inference calls, tool invocations, and memory operations inside an invocation — the only view that separates model time from tool time from coordination time.
  • X-Ray: distributed trace that attributes latency to a component so you don't answer a slow tool call with more capacity everywhere.
  • Token usage as a first-class dashboard metric: once the whole team can see token spend, behavior changes before any threshold is reached.

CloudWatch Logs retention defaults to "Never expire." Set an explicit retention period on every agent log group — it is a pure cost leak.

X-Ray default sampling rule: reservoir of 1 request/second plus 5% of additional requests, with 30-day trace retention. Low-volume agents can be effectively invisible under the default rule; adjust sampling deliberately.

Agent-specific optimization levers

  • Cap the loop: set a maximum iteration/step count so a self-correction cycle cannot run unbounded.
  • Cap output: bound max_tokens to control generated-token spend.
  • Trim history: summarize or window conversation context instead of resending it in full each turn.
  • Prompt caching: cache the stable prefix (system instructions + tool schemas); keep volatile content (timestamps, request IDs) out of the cached prefix or every request is a cache miss.
  • Model routing: send simple intents to a smaller model; reserve the large model for steps that need it.
  • Cache tool responses: repeated identical tool calls are pure waste.
  • Batch tolerant work: move non-interactive agent workloads off real-time paths.
  • Bound sessions: enforce max session duration and idle timeout so runtime isn't held during agent stalls.

Anti-patterns and distractor traps

  • Optimizing the account-level token total instead of per-invocation cost — hides which agent, step, or tool is responsible.
  • Monitoring model calls only. Tool calls and memory retrievals are where agent latency and cost actually hide.
  • Scaling capacity to fix latency that a trace would have attributed to one slow tool or one retrieval step.
  • Applying endpoint levers (instance right-sizing, auto scaling, Savings Plans) to an agent's token bill — or applying token levers to a hosted endpoint. The bills are not interchangeable.
  • Assuming a retry loop is visible. A silent retry loop grows the bill with zero change in traffic; watch retries per invocation, not just invocation count.
  • Treating prompt caching as a free discount. Cache writes can be billed higher than uncached input, checkpoints have a minimum prefix length, and any dynamic content in the prefix invalidates it — the saving must repay the writes.
  • Leaving X-Ray sampling at default and concluding an agent has no latency problem from an empty trace list.

Security and compliance for agent observability

  • IAM: separate the agent's execution role from the observability read role; grant AWSXRayDaemonWriteAccess-equivalent permissions to the trace-emitting component only.
  • KMS: encrypt CloudWatch Logs groups with a customer-managed key; enable CloudWatch Logs data protection policies to mask PII/PHI in prompt and response bodies.
  • VPC endpoints: use interface endpoints for logs, monitoring, and xray so telemetry never traverses the public internet.
  • X-Ray annotations: keep them free of sensitive data — annotations are indexed and queryable.
  • Least privilege on log destinations: Bedrock model invocation logging writes full prompts; restrict read access to the S3/Logs destination tightly.