Skill 4.2.2: Configure and Use Tools to Troubleshoot and Analyze Resources
Task 4.2: Operating, Monitoring, and Securing ML and AI Solutions — Optimize and Manage ML and AI Infrastructure Costs and Performance
Objective: attribute where time and money went using evidence — an aggregate metric, a log line, or a cross-component trace — so cost and performance changes are driven by measurement, not instinct.
Tool Selection — Which Tool Answers Which Question
| Question | Tool | What It Returns |
|---|---|---|
| Is the endpoint capacity-bound, or is the model itself slow? | Amazon CloudWatch | Time-series aggregates (ModelLatency, OverheadLatency, CPUUtilization, GPUUtilization) |
| Which component added the latency in a single request? | AWS X-Ray | End-to-end trace, segments/subsegments, service map |
| What did an agent do per invocation, and what did it cost? | Bedrock AgentCore Observability | Spans for inference calls, tool calls, memory ops, sessions |
| Why did errors spike, and what did the app emit? | CloudWatch Logs + Logs Insights / metric filters | Raw events, query results, derived metrics |
| Is spend on track to breach a threshold? | AWS Budgets / Cost Explorer | Forecast alert; grouped/filtered historical spend |
Hard rule: metrics, logs, and traces are complementary, not interchangeable. CloudWatch tells you that a request was slow; X-Ray tells you which component made it slow; neither counts tokens — AgentCore Observability does.
Amazon CloudWatch — Metrics, Logs, Alarms, Dashboards
| Namespace | Carries |
|---|---|
/aws/sagemaker/Endpoints |
Invocations, InvocationsPerInstance, ModelLatency, OverheadLatency, CPUUtilization, MemoryUtilization, DiskUtilization, GPUUtilization, GPUMemoryUtilization |
/aws/sagemaker/TrainingJobs, TransformJobs, ProcessingJobs |
Job-level utilization and throughput |
/aws/sagemaker/HostMetrics |
Instance/host-level infrastructure |
/aws/bedrock |
Invocations, latency, input/output token counts, throttles, errors |
- ModelLatency vs OverheadLatency: model latency = time inside the model container; overhead = SageMaker response overhead. Rising overhead signals a platform/queueing problem — right-sizing the model will not fix it.
- Monitoring resolution: standard = 5-minute periods; detailed = 1-minute; 1-second high-resolution metrics exist only for custom metrics.
- Alarms drive levers, not just pages: CloudWatch alarms trigger Application Auto Scaling target-tracking policies (e.g.,
InvocationsPerInstance,CPUUtilization) — the mechanism that releases instances overnight to cut cost. - Dashboard design principle: pair measures that move together — utilization beside latency (is slowness a capacity problem?) and token usage beside invocation count (is the bill more traffic or heavier prompts? — different fixes).
- Logs Insights / metric filters convert log lines into metrics and alarms; subscription filters stream logs to Lambda/Kinesis/Firehose.
- Extended statistics: use p95/p99 for latency, never averages — averages hide the tail users feel.
- CloudWatch Application Signals / Container Insights provide APM-style RED metrics for containerized inference.
Gotchas: CloudWatch Logs retention defaults to "Never expire" — set a retention policy or log storage becomes a silent cost driver. Sparse metrics generate INSUFFICIENT_DATA alarms; set "treat missing data" explicitly to avoid flapping. Metric retention is tiered: 1-second = 3 hours, 1-minute = 15 days, 5-minute = 63 days, 1-hour = 455 days. Long historical baselines are rolled up, not raw.
Amazon Bedrock AgentCore Observability — Agent-Specific
- Per-invocation internals: model inference calls, tool invocations, memory (short/long-term) operations, and session context — built on OpenTelemetry (OTEL) GenAI semantic conventions.
- Answers the agent cost question: one user request amplifies into a planning call + one or more tool calls + a synthesis call, so watch cost and latency per invocation, not account totals. Retry loops invisible to a human appear here as bill growth with flat traffic.
- Surfaces into CloudWatch GenAI Observability (and can export to third-party APM backends).
- Session-level views trace a multi-turn conversation across invocations.
Gotcha: trace ingestion into CloudWatch requires CloudWatch Transaction Search to be enabled. The agent's IAM role must also permit log/trace delivery — otherwise observability data has nowhere to land.
AWS X-Ray — Distributed Tracing
- Structure: traces contain segments and subsegments; annotations are indexed and searchable (filter expressions), metadata is stored but not indexed.
- Service map visualizes dependency and error/latency propagation: API Gateway → Lambda → SageMaker endpoint → vector store → tool API.
- Latency attribution for agents: a trace separates model inference from tool call from retrieval from agent-to-agent coordination — the only way to know which to fix.
- Sampling: default reservoir of 1 request/second plus 5% of additional requests; custom rules raise capture for low-traffic endpoints or cut cost at high volume.
- Ingestion: X-Ray daemon (UDP 2000) as sidecar/agent on EC2/ECS/EKS; Lambda uses active tracing; SageMaker endpoints support X-Ray tracing for the full request path.
Gotchas: traces are retained 30 days only. If the execution role lacks X-Ray write permissions, traces silently vanish — the application reports no error. Sampling means a low-traffic endpoint may have no trace during an incident; tune sampling rules in advance.
Dashboards, Alarms, and Cost/Performance Linkage
- Decision dashboard = measures that shorten a decision: utilization + latency + errors + throughput + token usage + cost per invocation.
- Optimize only after attribution: a trace showing latency in a downstream tool call means added endpoint capacity is pure waste.
- AI-specific cost patterns to instrument: input/output token counts, embedding computation volume, vector index storage growth, prompt-cache hit/miss rate.
Security, Compliance & Access
| Control | Detail |
|---|---|
| IAM | CloudWatchAgentServerPolicy, AWSXRayDaemonWriteAccess, log-delivery permissions for AgentCore; least-privilege read-only roles for analysts |
| Encryption at rest | CloudWatch Logs encrypted with KMS CMK per log group (default: service-managed key); X-Ray trace data encrypted at rest with CMK configurable |
| Encryption in transit | TLS to CloudWatch / X-Ray / Bedrock APIs; PrivateLink for private paths |
| VPC endpoints (PrivateLink) | Interface endpoints for monitoring, logs, xray, bedrock-runtime, bedrock-agentcore keep telemetry off the public internet; endpoint policies restrict access |
| Cross-account | CloudWatch cross-account observability (OAM sinks/links) and cross-account X-Ray so one monitoring account sees source accounts |
| Sensitive data | Model-invocation logging and agent traces capture prompts/responses → apply CloudWatch Logs data protection (PII masking) and KMS; unmanaged full logging is both a compliance and cost risk |
Anti-Patterns & Distractor Traps
- Adding capacity to fix latency without a trace — the bottleneck is frequently a downstream dependency, not the model host.
- Applying instance right-sizing to a Bedrock token bill — there are no instances; levers are model selection, prompt discipline, caching, and capacity mode.
- Monitoring account-level token totals only — hides which agent/model/team drives spend; use cost allocation tags for attribution.
- Trusting averages — use p95/p99 and per-invocation metrics.
- Confusing the tools — CloudWatch = aggregates and logs; X-Ray = cross-component traces (30-day retention); AgentCore Observability = agent-internal spans (model + tool + memory).
- CloudWatch vs Budgets vs Trusted Advisor vs Cost Explorer — CloudWatch monitors operational metrics; Budgets alerts on defined spend/usage thresholds including forecasted; Cost Explorer analyzes and forecasts historical spend but sends no threshold alerts; Trusted Advisor runs best-practice checks (idle/underused resources) with no custom budget monitoring.