Skip to content

Skill 4.2.3: Set up dashboards to monitor performance metrics

Task 4.2: Operating, Monitoring, and Securing ML and AI Solutions - Optimize and manage ML and AI infrastructure costs and performance

Primary Objective: Build a single-pane operational view that correlates performance (latency, throughput, errors, utilization) with cost (tokens, compute hours, storage) to shorten optimization decisions.

Core Question: What are you actually paying for? → The dashboard must reflect the billing driver: compute hours vs tokens vs storage.

Telemetry Source Decision Matrix

Source Answers Best For Not For
CloudWatch Metrics Aggregate numeric time-series Utilization, invocations, latency, token counts, error rates Root-cause tracing across components
CloudWatch Logs / Logs Insights Event-level detail Debugging errors, prompt/response audit (if enabled) High-cardinality aggregate dashboards
AWS X-Ray Distributed request trace Latency attribution: model vs tool vs retrieval vs coordination Cost tracking
Bedrock AgentCore Observability Agent per-invocation telemetry Agent inference calls, tool invocations, memory ops, consumption patterns Non-agent workloads
Cost Explorer / Budgets Spend & forecast Cost driver analysis, threshold alerts Real-time performance metrics

Exam Gotcha: CloudWatch metrics are regional. A dashboard can display multiple regions, but each widget pulls from a specific region. Cross-account requires sharing + IAM roles.

Must-Have Dashboard Widgets by Workload

Workload Performance Metrics Cost / Efficiency Metrics
SageMaker Real-Time Endpoint Invocations, ModelLatency, OverheadLatency, p50/p90/p99, 4XX/5XX, CPU/GPU/Memory utilization, InvocationsPerInstance Instance hours, cost per invocation, utilization vs capacity
SageMaker Batch/Async Queue depth, processing latency, failures Cost per job, instance hours
Bedrock On-Demand Invocations, InvocationLatency, Throttles, InputTokenCount, OutputTokenCount Cost per token, tokens per invocation, token usage trend
Bedrock Provisioned Throughput Utilization, throttles, latency Reserved model units vs actual throughput, break-even vs on-demand
Agents (AgentCore) Invocations, tool calls, memory ops, latency per step, errors Tokens per invocation, cost per invocation, retry loops
Vector DB / Embeddings Query latency, throughput, storage size Embedding compute cost, storage cost, index efficiency

Exam Gotcha: Bedrock model invocation logging is not enabled by default. To capture prompts/responses for debugging or token analysis, you must explicitly enable it.

Dashboard Design Principles (High-Yield)

  • Correlate, don't isolate: Place utilization beside latency to distinguish capacity issues from code issues. Place token usage beside invocation count to distinguish traffic growth from heavier prompts.
  • Use percentiles: p50, p90, p99 for latency. Averages hide tail latency that users feel.
  • Use metric math: Derive error rate = errors / invocations; cost per token = spend / token count; cost per invocation = spend / invocations.
  • Alarm on decisions: Set CloudWatch Alarms on p99 latency, error rate, throttle count, token usage thresholds, and budget forecasts.
  • Use Anomaly Detection: CloudWatch Anomaly Detection for dynamic baselines on token usage or cost.
  • Tag everything: Cost allocation tags are the prerequisite for attributing spend to model, team, or environment. Without tags, cost reports show only service totals.
  • Single pane: Latency, throughput, errors, utilization, token usage, and cost should be readable together.

Instrumentation & Integration

  • SageMaker: Built-in endpoint metrics flow to CloudWatch. Enable detailed monitoring for 1-minute granularity. Use SageMaker Model Monitor for drift and bias metrics.
  • Bedrock: CloudWatch metrics for invocations, latency, tokens, throttles. Enable model invocation logging for prompt/response data.
  • Agents: Use AgentCore Observability to emit traces, metrics, and logs per invocation. Integrates with CloudWatch GenAI Observability.
  • X-Ray: Instrument application code to trace across microservices, Lambda, and model endpoints. Use ServiceLens to combine traces with CloudWatch metrics.
  • Custom metrics: Publish business metrics (e.g., cost per successful answer). High-resolution metrics (1-second) cost more than standard 1-minute metrics.

Security, Governance & Access

  • IAM: Use IAM policies to control who can view/edit dashboards. Cross-account dashboards require resource policies and IAM roles.
  • Encryption: CloudWatch Logs can be encrypted with KMS. Use VPC endpoints for CloudWatch and X-Ray to keep traffic private.
  • Data protection: CloudWatch Logs data protection policies can mask sensitive data in prompts/responses.
  • Audit: CloudTrail logs API activity, not performance. Use it for compliance, not latency debugging.

Anti-Patterns & Exam Traps

  • Applying endpoint levers to Bedrock: Right-sizing instances does nothing for token-based billing. Use model selection, prompt discipline, caching, and provisioned throughput.
  • Monitoring only averages: Hides p99 latency and cold starts.
  • Ignoring token metrics: Token usage is the primary cost driver for Bedrock. Dashboard it like any other metric.
  • No per-invocation agent visibility: Agent cost is amplification (planning + tools + synthesis). Track cost per invocation, not just total.
  • Using X-Ray for cost: X-Ray traces latency, not spend. Use Cost Explorer for cost drivers.
  • Using Cost Explorer for performance: It reports spend, not latency or utilization.
  • Forgetting tags: Without cost allocation tags, you cannot attribute spend to a model or team.
  • Not monitoring provisioned throughput utilization: If reserved capacity is underutilized, you're paying for unused throughput.

Exam Gotcha: Prompt caching is not a free discount. Cached tokens are billed at a reduced rate, but cache writes can be billed higher. A dynamic prefix invalidates the cache. Dashboard cache hit rate and token mix to verify savings.

Quick Decision Table

Need Use
Aggregate performance metric over time CloudWatch Metrics
Root-cause latency across components AWS X-Ray
Agent-specific per-invocation consumption Bedrock AgentCore Observability
Cost driver analysis AWS Cost Explorer
Threshold alert on spend AWS Budgets
Idle/underused resource check AWS Trusted Advisor
Rightsizing recommendations AWS Compute Optimizer
Drift/bias monitoring SageMaker Model Monitor
Trace + metric correlation CloudWatch ServiceLens

Hard Rule: The billing driver dictates the dashboard. Compute-hours workloads need instance utilization and auto-scaling metrics. Token-based workloads need token counts, cache hit rate, and cost per token. Storage-heavy workloads need vector index size and query efficiency.