Skip to content

Skill 4.2.3: Set up dashboards to monitor performance metrics.

Question 1 - CloudWatch Dashboard for SageMaker Endpoint Performance

A financial services company has deployed a fraud detection model on Amazon SageMaker real-time inference endpoints. The ML engineering team needs to build a single monitoring dashboard that consolidates endpoint invocation latency, error rates, CPU/GPU utilization, and the number of invocations per instance. The team must be able to identify capacity bottlenecks quickly without having to navigate between separate consoles.

Which AWS service should the team use to build this dashboard with the least operational overhead?

  • A) Use Amazon CloudWatch to create a custom dashboard with metric widgets for Invocations, Invocation4XXErrors, ModelLatency, CPUUtilization, and InvocationModelErrors.
  • B) Use Amazon QuickSight to connect directly to SageMaker endpoint logs and build a visual analytics dashboard.
  • C) Use AWS X-Ray trace data to compute aggregate metrics and render them in a Grafana instance deployed on Amazon EC2.
  • D) Use AWS Glue DataBrew to transform SageMaker metrics into a dashboard-readable format and serve it using a static S3 website.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Amazon CloudWatch natively ingests SageMaker endpoint metrics such as Invocations, Invocation4XXErrors, Invocation5XXErrors, ModelLatency, OverheadLatency, CPUUtilization, GPUUtilization, and MemoryUtilization. A CloudWatch dashboard can consolidate these metrics into a single pane using metric widgets, requiring no additional infrastructure.

Distractor Analysis

  • B is Incorrect: Amazon QuickSight is a business intelligence service primarily used for visual analytics on data sources like S3, Redshift, or Athena. While it can visualize data, connecting QuickSight directly to endpoint logs requires heavy ETL and is not the native monitoring solution compared to CloudWatch dashboards.
  • C is Incorrect: AWS X-Ray provides distributed tracing analysis (service maps and trace details), not aggregate performance metric visualization for resource utilization. Deploying Grafana on EC2 introduces significant operational overhead for infrastructure patching and deployment.
  • D is Incorrect: AWS Glue DataBrew is a visual data preparation tool for analytical datasets, not a monitoring dashboard solution. Serving dashboards on a static S3 site would not provide real-time metric updates and violates the least operational overhead constraint.

Key Takeaway: Amazon CloudWatch dashboards provide native, zero-overhead visualization for SageMaker endpoint metrics; no ETL or third-party infrastructure is required.

Question 2 - Bedrock Agent Observability Dashboard

A retail company has deployed a customer service agent using Amazon Bedrock AgentCore with multiple action groups for inventory lookup, order tracking, and return processing. The team notices that some agent invocations are incurring unexpectedly high costs and wants to build a dashboard to monitor the number of LLM inference calls, tool invocations, and memory operations per agent invocation to identify inefficient orchestration loops or redundant tool calls.

Which combination of AWS tools should the team configure to build this dashboard? (Select TWO)

  • A) Enable Amazon Bedrock AgentCore Observability to stream agent trace data to Amazon CloudWatch.
  • B) Create a CloudWatch dashboard with widgets for BedrockAgent.Invocations, BedrockAgent.ToolInvocations, BedrockAgent.LLMCalls, and BedrockAgent.MemoryOperations metrics.
  • C) Enable AWS X-Ray tracing on the agent and configure the X-Ray console to display Bedrock agent trace segments.
  • D) Configure an Amazon S3 bucket to store agent invocation logs, then query them with Amazon Athena and visualize results using Amazon QuickSight.
  • E) Use Amazon CloudTrail to capture agent API calls and build a dashboard from CloudTrail event logs.
Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: Amazon Bedrock AgentCore Observability captures detailed telemetry for agent invocations, including inference calls, tool invocations, memory operations, and guardrail evaluations. When enabled, this observability data streams directly to Amazon CloudWatch for near-real-time monitoring.
  • B is Correct: With AgentCore Observability enabled, CloudWatch receives Bedrock Agent metrics. A CloudWatch dashboard can display metrics such as BedrockAgent.Invocations, BedrockAgent.ToolInvocations, BedrockAgent.LLMCalls, BedrockAgent.MemoryOperations, and BedrockAgent.GuardrailInvocations per agent, per agent alias, or aggregated across all agents.

Distractor Analysis

  • C is Incorrect: While AWS X-Ray can trace distributed application components, Amazon Bedrock AgentCore Observability is specifically designed to capture agent-internal telemetry (tool calls, memory, LLM inference). X-Ray alone does not expose the granular per-invocation agent metrics like tool invocation counts or memory operation counts without custom instrumentation.
  • D is Incorrect: Using an S3 bucket for logs, querying with Athena, and visualizing with QuickSight introduces significant architectural complexity and operational overhead compared to AgentCore Observability's native CloudWatch integration.
  • E is Incorrect: AWS CloudTrail captures API governance and compliance events (e.g., "CreateAgent", "InvokeAgent"), not the internal resource consumption breakdown of individual agent invocations.

Key Takeaway: Amazon Bedrock AgentCore Observability is the purpose-built feature for exposing per-invocation agent consumption metrics (LLM calls, tool calls, memory ops) to CloudWatch dashboards.

Question 3 - Token Usage Dashboard for Bedrock Application

An AI startup is running a multi-tenant generative AI application on Amazon Bedrock using multiple foundation models (Anthropic Claude and Amazon Titan). The application developers want to monitor token usage per model, per application, and per tenant to understand cost drivers. They need to visualize token metrics on a dashboard and set proactive alerts.

Which AWS service combination is MOST cost-effective for this requirement?

  • A) Publish custom metrics for input tokens, output tokens, and invocation counts tagged by model, application, and tenant to Amazon CloudWatch using PutMetricData, then build a CloudWatch dashboard.
  • B) Deploy a self-hosted Grafana instance on an EC2 instance and configure the Amazon Managed Service for Prometheus to scrape Bedrock token metrics.
  • C) Use Amazon OpenSearch Service to index Bedrock invocation logs and create a Kibana dashboard for token-level analytics.
  • D) Disable CloudWatch metrics for Bedrock and instead rely on AWS Cost Explorer's daily granularity reports to track token usage by model.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Amazon Bedrock already emits InputTokenCount, OutputTokenCount, Invocations, and InvocationClientErrors metrics to CloudWatch. Even when native metrics provide model-level granularity, adding custom dimensions (tenant, application) via PutMetricData allows for flexible grouping. CloudWatch dashboards can visualize these metrics, and CloudWatch Alarms can trigger alerts. This requires no additional infrastructure.

Distractor Analysis

  • B is Incorrect: Deploying and managing a self-hosted Grafana instance on EC2 introduces substantial operational overhead (patching, scaling, monitoring) and is not the most cost-effective option. Amazon Managed Service for Prometheus does not natively scrape Bedrock token metrics without a custom exporter.
  • C is Incorrect: Using OpenSearch Service and Kibana for token analytics is a valid pattern for log-based analytics but requires configuring log ingestion, index mappings, and managing an OpenSearch cluster, which is significantly more operationally complex than CloudWatch dashboards.
  • D is Incorrect: AWS Cost Explorer provides cost data at daily granularity but lacks the real-time, fine-grained token-level metrics required for monitoring per-model and per-tenant usage. Additionally, disabling CloudWatch metrics would remove the native ingestion path for token telemetry.

Key Takeaway: CloudWatch is the central observability hub for Bedrock token metrics; custom metric dimensions via PutMetricData enable per-tenant and per-application cost attribution without deploying third-party dashboards.

Question 4 - Multi-Model Endpoint Dashboard

A healthcare analytics company hosts multiple small ML models on a single Amazon SageMaker multi-model endpoint (MME) to reduce costs. The team wants to monitor the performance of each model deployed on the shared endpoint, including per-model invocation latency, per-model error rates, and per-model cache hit/miss metrics.

Which CloudWatch metric configuration should the team use to build this dashboard?

  • A) Use the ModelName dimension available in SageMaker endpoint metrics to filter Invocations, ModelLatency, and InvocationModelErrors per model in CloudWatch dashboard widgets.
  • B) Deploy a separate CloudWatch agent on each SageMaker instance to collect per-model metrics and publish them to a custom namespace.
  • C) Use AWS X-Ray subsegment annotations to manually instrument model loading and invocation, then create a dashboard in the X-Ray console.
  • D) Extract per-model metrics from SageMaker endpoint access logs stored in S3 using AWS Glue jobs scheduled hourly.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: SageMaker endpoint metrics automatically include a ModelName dimension for multi-model endpoints and inference components. This allows CloudWatch dashboards to filter metrics like Invocations, ModelLatency, OverheadLatency, and InvocationModelErrors by individual model name, providing per-model observability without custom instrumentation.

Distractor Analysis

  • B is Incorrect: Deploying a custom CloudWatch agent on SageMaker instances is unnecessary and introduces operational overhead. SageMaker already emits per-model metrics with the ModelName dimension. Custom agents also cannot easily access ModelLatency metrics internal to SageMaker hosting.
  • C is Incorrect: X-Ray is designed for distributed tracing analysis and debugging. Building a performance dashboard in the X-Ray console is not supported; X-Ray provides trace exploration, not dashboarding with aggregate metric widgets.
  • D is Incorrect: Extracting metrics from S3 access logs using periodic AWS Glue jobs introduces unnecessary latency and complexity. SageMaker's native CloudWatch metrics with the ModelName dimension provide real-time, per-model granularity without batch processing.

Key Takeaway: SageMaker multi-model endpoints automatically emit per-model metrics using the ModelName dimension—no custom monitoring agents or log processing pipelines are needed.

Question 5 - Prompt Caching Performance Dashboard

An AI engineering team has implemented prompt caching on Amazon Bedrock to reduce token processing costs for a document summarization application with a large, static system prompt. The team needs a dashboard that effectively tracks whether the cache is being utilized effectively, including cache hit rates, cached token usage, and non-cached token usage over time.

Which metric configuration should the team use for this dashboard?

  • A) Use CloudWatch metrics published by Amazon Bedrock for CachedInputTokenCount, NonCachedInputTokenCount, and OutputTokenCount to create dashboard widgets showing cached vs. non-cached input token proportions over time.
  • B) Configure AWS CloudTrail event logging to capture cache hit/miss events and build a dashboard in CloudTrail Insights.
  • C) Deploy a custom Lambda function that intercepts Bedrock API responses, parses token usage from the model response headers, and pushes cache utilization metrics to CloudWatch.
  • D) Use Amazon Bedrock Guardrails metrics to track cached and non-cached token usage, then visualize these metrics in a CloudWatch dashboard.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Amazon Bedrock emits CloudWatch metrics including CachedInputTokenCount, NonCachedInputTokenCount, and OutputTokenCount (where prompt caching is supported). These metrics can be visualized in a CloudWatch dashboard to track cache efficiency, with the ratio of cached to non-cached input tokens indicating whether the prompt prefix is being effectively reused.

Distractor Analysis

  • B is Incorrect: AWS CloudTrail captures API governance events (e.g., InvokeModel calls) but does not include cache hit/miss telemetry or token-level usage details. CloudTrail Insights is designed for anomaly detection in API activity, not for building performance dashboards.
  • C is Incorrect: Writing a custom Lambda function to intercept API responses and parse token usage is unnecessary. Bedrock already provides native CloudWatch metrics for cached and non-cached token counts, and this approach would not capture all invocations reliably without significant engineering effort.
  • D is Incorrect: Amazon Bedrock Guardrails metrics track policy evaluations, PII detection, and content filtering—not token caching. Trading on Guardrails for token metrics would provide no data for cache utilization tracking.

Key Takeaway: Amazon Bedrock natively emits CachedInputTokenCount and NonCachedInputTokenCount metrics to CloudWatch, enabling direct dashboarding of prompt caching effectiveness without custom instrumentation.

Question 6 - OpenSearch Vector Database Performance Dashboard

A knowledge management startup uses Amazon OpenSearch Serverless to store vector embeddings for a retrieval-augmented generation (RAG) application. The ML operations team needs to monitor vector index performance, including query latency, indexing throughput, and vector storage consumption.

Which TWO CloudWatch metrics should the team include in their dashboard? (Select TWO)

  • A) OpenSearchQueryLatency metric to track average query response time.
  • B) OpenSearchIndexingRate metric to track document ingestion throughput.
  • C) VectorStorageBytes metric tracked in the OpenSearch Serverless namespace.
  • D) InvocationModelErrors from SageMaker to detect vector retrieval failures.
  • E) TokenCount from Bedrock to estimate vector storage requirements.
Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: Amazon OpenSearch Serverless (via its OpenSearch Service integration) emits CloudWatch metrics including query latency. Monitoring OpenSearchQueryLatency allows the team to detect performance degradation in vector similarity search operations, which is critical for RAG latency.
  • B is Correct: OpenSearchIndexingRate tracks the rate at which documents and vectors are being ingested into the collection. This metric is essential for validating that indexing pipelines meet throughput requirements.

Distractor Analysis

  • C is Incorrect: While vector storage consumption is important, the specific metric name VectorStorageBytes is not a valid CloudWatch metric in the OpenSearch Serverless namespace. Valid storage metrics include OpenSearchStorageBytes or cluster-level storage measurements, not a dedicated VectorStorageBytes metric.
  • D is Incorrect: InvocationModelErrors is a SageMaker endpoint metric for tracking model invocation failures, not for monitoring vector database performance. Using SageMaker metrics in a vector database dashboard would provide no relevant data.
  • E is Incorrect: TokenCount from Bedrock relates to foundation model inference token usage, not vector storage requirements. Vector storage depends on embedding dimensions and vector count, not token consumption.

Key Takeaway: OpenSearch Serverless metrics are emitted to CloudWatch for query latency and indexing throughput; monitoring these ensures RAG performance metrics are visible and actionable.

Question 7 - Cross-Service Dashboard for RAG Application

An enterprise has deployed a complete RAG application using Amazon Bedrock Knowledge Bases (with an OpenSearch Serverless vector store), Amazon SageMaker for a custom embedding model, and Amazon Bedrock for LLM inference. The platform team wants to create a unified dashboard to identify whether performance issues originate from the LLM, the embedding model, or the vector database during a single end-to-end invocation.

Which TWO AWS services should the team configure for comprehensive observability? (Select TWO)

  • A) Use AWS X-Ray to trace the end-to-end invocation across SageMaker embedding endpoints, OpenSearch Serverless vector retrieval, and Bedrock LLM inference.
  • B) Use Amazon CloudWatch dashboards with metric widgets from SageMaker, Bedrock, and OpenSearch namespaces to provide service-level performance visibility.
  • C) Use AWS Step Functions execution history to build a dashboard showing component-level latency decomposition.
  • D) Use Amazon Detective to analyze service interactions and recommend performance optimizations.
  • E) Use AWS CloudFormation drift detection to identify performance bottlenecks in the RAG pipeline.
Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: AWS X-Ray integrates with Amazon API Gateway, AWS Lambda, and other AWS services. By instrumenting the full request path, X-Ray can trace across custom embedding endpoints on SageMaker, OpenSearch vector retrieval calls, and Bedrock LLM invocations, providing an end-to-end latency breakdown per component.
  • B is Correct: CloudWatch dashboards can consolidate metrics from multiple AWS service namespaces (SageMaker ModelLatency, Bedrock InputTokenCount / Invocations, OpenSearch query latency) into a single pane of glass. This enables side-by-side comparison of service-level metrics to identify which component is exhibiting degraded performance.

Distractor Analysis

  • C is Incorrect: AWS Step Functions execution history provides orchestration-level logs when Step Functions is used to coordinate the pipeline. While it offers step-by-step state transitions, it is not a native dashboarding tool for performance metrics and does not capture internal service telemetry (like OpenSearch latency or Bedrock token usage) unless explicitly instrumented.
  • D is Incorrect: Amazon Detective is a security investigation service that analyzes AWS CloudTrail, VPC Flow Logs, and GuardDuty findings to find security anomalies. It is not designed for performance monitoring or latency analysis.
  • E is Incorrect: AWS CloudFormation drift detection checks whether deployed resources have diverged from their declared template configuration. It has no role in monitoring application performance or identifying latency bottlenecks.

Key Takeaway: For cross-service observability, combine CloudWatch dashboards for aggregate service-level metrics with X-Ray tracing for per-request component-level latency attribution.

Question 8 - GPU Utilization Dashboard for SageMaker Inference

A robotics company runs computer vision models on GPU-accelerated SageMaker inference endpoints (ml.g4dn.xlarge instances). The team needs to monitor GPU utilization to determine whether the GPU instances are being effectively utilized or if they are over-provisioned for a CPU-bound workload.

Which CloudWatch metric should the team use in their dashboard to directly observe GPU utilization?

  • A) GPUUtilization metric emitted by SageMaker inference endpoints, visualized in a CloudWatch dashboard widget with a time-series graph.
  • B) CPUUtilization metric, since GPU utilization is derived from CPU metrics on SageMaker.
  • C) DCGM_FI_DEV_GPU_UTIL collected by AWS X-Ray tracing.
  • D) GPUReservation metric from AWS Compute Optimizer.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: SageMaker inference endpoints automatically emit GPUUtilization and MemoryUtilization metrics to CloudWatch when running on GPU instances (ml.g4dn, ml.g5, ml.p3, ml.p4d, etc.). This metric reflects the percentage of GPU compute capacity being used, enabling the team to right-size the endpoint.

Distractor Analysis

  • B is Incorrect: CPUUtilization captures CPU usage on the SageMaker instance, not GPU compute activity. For GPU-bound workloads, relying on CPU metrics would provide an incomplete picture of actual utilization.
  • C is Incorrect: DCGM_FI_DEV_GPU_UTIL (from NVIDIA DCGM) is a GPU metric format, but it is not collected by AWS X-Ray. X-Ray is a tracing service, not a GPU telemetry collection tool. SageMaker's native CloudWatch GPUUtilization metric eliminates the need for DCGM integration.
  • D is Incorrect: AWS Compute Optimizer provides resource recommendations based on historical utilization, but GPUReservation is not a valid metric it exposes. Compute Optimizer recommendations are not real-time metrics suitable for dashboards.

Key Takeaway: SageMaker endpoints automatically emit GPUUtilization to CloudWatch on GPU instance types—use this metric directly in dashboards to assess GPU provisioning efficiency.

Question 9 - Vector Database Storage Optimization Dashboard

A media streaming company uses Amazon OpenSearch Service (provisioned, not Serverless) to store embeddings for a recommendation engine. The operations team wants to monitor storage utilization and implement storage optimization strategies. They also need visibility into whether the vector index is approaching storage capacity limits.

Which CloudWatch metrics should the team include in their dashboard to monitor storage health? (Select ONE)

  • A) ClusterUsedSpace and ClusterTotalSpace from the OpenSearch Service namespace.
  • B) S3BucketSizeBytes from the Amazon S3 namespace, since vector storage is backed by S3.
  • C) StorageClassBytes from the Kinesis Data Streams namespace.
  • D) Only JVMGCOldCollectionCount, as storage is managed entirely by JVM heap.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Amazon OpenSearch Service (provisioned) emits ClusterUsedSpace and ClusterTotalSpace metrics to CloudWatch. These metrics indicate the amount of storage consumed and the total provisioned storage capacity, allowing the team to monitor storage headroom and trigger alerts before running out of capacity.

Distractor Analysis

  • B is Incorrect: Amazon OpenSearch Service stores its indices on EBS volumes or instance store attached to the cluster, not on Amazon S3. S3BucketSizeBytes would only track S3 bucket storage, which is not relevant for OpenSearch cluster storage management.
  • C is Incorrect: Kinesis Data Streams metrics relate to streaming data ingestion and retention, not to OpenSearch storage monitoring.
  • D is Incorrect: JVMGCOldCollectionCount tracks garbage collection frequency on JVM heap, which is a performance metric related to memory management, not storage capacity. While JVM pressure affects performance, storage space is a distinct metric.

Key Takeaway: For OpenSearch Service (provisioned), monitor ClusterUsedSpace and ClusterTotalSpace in CloudWatch to proactively manage vector index storage capacity.

Question 10 - Unified Dashboard for ML Cost and Performance

A large e-commerce company wants to build a single executive-level dashboard that shows both performance metrics (endpoint latency, error rates) and cost metrics (estimated daily spend, token usage costs) for their ML infrastructure. The dashboard should be shareable with non-technical stakeholders.

Which combination should the team use to build this dashboard? (Select ONE)

  • A) Build a custom Amazon CloudWatch dashboard with metric widgets for performance metrics and estimated cost widgets from AWS Cost Explorer data using the CloudWatch console.
  • B) Use Amazon QuickSight to query CloudWatch metrics and AWS Cost Explorer APIs directly and build a shareable dashboard.
  • C) Deploy a Tableau server on EC2 and connect it to both CloudWatch and AWS Billing APIs.
  • D) Export CloudWatch metrics to Amazon S3 and load them into AWS Glue Data Catalog, then use Amazon Athena to build dashboards in a custom web portal.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Amazon CloudWatch dashboards support native metric widgets for all AWS service metrics (SageMaker, Bedrock, OpenSearch, etc.) and also support estimated cost widgets if billing alerts are enabled. This provides a unified dashboard for both performance and cost metrics without requiring any additional infrastructure. CloudWatch dashboards can be shared via IAM permissions or made public via a shareable link.

Distractor Analysis

  • B is Incorrect: While Amazon QuickSight can query some AWS data sources, it does not natively integrate with CloudWatch metrics and AWS Cost Explorer APIs without significant ETL/API integration work. QuickSight is designed for business analytics on data warehouses, not for operational monitoring dashboards.
  • C is Incorrect: Deploying a self-hosted Tableau server introduces substantial licensing costs and operational overhead for infrastructure management, patching, and scaling. This violates the requirement for a straightforward, cost-effective solution that can be shared with stakeholders.
  • D is Incorrect: Exporting CloudWatch metrics to S3 and building a custom Athena-based web dashboard is a multi-step, operationally complex data pipeline that requires custom web development, Athena query scheduling, and data refresh mechanisms. This is significantly more complex than a native CloudWatch dashboard.

Key Takeaway: CloudWatch dashboards natively combine performance metrics with cost widgets (powered by AWS Cost Explorer data), providing a single-pane-of-glass view for both operational and executive stakeholders.