Skill 4.2.3: Set up dashboards to monitor performance metrics

Question	Answer & Explanation
### Question 1 - Monitoring SageMaker Real-Time Inference Latency A machine learning team has deployed a recommendation model to an Amazon SageMaker real-time endpoint. The endpoint handles interactive traffic from a web application, and the team must ensure the 99^th percentile latency remains under 100 milliseconds. To troubleshoot a recent spike in latency, the team needs to visualize the endpoint's CPU utilization alongside the model latency in a single view. Which approach meets these requirements with the LEAST operational overhead? Select ONE: - A) Configure AWS X-Ray to trace requests from the web application to the SageMaker endpoint and build a custom trace map in the X-Ray console. - B) Create an Amazon CloudWatch dashboard combining the `CPUUtilization` metric from SageMaker and the `ModelLatency` metric from the endpoint's CloudWatch namespace. - C) Stream the SageMaker endpoint logs to Amazon OpenSearch Service and create a Kibana dashboard to query and visualize the latency logs. - D) Deploy a custom AWS Lambda function to periodically invoke the SageMaker endpoint, measure the response time, and publish a custom metric to CloudWatch.	Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon CloudWatch natively provides SageMaker endpoint metrics such as `CPUUtilization` and `ModelLatency`. Creating a CloudWatch dashboard to display these metrics together allows the team to correlate compute utilization with latency spikes in a single view with zero custom code, meeting the requirement with the least operational overhead. Distractor Analysis: - A is Incorrect: While AWS X-Ray is excellent for tracing request paths across microservices, it introduces more operational overhead (requiring SDK instrumentation) and is not the simplest way to plot standard SageMaker CloudWatch metrics like CPU utilization against latency. - C is Incorrect: Streaming logs to OpenSearch and using Kibana is a valid observability pattern, but it introduces unnecessary operational overhead and cost for simply viewing standard CloudWatch metrics that are already available. - D is Incorrect: Writing a custom Lambda function to invoke the endpoint and publish custom metrics introduces custom code, maintenance, and additional invocation costs, violating the least operational overhead constraint. > Key Takeaway: For standard AWS service metrics, always prefer native Amazon CloudWatch dashboards over custom logging pipelines or synthetic checks to minimize operational overhead.
### Question 2 - Visualizing Amazon Bedrock Token Consumption An AI engineering team is building an enterprise chatbot using Amazon Bedrock with the Anthropic Claude model. The product manager wants a weekly report showing the total number of input tokens and output tokens consumed by the chatbot to track cost trends. The solution must use AWS native services and require minimal maintenance. How should the engineer set up this monitoring solution? Select ONE: - A) Enable AWS CloudTrail to log all `InvokeModel` API calls, parse the JSON payloads using Amazon Athena, and schedule a daily SQL query report. - B) Create an Amazon CloudWatch dashboard and add the `InputTokenCount` and `OutputTokenCount` metrics provided by Amazon Bedrock. - C) Implement a custom proxy API Gateway in front of Bedrock to count tokens in the request/response payloads and log them to CloudWatch Logs. - D) Use AWS Cost Explorer to filter by the Amazon Bedrock service and group the costs by usage type on a weekly basis.	Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock automatically publishes operational metrics to Amazon CloudWatch, including `InputTokenCount` and `OutputTokenCount`. Creating a CloudWatch dashboard using these native metrics provides a low-maintenance, real-time view of token consumption. Distractor Analysis: - A is Incorrect: CloudTrail logs management API calls, but parsing token counts from large payloads via Athena introduces unnecessary complexity and maintenance overhead. - C is Incorrect: Deploying a custom proxy API Gateway to count tokens requires custom code, introduces latency, and violates the minimal maintenance constraint. - D is Incorrect: AWS Cost Explorer shows the dollar cost of usage, but does not display the raw operational metric of token counts, which the product manager specifically requested for tracking volume trends. > Key Takeaway: Amazon Bedrock natively publishes `InputTokenCount` and `OutputTokenCount` metrics to CloudWatch, making it the most efficient way to monitor token-based usage without custom logging pipelines.
### Question 3 - Monitoring SageMaker Multi-Model Endpoints (MME) A data science team has deployed 50 distinct XGBoost models to an Amazon SageMaker Multi-Model Endpoint (MME) to serve intermittent prediction requests. The team needs to monitor how often models are being loaded into memory and whether the endpoint is running out of memory due to high model churn. Which CloudWatch metrics should the team include in a dashboard to monitor these specific performance characteristics? Select TWO: - A) `InvocationsPerInstance` - B) `ModelLoadingWaitTime` - C) `MemoryUsed` - D) `OverheadLatency` - E) `ApproximateBacklogSize`	Correct Answers: B and C Correct Answer Analysis: - B is Correct: `ModelLoadingWaitTime` measures the time it takes for a model to be loaded from Amazon S3 into the container's memory. High values indicate frequent model loading (churn), which impacts latency. - C is Correct: `MemoryUsed` tracks the total memory utilized by the endpoint container. Monitoring this ensures the instance hasn't exhausted its RAM, a common issue when many models are loaded dynamically in an MME. Distractor Analysis: - A is Incorrect: `InvocationsPerInstance` tracks traffic volume but does not give insight into model loading times or memory pressure. - D is Incorrect: `OverheadLatency` measures the time spent on framework overhead, not specifically tied to the memory or model loading dynamics of an MME. - E is Incorrect: `ApproximateBacklogSize` is a metric used for Asynchronous Inference or Serverless endpoints, not Multi-Model Endpoints. > Key Takeaway: For SageMaker Multi-Model Endpoints, `ModelLoadingWaitTime` and `MemoryUsed` are the critical metrics to monitor model churn and capacity, as models are dynamically loaded into shared memory.
### Question 4 - Tracing Bedrock Agent Tool Invocations An enterprise is running a customer support agent built with Amazon Bedrock Agents. The agent orchestrates multiple steps: retrieving a user profile from an internal API, querying a knowledge base, and synthesizing a response. Users are reporting intermittent high latency, and the team needs to identify exactly which component (API call, knowledge base retrieval, or FM synthesis) is causing the delay on a per-request basis. Which AWS service should the team use to set up this per-request tracing dashboard? Select ONE: - A) Amazon CloudWatch Container Insights - B) AWS X-Ray with Bedrock AgentCore Observability - C) AWS CloudTrail with Amazon Athena - D) Amazon OpenSearch Service with VPC Flow Logs	Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock AgentCore Observability integrates with AWS X-Ray to provide end-to-end tracing of agent invocations. This allows engineers to see the exact latency breakdown for each step of the agent's orchestration (model inference, API tool calls, knowledge base retrieval) on a per-request basis. Distractor Analysis: - A is Incorrect: CloudWatch Container Insights is designed for monitoring containerized applications (like ECS/EKS) and does not provide semantic tracing of Bedrock Agent orchestration steps. - C is Incorrect: CloudTrail records API audit logs, not granular performance traces with timing breakdowns for internal agent steps. - D is Incorrect: VPC Flow Logs capture network traffic metadata, which cannot diagnose application-level orchestration latency within a Bedrock Agent. > Key Takeaway: To diagnose per-request latency in multi-step AI orchestrations (like Bedrock Agents), use AWS X-Ray combined with AgentCore Observability to trace the execution path of tool calls and model invocations.
### Question 5 - SageMaker Asynchronous Inference Queue Monitoring A media processing company uses an Amazon SageMaker Asynchronous Inference endpoint to process large video files. The workload is highly sporadic, sometimes receiving hundreds of files in a few minutes. The team must set up a dashboard that alerts them if the queue of pending inference requests grows too large, indicating the endpoint needs to scale out. Which metric is the MOST appropriate to display on this dashboard? Select ONE: - A) `Invocations` - B) `ApproximateBacklogSize` - C) `ModelLatency` - D) `DiskUtilization`	Correct Answer: B Correct Answer Analysis: - B is Correct: For SageMaker Asynchronous Inference endpoints, requests are placed in an Amazon SQS queue. The `ApproximateBacklogSize` metric measures the number of requests waiting in the queue. Monitoring this metric is essential for configuring auto-scaling policies and alerting when the endpoint cannot keep up with incoming traffic. Distractor Analysis: - A is Incorrect: `Invocations` tracks the number of completed or processing invocations, not the pending backlog in the queue. - C is Incorrect: `ModelLatency` measures the time it takes the model to process a single request, not the queue depth. - D is Incorrect: `DiskUtilization` tracks storage on the instance and does not provide visibility into the pending request queue size. > Key Takeaway: For Asynchronous Inference endpoints, auto-scaling and queue monitoring rely on the `ApproximateBacklogSize` metric to measure the pending work in the underlying SQS queue.
### Question 6 - Monitoring OpenSearch Serverless for RAG An AI team has built a Retrieval-Augmented Generation (RAG) pipeline using Amazon Bedrock Knowledge Bases backed by an Amazon OpenSearch Serverless collection. They need to monitor the vector database performance to ensure retrieval latency is not degrading the chatbot experience. The solution must focus on search-specific performance metrics without provisioning unnecessary infrastructure. Which metric should the team monitor in CloudWatch? Select ONE: - A) `IndexingLatency` - B) `SearchLatency` - C) `IngestionBytes` - D) `ResponseTime`	Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon OpenSearch Serverless emits the `SearchLatency` metric in CloudWatch, which specifically measures the time it takes for the vector collection to execute a search query. Monitoring this directly correlates to the retrieval step's impact on the RAG pipeline's overall latency. Distractor Analysis: - A is Incorrect: `IndexingLatency` measures the time taken to index documents (the ingestion phase), not the search query performance which impacts user-facing latency. - C is Incorrect: `IngestionBytes` measures the volume of data ingested, not performance latency. - D is Incorrect: `ResponseTime` is a generic metric and not a standard OpenSearch Serverless CloudWatch metric for measuring search query execution time. > Key Takeaway: When monitoring RAG vector databases on OpenSearch Serverless, use the `SearchLatency` CloudWatch metric to track the performance impact of the retrieval step on your application.
### Question 7 - Custom Metrics for PII Redaction Latency A compliance team requires all prompts sent to an Amazon Bedrock model to pass through an AWS Lambda function that redacts Personally Identifiable Information (PII) using Amazon Comprehend. The ML engineering team needs to monitor the exact time spent on this redaction process per request to ensure it stays under 500ms. How can the team visualize this specific latency metric? Select ONE: - A) Monitor the `Duration` metric of the Lambda function in CloudWatch. - B) Enable AWS X-Ray active tracing on the Lambda function and view the service map. - C) Use the `OverheadLatency` metric from the Amazon Bedrock CloudWatch namespace. - D) Publish a custom CloudWatch metric from within the Lambda function calculating the Comprehend API call duration.	Correct Answer: D Correct Answer Analysis: - D is Correct: Since the requirement is to monitor the exact time spent on the Comprehend redaction step (not the total Lambda execution time), the best practice is to calculate this duration in the code and publish it as a custom metric to CloudWatch using the `PutMetricData` API or CloudWatch Embedded Metric Format (EMF). Distractor Analysis: - A is Incorrect: The Lambda `Duration` metric includes cold starts, initialization, and the time taken to call Bedrock. It does not isolate the Amazon Comprehend redaction latency. - B is Incorrect: X-Ray traces provide visibility into downstream calls, but creating a precise dashboard for a specific sub-execution latency across thousands of requests is better handled by extracting that duration into a custom metric. - C is Incorrect: Bedrock's `OverheadLatency` measures Bedrock's internal framework overhead, not external Lambda/Comprehend processing time. > Key Takeaway: When you need to monitor the latency of a specific sub-step within a Lambda function (like a single downstream API call), publish a custom CloudWatch metric using the Embedded Metric Format (EMF).
### Question 8 - Correlating SageMaker Endpoint Traffic and Utilization A SageMaker real-time endpoint is experiencing intermittent throttling errors. The ML engineer suspects that the auto-scaling policy is not reacting fast enough to traffic bursts. To troubleshoot this, they need to visualize the number of requests hitting the endpoint against the capacity of the endpoint over the same time window. Which two CloudWatch metrics should be plotted together on a dashboard to diagnose this issue? Select TWO: - A) `Invocations` - B) `InvocationsPerInstance` - C) `ModelLatency` - D) `CPUUtilization` - E) `Invocation4XXErrors`	Correct Answers: B and D Correct Answer Analysis: - B is Correct: `InvocationsPerInstance` shows the actual traffic load being handled by each instance, which is the metric typically used to drive auto-scaling target tracking policies. - D is Correct: `CPUUtilization` shows the compute capacity usage. Plotting `InvocationsPerInstance` alongside `CPUUtilization` reveals if the instances are maxing out their CPU before the auto-scaling policy adds new instances, leading to throttling. Distractor Analysis: - A is Incorrect: `Invocations` shows the total traffic to the endpoint, but does not account for the number of instances currently running, making it harder to evaluate per-instance capacity pressure. - C is Incorrect: `ModelLatency` measures model processing time, not the traffic-to-capacity ratio that causes throttling. - E is Incorrect: While `Invocation4XXErrors` indicates that throttling is happening, it does not help diagnose the capacity bottleneck relative to incoming traffic. > Key Takeaway: To diagnose auto-scaling lag and throttling on SageMaker endpoints, correlate `InvocationsPerInstance` (load) against `CPUUtilization` (capacity) to understand the scaling behavior.
### Question 9 - Unified Dashboard for Hybrid AI Architecture An application uses an Amazon SageMaker Asynchronous endpoint for batch video processing and an Amazon Bedrock real-time model for interactive chat. The operations team requires a single pane of glass to monitor the `ModelLatency` of the Bedrock model and the `ApproximateBacklogSize` of the SageMaker endpoint. What is the MOST operationally efficient way to provide this view? Select ONE: - A) Create an AWS Cost Explorer report grouped by service and filter for SageMaker and Bedrock. - B) Configure a single Amazon CloudWatch dashboard with widgets pulling from both the `AWS/SageMaker` and `AWS/Bedrock` namespaces. - C) Export CloudWatch metrics to Amazon S3 and use Amazon QuickSight to build a unified BI dashboard. - D) Use Amazon Managed Grafana to query both Prometheus endpoints and build a custom panel.	Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon CloudWatch dashboards are designed to be cross-service. You can create widgets for metrics from different AWS namespaces (e.g., `AWS/Bedrock` for `ModelLatency` and `AWS/SageMaker` for `ApproximateBacklogSize`) on the same dashboard, providing a single pane of glass natively without additional tooling. Distractor Analysis: - A is Incorrect: AWS Cost Explorer is for financial cost tracking, not operational performance metrics like latency or queue size. - C is Incorrect: Exporting to S3 and using QuickSight introduces significant latency, ongoing ETL maintenance, and unnecessary cost for operational monitoring that CloudWatch already provides natively. - D is Incorrect: Amazon Managed Grafana is a powerful tool but introduces unnecessary operational overhead compared to native CloudWatch dashboards for simply viewing two standard AWS metrics. > Key Takeaway: Amazon CloudWatch dashboards support cross-service metric visualization natively, making them the ideal, low-overhead choice for monitoring hybrid AWS architectures.
### Question 10 - Setting Alarms on Bedrock Provisioned Throughput A company has purchased Amazon Bedrock Provisioned Throughput to handle a steady baseline of inference requests. They want to ensure they are fully utilizing their provisioned model units. The team needs a dashboard metric that shows the percentage of provisioned capacity being utilized. Which CloudWatch metric should they use to monitor this utilization? Select ONE: - A) `ProvisionedThroughputUtilization` - B) `InvocationsPerInstance` - C) `TokenUtilization` - D) `InputTokenCount`	Correct Answer: A Correct Answer Analysis: - A is Correct: The `ProvisionedThroughputUtilization` metric in the `AWS/Bedrock` namespace tracks the utilization of the provisioned model units. This allows the team to see if they are maxing out their purchased baseline or if they are over-provisioned. Distractor Analysis: - B is Incorrect: `InvocationsPerInstance` is a SageMaker metric, not applicable to serverless Bedrock Provisioned Throughput. - C is Incorrect: While token usage is related, `TokenUtilization` is not the specific metric used to measure capacity utilization against purchased provisioned throughput units. - D is Incorrect: `InputTokenCount` measures the absolute volume of input tokens, not the percentage of provisioned capacity being utilized. > Key Takeaway: To monitor if you are maximizing your Amazon Bedrock Provisioned Throughput commitment, use the `ProvisionedThroughputUtilization` metric in CloudWatch.