Skill 4.1.1: Monitor model performance in production by using Amazon CloudWatch generative AI observability, Amazon Bedrock Model Evaluation, and drift detection pipelines.
| Question | Answer & Explanation |
|---|---|
| ### Question 1 - CloudWatch GenAI Observability vs. Standard CloudWatch An AI engineering team has deployed a generative AI assistant using Amazon Bedrock Agents that orchestrates multiple tool calls. They need a single, consolidated view to monitor token usage, request latency, error rates, and end-to-end prompt execution traces. Constraint: The solution must use native AWS features with the lowest operational overhead and avoid writing custom parsing logic. Which solution meets these requirements? (Select ONE) - A) Enable detailed Amazon CloudWatch logging and use CloudWatch Logs Insights to manually parse JSON payloads for token counts and latency. - B) Deploy a third-party observability tool via AWS Marketplace to capture distributed traces. - C) Enable Amazon CloudWatch generative AI observability to aggregate model invocation metrics and prompt traces automatically. - D) Implement AWS X-Ray tracing manually on the Bedrock Agent orchestration layer to capture tool invocations. |
Correct Answer: C Correct Answer Analysis: - C is Correct: Amazon CloudWatch generative AI observability provides pre-built dashboards specifically designed for LLMs and agentic workloads. It automatically aggregates model-level metrics (token consumption, latency, errors, throttling) alongside distributed X-Ray prompt traces without requiring custom code. Distractor Analysis: - A is Incorrect: Using CloudWatch Logs Insights to manually parse logs introduces unnecessary operational overhead and does not provide built-in, end-to-end distributed prompt tracing. - B is Incorrect: Deploying a third-party tool from the Marketplace introduces higher operational overhead, cost, and integration complexity compared to the native CloudWatch generative AI observability feature. - D is Incorrect: Implementing X-Ray manually on the Bedrock Agent orchestration layer requires custom instrumentation and coding, violating the lowest operational overhead constraint. > Exam Tip / AWS Best Practice: CloudWatch generative AI observability provides out-of-the-box dashboards for LLMs and agents, eliminating the need for custom tracing code. |
| ### Question 2 - Drift Detection Pipelines using SageMaker Model Monitor A retail company operates a product recommendation model on an Amazon SageMaker endpoint. The data science team wants to establish an automated drift detection pipeline to capture a baseline of the input data distribution and alert if current production traffic deviates significantly. Constraint: The solution must be fully automated and use native AWS services to minimize operational maintenance. Which approach fulfills these requirements? (Select ONE) - A) Create an AWS Lambda function triggered by CloudWatch Events that periodically samples endpoint traffic and computes the Kolmogorov-Smirnov statistic manually. - B) Enable Amazon SageMaker Model Monitor with a baseline constraint file generated from a stable training dataset, and configure CloudWatch Alarms for violations. - C) Use AWS Glue Data Quality rules to inspect the training dataset and manually compare it to production logs on a weekly basis. - D) Rely on CloudWatch Anomaly Detection on the Invocations metric to detect changes in traffic volume. |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon SageMaker Model Monitor automatically captures baseline statistics and constraints from a training dataset, continuously monitors live endpoint traffic, and emits metrics to CloudWatch when data drift is detected. It natively supports alerting via CloudWatch Alarms. Distractor Analysis: - A is Incorrect: Building a custom Lambda function to calculate statistics manually introduces significant operational overhead and requires custom maintenance, violating the constraints. - C is Incorrect: Manually comparing Glue Data Quality output to production logs on a weekly basis is not an automated, continuous pipeline. - D is Incorrect: CloudWatch Anomaly Detection on the Invocations metric only detects changes in the volume of traffic, not changes in the statistical distribution of the input features (data drift).> Exam Tip / AWS Best Practice: SageMaker Model Monitor is the native, automated solution for creating drift detection pipelines, capturing both baseline and live data distributions to compare. |
| ### Question 3 - Concept Drift vs. Data Drift An enterprise's customer support assistant receives prompts that are statistically close to the recorded training baseline. However, the business notices a steady decline in user satisfaction scores because user expectations of what constitutes a "helpful" response have evolved, and the previous correct answers are no longer preferred. Constraint: The team must accurately classify this degradation to choose the correct remediation strategy. Which condition is the application experiencing? (Select ONE) - A) Data drift - B) Concept drift - C) Model throttling - D) Tool invocation failure |
Correct Answer: B Correct Answer Analysis: - B is Correct: Concept drift occurs when the relationship between the input and the expected output ($P(Y |
| ### Question 4 - Amazon Bedrock Model Evaluation (Automatic vs. Human) A data science team needs to evaluate a newly fine-tuned foundation model before deploying it to production. They want to assess the model on correctness, toxicity, and factual accuracy. The team requires immediate results and explicitly states they have no budget or time to hire human reviewers for this iteration. Constraint: The solution must use Amazon Bedrock Model Evaluation with the fastest turnaround time and no human reviewers. Which evaluation method should they use? (Select ONE) - A) Bedrock Model Evaluation - Human - B) Bedrock Model Evaluation - Automatic - C) Bedrock Guardrails content filtering - D) CloudWatch generative AI observability metric analysis |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock Model Evaluation Automatic jobs use an LLM-as-a-judge (judge model) to score model responses on metrics like correctness, toxicity, and factual accuracy. This provides fast, automated results without the need for human reviewers. Distractor Analysis: - A is Incorrect: Human evaluation jobs require sending prompts and responses to human reviewers, which takes significant time and budget, violating the constraints. - C is Incorrect: Bedrock Guardrails are used at inference time to block harmful content; they do not evaluate model correctness or accuracy across an evaluation dataset. - D is Incorrect: CloudWatch GenAI observability tracks operational telemetry (latency, token usage) but does not perform semantic evaluation of response quality or accuracy. > Exam Tip / AWS Best Practice: Use Bedrock Automatic Model Evaluation (LLM-as-a-judge) for rapid, code-free assessment of model quality, and Human Evaluation for nuanced, subjective human-in-the-loop scoring. |
| ### Question 5 - A/B Testing with SageMaker Production Variants A machine learning team wants to introduce a new model version to live traffic. They need the new version to serve exactly 10% of live inference requests while the existing production version serves the remaining 90%. Both versions must be hosted on the same endpoint to simplify deployment and be evaluated on the same CloudWatch metrics. Constraint: Use the most cost-effective and native SageMaker feature for traffic distribution. Which configuration supports this requirement? (Select ONE) - A) Create two separate SageMaker endpoints and use Amazon Route 53 weighted routing to distribute traffic. - B) Deploy both models as production variants on a single SageMaker endpoint with a 1:9 weight ratio. - C) Use a SageMaker Shadow variant with a 10% weight ratio. - D) Implement an AWS Lambda function in front of the endpoint to randomly route 10% of traffic to the new model. |
Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker Endpoints support multiple production variants. By assigning a weight of 1 to the new model and 9 to the existing model (a 1:9 ratio), SageMaker natively distributes 10% of the live traffic to the new variant, allowing direct comparison via CloudWatch metrics. Distractor Analysis: - A is Incorrect: Creating two separate endpoints and using Route 53 increases operational overhead and cost, and complicates metric aggregation compared to a single endpoint with variants. - C is Incorrect: A Shadow variant receives a copy of the traffic, but its responses are not returned to the user. The requirement is to serve 10% of live requests to users, not to test in shadow mode. - D is Incorrect: Using a Lambda function for routing introduces unnecessary custom code, latency, and operational overhead. > Exam Tip / AWS Best Practice: Use SageMaker Production Variants with assigned weights for native A/B testing on a single endpoint, simplifying metric comparison and deployment. |
| ### Question 6 - Amazon Bedrock AgentCore Observability An enterprise is running complex, multi-agent workflows using Amazon Bedrock AgentCore. They need to detect specific failure modes such as agent coordination failures, truncated streaming responses, and tool invocation errors. Constraint: The solution must use the default, OpenTelemetry-compatible telemetry provided by AgentCore with the least operational overhead. Which approach fulfills these requirements? (Select ONE) - A) Manually wrap the agent's Python code with custom print statements and log them to Amazon S3. - B) Enable Amazon Bedrock AgentCore Observability to capture model inference calls, tool invocations, and memory operations automatically. - C) Enable AWS CloudTrail API logging to monitor the AgentCore API calls. - D) Use Amazon Macie to analyze the agent communications for sensitive data that might cause failures. |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock AgentCore Observability provides default, OpenTelemetry-compatible instrumentation for agents running on AgentCore Runtime. It automatically captures traces for model inference, tool invocations, and memory operations without requiring custom spans. Distractor Analysis: - A is Incorrect: Manually wrapping code with custom print statements is an anti-pattern that introduces high operational overhead and lacks proper tracing structure. - C is Incorrect: AWS CloudTrail logs API control plane operations for audit purposes but does not provide application-level telemetry like truncated streaming or tool invocation success/failure. - D is Incorrect: Amazon Macie is a data security service for discovering sensitive data in S3; it has no capability to monitor agent coordination or runtime failures. > Exam Tip / AWS Best Practice: AgentCore Observability offers built-in OpenTelemetry support, automatically tracing agent reasoning, tool calls, and memory without custom instrumentation. |
| ### Question 7 - Pipeline Anomaly Detection (Checking Output vs. Completion) A daily scheduled AWS Glue ETL job processes feature data and writes it to Amazon S3 for a SageMaker endpoint to use for batch inference. The job reports success, but model invocation errors spike immediately afterward. The ML engineer discovers that the Glue job outputted empty partitions due to an upstream API change. Constraint: Identify the monitoring gap that allowed this to happen. Which monitoring gap does this pattern reveal? (Select ONE) - A) The endpoint has too few production variants for the traffic volume. - B) The pipeline monitors job completion status without checking the output data quality. - C) The account has no cost allocation tags applied to the SageMaker workload. - D) The model was evaluated by an LLM-as-a-judge rather than human reviewers. |
Correct Answer: B Correct Answer Analysis: - B is Correct: The Glue job completed with a success exit code but produced empty partitions. Monitoring only checked job completion status rather than validating the actual output data quality (e.g., verifying record counts, schema integrity, or null constraints), leading to downstream inference failures. Distractor Analysis: - A is Incorrect: The number of production variants affects traffic distribution, not data-driven invocation failures caused by empty partitions. - C is Incorrect: Cost allocation tags are used for financial tracking and have no impact on runtime functionality or data quality. - D is Incorrect: The evaluation method (LLM-as-a-judge vs. human) pertains to model quality assessment, not runtime failures caused by malformed or empty data inputs. > Exam Tip / AWS Best Practice: Monitoring a pipeline must include data quality checks on the output of a job, not just the job's exit status, to prevent silent handoff failures. |
| ### Question 8 - SageMaker Shadow Variants for Safe Testing A healthcare AI startup is updating its medical summarization model. They need to test the new model's inference stack under real production load without exposing any real users to the new model's responses. They want to record the new model's responses for offline analysis while users continue to receive responses from the old model. Constraint: The solution must ensure zero user exposure to the new model's output and use native SageMaker features. Which configuration meets these requirements? (Select ONE) - A) Configure the new model as a Production variant with a 0% weight. - B) Configure the new model as a Shadow variant on the existing SageMaker endpoint. - C) Deploy the new model as an Asynchronous endpoint and trigger it via an EventBridge rule. - D) Run an Amazon Bedrock Model Evaluation job using the production endpoint's historical data. |
Correct Answer: B Correct Answer Analysis: - B is Correct: A Shadow variant in SageMaker receives a copy of the live traffic. It processes the requests and records the responses in CloudWatch/logs for offline analysis, but its responses are NOT returned to the user, ensuring zero user exposure to the new model. Distractor Analysis: - A is Incorrect: A Production variant with a 0% weight receives no traffic at all, meaning the team cannot test the inference stack under real production load. - C is Incorrect: An Asynchronous endpoint does not natively receive an automatic copy of live traffic from another endpoint without complex custom routing, which increases operational overhead. - D is Incorrect: Bedrock Model Evaluation is a point-in-time batch assessment, not a mechanism for testing the real-time serving stack under live load. > Exam Tip / AWS Best Practice: Use SageMaker Shadow Variants to test a new model's serving stack under real production load without returning the new model's responses to end users. |
| ### Question 9 - RAG Evaluation with Bedrock Model Evaluation A company has built a Retrieval-Augmented Generation (RAG) application using Amazon Bedrock Knowledge Bases. They need to measure end-to-end performance, specifically focusing on retrieval accuracy and response faithfulness, to ensure the application does not hallucinate. The team has no resources to write and maintain custom evaluation scripts. Constraint: The solution must use the most operationally efficient native AWS tool for evaluating RAG workloads. Which service should they use? (Select ONE) - A) Amazon CloudWatch generative AI observability - B) Amazon Bedrock Model Evaluation for Knowledge Bases - C) AWS Glue DataBrew - D) Amazon OpenSearch Serverless Dashboard |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock Model Evaluation includes built-in support for evaluating Knowledge Bases (RAG). It automatically assesses retrieval performance (e.g., relevance of retrieved chunks) and generation performance (e.g., faithfulness and correctness) without requiring custom evaluation scripts. Distractor Analysis: - A is Incorrect: CloudWatch GenAI observability tracks operational metrics like token count and latency, but it does not semantically evaluate RAG retrieval accuracy or response faithfulness. - C is Incorrect: AWS Glue DataBrew is a visual data preparation tool; it is not designed for evaluating LLM or RAG output quality. - D is Incorrect: OpenSearch Serverless Dashboards visualize the vector search index metrics, but they do not evaluate the semantic faithfulness or correctness of the final LLM response. > Exam Tip / AWS Best Practice: When evaluating a RAG pipeline, use Bedrock Model Evaluation to assess both retrieval quality and generation faithfulness natively. |
| ### Question 10 - Distance Metrics for High-Dimensional Drift An ML engineer is configuring Amazon SageMaker Model Monitor to detect data drift on a tabular dataset. The dataset contains a mix of categorical and numerical features. The engineer wants to use the default statistical tools provided by Model Monitor to establish the baseline and detect drift. Constraint: Rely on default SageMaker Model Monitor capabilities to minimize operational overhead. Which statistical tests does Model Monitor natively use by default to compare the baseline and current distributions? (Select TWO) - A) Kolmogorov-Smirnov (K-S) test for numerical features. - B) Chi-Squared test for categorical features. - C) Wasserstein distance for high-dimensional embeddings. - D) Population Stability Index (PSI) for all features. |
Correct Answers: A and B Correct Answer Analysis: - A is Correct: By default, SageMaker Model Monitor uses the Kolmogorov-Smirnov (K-S) test to compare the distributions of numerical features between the baseline and current datasets. - B is Correct: For categorical features, Model Monitor natively uses the Chi-Squared test to determine if the distribution of categories has shifted from the baseline. Distractor Analysis: - C is Incorrect: While Wasserstein distance is a valid metric for high-dimensional spaces like embeddings, it is not the default metric used out-of-the-box by SageMaker Model Monitor for standard tabular data. - D is Incorrect: Population Stability Index (PSI) is a common drift metric in finance and industry, but SageMaker Model Monitor defaults to K-S and Chi-Squared, not PSI, for its built-in constraints. > Exam Tip / AWS Best Practice: SageMaker Model Monitor handles tabular data drift out-of-the-box using the K-S test for numerical data and the Chi-Squared test for categorical data. |