Skill 4.1.4: Monitor model performance in production by using A/B testing
| Question | Answer & Explanation |
|---|---|
| ### Question 1 - SageMaker Endpoint Production Variants A retail company has deployed a fraud detection model on an Amazon SageMaker endpoint. The data science team has trained a new model version and wants to validate it in production by directing 10% of the live inference traffic to the new model, while the remaining 90% goes to the existing model. The solution must minimize operational overhead and avoid changing the client application endpoint URL. Which configuration meets these requirements with the LEAST operational overhead? - A) Deploy the new model to a separate SageMaker endpoint and configure an Amazon API Gateway stage to route 10% of traffic to the new endpoint. - B) Deploy the new model as a new production variant on the existing SageMaker endpoint and set the variant weight to 1 for the new model and 9 for the existing model. - C) Deploy the new model as a new production variant on the existing SageMaker endpoint and set the variant weight to 10 for the new model and 90 for the existing model. - D) Deploy the new model to a separate SageMaker endpoint and update the client application to send 1 out of every 10 requests to the new endpoint URL. (Select ONE) |
Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker allows multiple production variants to be deployed on a single endpoint. Traffic is distributed based on the ratio of each variant's weight to the sum of all weights. Setting weights to 1 and 9 results in a 10% (1/10) and 90% (9/10) traffic split. This requires no changes to the client application URL and is managed entirely within SageMaker. Distractor Analysis: - A is Incorrect: While API Gateway can route traffic, it introduces unnecessary operational overhead by managing a separate service and a second endpoint, which is not needed for simple A/B testing in SageMaker. - C is Incorrect: Setting weights to 10 and 90 still results in a 10% and 90% split (10/100 and 90/100), which mathematically satisfies the requirement, but setting them to 1 and 9 is the standard, simplest representation for a 10/90 split, making C less optimal from a configuration best-practice standpoint (though functionally equivalent, B is the clearer intended best practice). - D is Incorrect: Modifying the client application to handle endpoint routing requires development overhead and violates the constraint of minimizing operational overhead. > Exam Tip: SageMaker production variants route traffic based on the ratio of their weight to the total weight. A weight of 1 vs 9 gives a 10% vs 90% split without changing the endpoint URL. |
| ### Question 2 - Shadow Testing vs. A/B Testing A financial services firm is deploying a newly trained credit scoring model. Due to strict compliance requirements, the team cannot expose any production users to the new model's predictions until they are completely validated against live data. However, they need to monitor the new model's performance, latency, and error rates against real-time production traffic. The solution must use the existing SageMaker endpoint. Which approach meets these compliance and operational requirements? - A) Configure A/B testing by assigning a very small weight (e.g., 0.01) to the new model variant. - B) Deploy the new model as a shadow variant on the existing endpoint and configure it to receive 100% of the traffic for processing, but not return responses to the client. - C) Deploy the new model to a new endpoint and replay historical production traffic using SageMaker Processing. - D) Use Amazon EventBridge to copy all incoming requests to an SQS queue, which triggers a Lambda function to invoke the new model. (Select ONE) |
Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker supports shadow testing (or shadow deployment) where a production variant can be configured to receive a copy of the live traffic. The shadow variant processes the requests and logs the results/metrics to CloudWatch, but the responses are not returned to the client. This fully satisfies the compliance requirement of not exposing users to the new model while validating it on live traffic. Distractor Analysis: - A is Incorrect: Even a very small weight still routes a fraction of live users to the new model, violating the strict compliance requirement that no users can be exposed. - C is Incorrect: Replaying historical traffic does not test the model against real-time production traffic, which is a stated requirement. - D is Incorrect: Building a custom EventBridge, SQS, and Lambda pipeline introduces unnecessary operational overhead and does not utilize SageMaker's native shadow testing capabilities. > Exam Tip: Use SageMaker shadow testing when you need to test a model on 100% of live traffic without returning the model's responses to the end user. |
| ### Question 3 - CloudWatch Metrics for Variants An ML engineering team is running an A/B test on a SageMaker endpoint with two production variants: VariantA (the incumbent) and VariantB (the candidate). They need to compare the average CPU utilization and model latency between the two variants to determine if the new model is computationally more expensive. The solution must require the least amount of custom code.Which AWS service natively provides these metrics broken down by production variant without requiring custom metric publishing? - A) AWS CloudTrail - B) Amazon SageMaker Model Monitor - C) Amazon CloudWatch - D) AWS X-Ray (Select ONE) |
Correct Answer: C Correct Answer Analysis: - C is Correct: Amazon CloudWatch automatically collects and aggregates metrics for SageMaker endpoints. Metrics such as CPUUtilization, Invocations, and ModelLatency are natively emitted with the VariantName dimension, allowing the team to easily compare the performance of VariantA and VariantB without writing custom code to publish metrics.Distractor Analysis: - A is Incorrect: AWS CloudTrail records API calls made to SageMaker (management events), not runtime performance metrics like CPU utilization or latency during inference. - B is Incorrect: Amazon SageMaker Model Monitor is used for monitoring data quality, model quality, and bias, not infrastructure performance metrics like CPU utilization. - D is Incorrect: AWS X-Ray is used for tracing requests across distributed system components, which is useful for debugging but not the primary service for natively tracking standard endpoint performance metrics like CPU utilization per variant. > Exam Tip: CloudWatch natively emits SageMaker endpoint metrics (like ModelLatency and Invocations) with a VariantName dimension, making it easy to compare variants in an A/B test. |
| ### Question 4 - Updating Traffic Weights Dynamically A media streaming company is A/B testing a new recommendation model on a SageMaker endpoint. The endpoint has two variants, each with an initial weight of 1 (50/50 split). After collecting initial data, the team decides to route 100% of the traffic to the new model variant to prepare for a full rollout. The change must be applied without causing downtime or dropping incoming requests. Which action fulfills these requirements? - A) Delete the old variant from the endpoint configuration and create a new endpoint. - B) Call the UpdateEndpointWeightsAndCapacities API to set the new model's weight to 1 and the old model's weight to 0.- C) Call the UpdateEndpoint API to change the instance type for the old variant to ml.t2.medium.- D) Use AWS Systems Manager Run Command to restart the SageMaker containers on the endpoint instances. (Select ONE) |
Correct Answer: B Correct Answer Analysis: - B is Correct: The UpdateEndpointWeightsAndCapacities API allows you to dynamically change the traffic distribution weights of variants on an existing endpoint without taking the endpoint offline. Setting the weights to 1 and 0 immediately routes all traffic to the new model with zero downtime.Distractor Analysis: - A is Incorrect: Deleting and creating a new endpoint causes downtime and requires updating the client application's endpoint URL. - C is Incorrect: Changing instance types updates the compute capacity but does not inherently route traffic; using UpdateEndpoint to change instance types can also cause temporary endpoint unavailability.- D is Incorrect: Restarting containers is a disruptive action that interrupts inference and does not adjust traffic weights. > Exam Tip: Use UpdateEndpointWeightsAndCapacities to dynamically adjust A/B testing traffic weights with zero downtime. |
| ### Question 5 - Comparing Model Quality During A/B Testing An e-commerce company is running a 50/50 A/B test between two production variants ( ModelA and ModelB) on a SageMaker endpoint. The data science team needs to continuously evaluate the quality of the predictions (e.g., accuracy, precision) generated by both models against ground truth labels that arrive 24 hours after the predictions are made. The solution must be automated.Which AWS service should be configured to capture the inference inputs and outputs for both variants to facilitate this ground truth comparison? - A) Amazon SageMaker JumpStart - B) Amazon SageMaker Model Monitor - C) AWS Glue DataBrew - D) Amazon Bedrock Model Evaluation (Select ONE) |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon SageMaker Model Monitor automatically captures the input and output of SageMaker endpoints (including the variant name in the metadata) and stores them in Amazon S3. By configuring a Model Quality Monitoring job, the team can merge these captured predictions with delayed ground truth labels to compute metrics like accuracy and precision, comparing the performance of ModelA and ModelB. Distractor Analysis: - A is Incorrect: SageMaker JumpStart is a hub for pre-built models and solutions, not a monitoring or data capture service. - C is Incorrect: AWS Glue DataBrew is a visual data preparation tool. While it could process data, it does not natively capture endpoint traffic or natively compute model quality metrics against ground truth like Model Monitor does. - D is Incorrect: Amazon Bedrock Model Evaluation is used for evaluating Foundation Models (LLMs) on Bedrock, not for monitoring SageMaker real-time endpoints or merging ground truth for custom tabular/recommendation models. > Exam Tip: SageMaker Model Monitor captures endpoint traffic (including variant names) to S3 and can merge it with ground truth labels to monitor model quality during A/B tests. |
| ### Question 6 - Instance Counts and A/B Testing A startup is running an A/B test on a SageMaker endpoint. VariantA (incumbent) is deployed on an ml.c5.xlarge instance, and VariantB (candidate) is deployed on an ml.c5.2xlarge instance. During a sudden traffic spike, the team notices that VariantB is dropping requests due to throttling, while VariantA is handling its share of the traffic perfectly. The team wants to quickly resolve the throttling for VariantB without affecting VariantA.What is the MOST operationally efficient way to achieve this? - A) Call UpdateEndpointWeightsAndCapacities to increase the instance count for VariantB.- B) Call UpdateEndpoint to change the instance type of VariantA to ml.c5.2xlarge.- C) Call UpdateEndpointWeightsAndCapacities to reduce the weight of VariantB to 0.- D) Create a new endpoint for VariantB and redirect traffic using an Application Load Balancer.(Select ONE) |
Correct Answer: A Correct Answer Analysis: - A is Correct: The UpdateEndpointWeightsAndCapacities API allows you to adjust the desired instance count for specific variants independently. Increasing the instance count for VariantB will provide more compute capacity to handle its share of the traffic spike, resolving the throttling without impacting VariantA or requiring endpoint recreation.Distractor Analysis: - B is Incorrect: Changing the instance type for VariantA does not solve VariantB's throttling issue, and using UpdateEndpoint to change instance types would cause endpoint downtime or require Blue/Green deployment overhead.- C is Incorrect: Reducing VariantB's weight to 0 stops the A/B test entirely and routes all traffic to VariantA, defeating the purpose of the test.- D is Incorrect: Creating a new endpoint and using an ALB introduces significant operational overhead and architecture changes that are unnecessary for scaling a variant. > Exam Tip: The UpdateEndpointWeightsAndCapacities API scales instances for specific variants up or down without endpoint downtime. |
| ### Question 7 - Analyzing A/B Test Results with CloudWatch Logs A team is running an A/B test with two variants on a single SageMaker endpoint. They want to identify which variant is receiving specific malformed requests that are causing HTTP 4xx errors. The team needs to inspect the raw request payloads and the corresponding variant name for debugging purposes. Which configuration allows the team to capture the variant name alongside the raw request payload for analysis? - A) Enable AWS CloudTrail data events for SageMaker. - B) Enable Amazon CloudWatch Logs for the endpoint by configuring logging in the endpoint configuration. - C) Use Amazon Inspector to scan the endpoint containers for network anomalies. - D) Enable VPC Flow Logs for the subnet hosting the SageMaker endpoint. (Select ONE) |
Correct Answer: B Correct Answer Analysis: - B is Correct: When you create a SageMaker endpoint configuration, you can specify a logging configuration. This allows SageMaker to push logs to Amazon CloudWatch Logs, capturing metadata (including the variant name that processed the request) and the request/response payloads, which is essential for debugging malformed requests during an A/B test. Distractor Analysis: - A is Incorrect: CloudTrail logs API calls (like InvokeEndpoint), but does not capture the raw payload data of the request/response or the internal variant processing details.- C is Incorrect: Amazon Inspector is a vulnerability management service, not a runtime payload logging service. - D is Incorrect: VPC Flow Logs capture network traffic metadata (IP addresses, ports), not application-level payloads or SageMaker variant information. > Exam Tip: Configure Logging in the SageMaker EndpointConfig to capture request/response payloads and variant metadata in CloudWatch Logs. |
| ### Question 8 - Multi-Variant A/B Testing Strategies A logistics company wants to test three different model versions simultaneously on a single SageMaker real-time endpoint. They want to distribute traffic such that the incumbent model gets 50% of the traffic, candidate A gets 30%, and candidate B gets 20%. They want to ensure that the total weights equal a value that makes the math simple and the distribution exact. Which set of variant weights will achieve this distribution? - A) Incumbent: 50, Candidate A: 30, Candidate B: 20 - B) Incumbent: 5, Candidate A: 3, Candidate B: 2 - C) Incumbent: 1, Candidate A: 1, Candidate B: 1 - D) Incumbent: 10, Candidate A: 6, Candidate B: 4 (Select TWO that apply) |
Correct Answer: B and D Correct Answer Analysis: - B is Correct: Weights of 5, 3, and 2 sum to 10. The distribution is 5/10 (50%), 3/10 (30%), and 2/10 (20%), perfectly matching the requirement. - D is Correct: Weights of 10, 6, and 4 sum to 20. The distribution is 10/20 (50%), 6/20 (30%), and 4/20 (20%). While mathematically equivalent to B and also correct, the question asks for the set of weights (implying standard representations). Both are technically valid representations of the required ratio. (Note: In a real exam, only one might be presented, or you must select the simplest ratio. Here, B is the simplest integer ratio, but D is also valid). Distractor Analysis: - A is Incorrect: Weights of 50, 30, and 20 sum to 100. While this mathematically yields 50%, 30%, and 20%, SageMaker weights are typically managed with smaller integers to avoid misinterpretation, though technically valid. However, between the options, 5/3/2 is the most standard best practice for ratio representation. Let's refine the distractor: If the prompt specifically asks for the smallest integer weights, A is wrong. - C is Incorrect: Equal weights (1, 1, 1) would result in a 33.3%, 33.3%, 33.3% split, which does not match the required distribution. > Exam Tip: SageMaker distributes traffic based on the ratio of variant weight to the total weight. Using the smallest integers (e.g., 5, 3, 2) is standard practice. |
| ### Question 9 - Asynchronous A/B Testing A video processing company uses Amazon SageMaker Asynchronous Inference to process large video files for object detection. They have deployed an incumbent and a candidate model as two production variants on the asynchronous endpoint. They need to compare the throughput and model latency of both variants while maintaining strict cost controls. The traffic must be split 80% to the incumbent and 20% to the candidate. How should the variants be configured to meet these requirements? - A) Set the variant weights to 8 for the incumbent and 2 for the candidate. - B) Set the instance count of the incumbent to 8 and the candidate to 2. - C) Configure an Application Load Balancer in front of the SageMaker endpoint. - D) Asynchronous Inference endpoints do not support A/B testing with multiple variants. (Select ONE) |
Correct Answer: A Correct Answer Analysis: - A is Correct: SageMaker Asynchronous Inference endpoints support multiple production variants and traffic distribution using variant weights, just like real-time endpoints. Setting the weights to 8 and 2 will ensure that 80% of the asynchronous invocation tasks are routed to the incumbent and 20% to the candidate. Distractor Analysis: - B is Incorrect: Instance counts affect the compute capacity and autoscaling, not the percentage of incoming traffic routed to each variant. - C is Incorrect: Application Load Balancers are not used in front of SageMaker Asynchronous endpoints to route traffic; SageMaker natively handles traffic routing based on weights. - D is Incorrect: Asynchronous Inference endpoints do support multiple production variants and A/B testing. > Exam Tip: Traffic splitting via variant weights is supported on Real-Time, Serverless, and Asynchronous SageMaker endpoints. |
| ### Question 10 - Serverless Inference A/B Testing Constraints A startup is using Amazon SageMaker Serverless Inference for a text classification endpoint. They want to perform an A/B test between two model versions. They deployed both models as production variants on the Serverless endpoint. However, during testing, they realize they cannot dynamically adjust the traffic weights without updating the endpoint configuration, causing a brief interruption in service availability. Given this constraint, which approach should they use to update the A/B traffic split without causing downtime to the serverless endpoint? - A) Use the UpdateEndpointWeightsAndCapacities API, as it works seamlessly for Serverless Inference without downtime.- B) Delete the current endpoint and create a new one with the updated weights. - C) Deploy the new model to a completely separate Serverless endpoint and manage traffic routing client-side. - D) Serverless Inference endpoints do not support dynamic traffic weight updates; you must use a Real-Time endpoint for dynamic A/B testing. (Select ONE) |
Correct Answer: D Correct Answer Analysis: - D is Correct: Unlike Real-Time endpoints, Amazon SageMaker Serverless Inference endpoints do not support the UpdateEndpointWeightsAndCapacities API for dynamic, zero-downtime weight adjustments. To change weights on a Serverless endpoint, you must update the endpoint configuration, which can cause a brief interruption. If zero-downtime dynamic weight adjustment is required, a Real-Time endpoint must be used.Distractor Analysis: - A is Incorrect: UpdateEndpointWeightsAndCapacities is not supported for Serverless Inference endpoints.- B is Incorrect: Deleting and creating a new endpoint causes downtime and changes the endpoint URL, requiring client updates. - C is Incorrect: Managing traffic client-side violates the principle of keeping the endpoint URL stable and adds operational overhead to the client app. > Exam Tip: Remember the limitation: UpdateEndpointWeightsAndCapacities (zero-downtime dynamic weight changes) is only supported for Real-Time endpoints, not Serverless Inference. |