Skill 2.3.3: Compare the performance of shadow variants to production variants
| Question | Answer & Explanation |
|---|---|
| ### Question 1 - Safe Validation of a New XGBoost Model An enterprise retail company is updating its recommendation engine from an older XGBoost model to a newly trained version. The engineering team must validate the new model's performance against real-time production traffic to ensure it can handle current payload sizes and request concurrency. Compliance policies strictly mandate that no customer can receive a response from the new model until it is fully approved, and the solution must introduce the least operational overhead. Which approach satisfies these requirements? - A) Deploy the new model as a shadow variant on the existing SageMaker endpoint, routing a copy of live traffic to it while production continues to serve responses. - B) Configure an A/B test by splitting traffic 50/50 between the production model and the new model on the SageMaker endpoint. - C) Use Amazon Bedrock Evaluations to run a programmatic evaluation job comparing both models against a stored historical dataset. - D) Deploy the new model to a separate SageMaker endpoint and manually route a portion of API Gateway traffic to it using a weighted distribution. |
Correct Answer: A Correct Answer Analysis: - A is Correct: SageMaker AI shadow testing natively deploys a candidate model alongside a production variant on the same endpoint. It asynchronously routes a copy of live requests to the shadow variant without returning the shadow variant's responses to the caller, satisfying the compliance constraint with zero impact to users. Distractor Analysis: - B is Incorrect: A/B testing routes live traffic to the candidate model and returns the candidate's responses to a percentage of users, violating the compliance mandate. - C is Incorrect: Bedrock Evaluations (or any offline programmatic evaluation) relies on stored datasets rather than real-time live traffic, failing to test against current payload sizes and concurrency. - D is Incorrect: Deploying a separate endpoint and manually configuring API Gateway routing introduces unnecessary operational overhead and requires custom traffic mirroring logic. > Key Takeaway: Shadow testing validates a candidate model on live traffic with zero impact to the end-user experience by returning only production responses. |
| ### Question 2 - Validating Infrastructure and Container Updates A machine learning team has updated the Docker container serving their PyTorch model and upgraded the underlying SageMaker endpoint instance type to lower operational costs. They need to verify that this new configuration can handle the specific request shapes and payload volumes currently hitting their production endpoint. The validation must occur under live load, but the current production responses must remain undisturbed. Which AWS service feature allows the team to meet these constraints? - A) Amazon SageMaker Model Monitor - B) Amazon SageMaker AI Shadow Testing - C) Amazon Bedrock Prompt Management - D) Amazon SageMaker Experiments with MLflow (Select ONE) |
Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker AI Shadow Testing is specifically designed to validate changes to the model, the container, or the instance type against real request patterns. It routes a copy of live traffic to the shadow variant without returning the shadow responses to the caller, allowing the team to validate the new container and instance under real load. Distractor Analysis: - A is Incorrect: SageMaker Model Monitor monitors the quality of incoming data and model predictions post-deployment, but it does not route live traffic to an undeployed candidate variant for testing. - C is Incorrect: Bedrock Prompt Management is used for versioning and comparing prompt text templates for LLMs, not for validating Docker containers or endpoint instance types. - D is Incorrect: MLflow tracks training runs, parameters, and artifacts, but does not handle live endpoint traffic mirroring. > Key Takeaway: Use SageMaker Shadow Testing when you need to evaluate endpoint configuration changes (like instance type or container updates) against real live traffic without affecting production. |
| ### Question 3 - Monitoring Shadow Variant Performance Metrics A data science team has configured a SageMaker shadow test for a new fraud detection model. They need to analyze the operational behavior of the shadow variant, specifically looking at the latency and invocation errors it experiences when processing copied live requests. Which AWS service provides the native metrics and logs required to make this shadow variant measurable and comparable to the production variant? - A) AWS CloudTrail - B) Amazon OpenSearch Service - C) Amazon CloudWatch - D) Amazon S3 Access Logs (Select ONE) |
Correct Answer: C Correct Answer Analysis: - C is Correct: Amazon CloudWatch holds the metrics and logs that make a baseline measurable and a departure from that baseline visible over time. SageMaker automatically emits operational metrics like ModelLatency, OverheadLatency, and Invocation4XXErrors to CloudWatch, broken down by variant name (Production vs. Shadow).Distractor Analysis: - A is Incorrect: CloudTrail records API-level audit events (like CreateEndpoint or InvokeEndpoint calls) but does not contain the granular operational metrics (latency, memory, errors) of the model variants.- B is Incorrect: OpenSearch can be used to ingest and search logs, but it is not the native source of SageMaker endpoint operational metrics. - D is Incorrect: S3 Access Logs track bucket-level access requests, completely irrelevant to real-time ML endpoint invocation metrics. > Key Takeaway: Operational metrics for SageMaker production and shadow variants are natively captured in Amazon CloudWatch, allowing for direct performance comparison. |
| ### Question 4 - Analyzing Side-by-Side Predictions A team is running a shadow test on a SageMaker endpoint to evaluate a candidate text-classification model against the production model. The data scientists need to visually inspect the actual JSON payloads and predictions generated by both models for the same live requests to understand where they diverge. They want to achieve this with the least operational overhead, avoiding the need to build custom query dashboards. Which AWS tool provides a built-in view for this comparison? - A) Amazon Bedrock Evaluation reports - B) SageMaker Shadow Testing dashboard in SageMaker Studio - C) AWS Lambda function triggered by CloudWatch Events - D) Amazon QuickSight connected to CloudWatch Logs (Select ONE) |
Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker AI Shadow Testing provides a native dashboard within SageMaker Studio that allows data scientists to view side-by-side predictions of the production and shadow variants for the same live requests. This built-in UI requires no custom code or dashboarding tools. Distractor Analysis: - A is Incorrect: Bedrock Evaluations generate reports for offline evaluation jobs against stored datasets, not for analyzing live shadow test traffic from a SageMaker endpoint. - C is Incorrect: Writing a custom Lambda function to parse logs and compare outputs introduces unnecessary operational overhead compared to the native SageMaker Studio dashboard. - D is Incorrect: While QuickSight could visualize data, it requires setting up data ingestion and custom dashboards, violating the least operational overhead constraint. > Key Takeaway: SageMaker Studio provides a native Shadow Testing dashboard to visually compare the outputs of production and shadow variants side-by-side without writing custom analysis code. |
| ### Question 5 - Endpoint Type Limitations for Shadow Testing An e-commerce company wants to use SageMaker AI shadow testing to validate a newly fine-tuned Foundation Model. Their current model is deployed on a SageMaker Serverless Inference endpoint to handle intermittent, bursty traffic patterns. They must use shadow testing to compare the performance of the new model against live traffic without affecting customer responses. Which action should the team take to implement this architecture? - A) Deploy the shadow variant to the existing Serverless Inference endpoint and route 10% of traffic to it. - B) Re-architect the endpoint to a SageMaker Real-Time Inference endpoint before configuring the shadow test. - C) Use Amazon Bedrock Evaluations with human workers to simulate the live traffic concurrently. - D) Deploy the new model to an AWS Lambda function and use API Gateway to mirror traffic asynchronously. (Select ONE) |
Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker Shadow Testing requires a Real-Time Inference endpoint. Some endpoint configurations, such as Serverless Inference, do not support shadow variants. The team must migrate the deployment to a Real-Time endpoint to utilize the native shadow testing feature. Distractor Analysis: - A is Incorrect: SageMaker Serverless Inference endpoints do not support shadow testing natively. - C is Incorrect: Bedrock Evaluations with human workers runs an offline assessment, not a live traffic shadow deployment. - D is Incorrect: Deploying to Lambda and configuring API Gateway to mirror traffic is a highly complex, custom workaround that violates the constraint of using native shadow testing features. > Key Takeaway: Shadow testing is a feature of SageMaker Real-Time Inference endpoints and is not supported on Serverless or Asynchronous endpoints. |
| ### Question 6 - Asynchronous Traffic Routing Mechanics A financial services firm is deploying a candidate model as a shadow variant on a SageMaker Real-Time endpoint. The production endpoint currently serves requests with a strict 100ms p99 latency Service Level Agreement (SLA). The team must evaluate the candidate model under live load, but the architecture must guarantee that the performance of the shadow variant does not impact the production SLA. How does SageMaker Shadow Testing natively handle this constraint? - A) The endpoint synchronously waits for both variants to return a response before sending the production response to the caller. - B) A copy of the live request is asynchronously sent to the shadow variant, while only the production variant's response is returned to the caller immediately. - C) The shadow variant processes requests in a batch job overnight based on the traffic logged during the day. - D) Traffic is split 80/20, where 80% of callers receive the production response and 20% receive the shadow response. (Select ONE) |
Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker AI shadow testing asynchronously forks a copy of the real-time production request to the shadow variant. The production endpoint responds to the caller immediately without waiting for the shadow variant to finish processing, ensuring production latency is unaffected. Distractor Analysis: - A is Incorrect: The endpoint does not wait synchronously for the shadow variant, as this would introduce severe latency to the production response. - C is Incorrect: Shadow testing evaluates live requests in real-time, not in an overnight batch process. - D is Incorrect: This describes an 80/20 A/B test traffic split, which would expose 20% of users to the candidate model's responses, violating the shadow testing paradigm. > Key Takeaway: Shadow traffic is routed asynchronously; the caller's response time depends solely on the production variant, completely isolating production SLAs from shadow variant performance. |
| ### Question 7 - Selecting the Right Evaluation Methodology An AI engineering team has developed a new prompt template for their RAG application using Amazon Bedrock. They want to compare this new prompt against the prompt currently in production to see which one generates more concise answers. They need to evaluate the prompts using live traffic patterns, but the business dictates that no user can receive an answer generated by the unapproved new prompt. Which AWS service or feature should the team use? - A) Amazon Bedrock Prompt Management - B) Amazon Bedrock Knowledge Bases - C) SageMaker AI Shadow Testing - D) Amazon Bedrock Evaluations (Select ONE) |
Correct Answer: C Correct Answer Analysis: - C is Correct: While Bedrock Prompt Management handles prompt versioning, comparing a candidate against a production variant using live requests without changing what callers receive requires SageMaker AI Shadow Testing. If the models are deployed on SageMaker endpoints, this feature routes a copy of live traffic to the shadow variant for evaluation. Distractor Analysis: - A is Incorrect: Bedrock Prompt Management stores and versions prompts, allowing for side-by-side comparison, but it does not handle live endpoint traffic mirroring. - B is Incorrect: Bedrock Knowledge Bases is the retrieval source, not the mechanism for evaluating prompt variants against live traffic. - D is Incorrect: Amazon Bedrock Evaluations runs programmatic, automated, or human evaluation jobs against pre-compiled datasets, not live traffic. > Key Takeaway: When validating any change against live traffic with zero impact to callers, SageMaker Shadow Testing is the correct tool, even when comparing prompts or configurations. |
| ### Question 8 - Distinguishing Shadow Testing from Offline Evaluation A machine learning team is preparing to deploy a new computer vision model. They have rigorously tested the model on a static validation dataset, achieving high accuracy. However, they are concerned that the model may fail when exposed to the specific image resolutions and payload sizes currently hitting their SageMaker production endpoint. They must ensure the new model handles these live patterns before fully replacing the production model. Which approach will definitively prove the model's readiness under these constraints? - A) Run another programmatic evaluation job in Amazon Bedrock using the same static validation dataset. - B) Increase the size of the validation dataset and re-run the training pipeline with SageMaker Experiments. - C) Deploy the new model as a shadow variant on the SageMaker Real-Time endpoint to evaluate it against live request patterns. - D) Deploy the new model to the production endpoint and rely on SageMaker Model Monitor to catch drift. (Select ONE) |
Correct Answer: C Correct Answer Analysis: - C is Correct: Offline evaluation and live traffic often disagree because stored datasets rarely contain the exact request shapes, payload sizes, and concurrency of live production traffic. Deploying the model as a shadow variant evaluates it against real request patterns without risking production availability. Distractor Analysis: - A is Incorrect: Re-running the programmatic evaluation on a static dataset does not test the model against live request shapes or payload sizes. - B is Incorrect: Re-running the training pipeline addresses training accuracy but does not validate inference performance against live payloads. - D is Incorrect: Replacing the production model immediately and relying on Model Monitor exposes users to potential failures before readiness is validated. > Key Takeaway: A model that scores well on a stored dataset can still fail on live request shapes and payload sizes; shadow testing bridges this gap by testing against real traffic. |
| ### Question 9 - Configuring Shadow Test Traffic Percentage A logistics company is deploying a shadow test for a new routing optimization model. The production SageMaker endpoint receives highly variable traffic, with extreme bursts occurring during peak hours. The engineering team wants to ensure that the shadow variant's logs contain enough data to be statistically significant without overwhelming the shadow variant's instance during these bursts. Which configuration option in SageMaker Shadow Testing allows the team to control the amount of live traffic sent to the shadow variant? - A) Configuring the traffic split percentage for the shadow variant (e.g., 0-100%). - B) Using an AWS Lambda function to throttle the requests before they reach the shadow variant. - C) Configuring an Application Load Balancer with a weighted target group in front of the SageMaker endpoint. - D) Using SageMaker Inference Recommender to automatically calculate the optimal shadow traffic volume. (Select ONE) |
Correct Answer: A Correct Answer Analysis: - A is Correct: SageMaker AI shadow testing natively allows you to specify the percentage of live production traffic that is copied to the shadow variant. This ensures you can control the load on the shadow variant and gather statistically significant logs without overwhelming the candidate infrastructure. Distractor Analysis: - B is Incorrect: Implementing a custom Lambda function to throttle traffic introduces unnecessary operational overhead and breaks the native shadow testing architecture. - C is Incorrect: Application Load Balancers are not used to manage internal SageMaker endpoint variant traffic routing. - D is Incorrect: SageMaker Inference Recommender is used to recommend instance types for deployment, not to manage live shadow traffic percentages. > Key Takeaway: SageMaker Shadow Testing natively supports configuring the percentage of live traffic copied to the shadow variant, ensuring the candidate infrastructure is not overwhelmed. |
| ### Question 10 - Comparing GenAI System Components Safely An enterprise is running a traditional ML model on a SageMaker endpoint to classify support ticket severity. They have fine-tuned a new LLM on Bedrock to replace this traditional ML model, aiming to improve classification accuracy. They want to compare the performance of the Bedrock LLM against the existing SageMaker production model using live traffic. The architecture must ensure that the production responses are what the customer support agents continue to see. Which architectural pattern fulfills this requirement? - A) Expose the Bedrock LLM via a SageMaker endpoint container, and deploy it as a shadow variant alongside the production SageMaker endpoint. - B) Configure an Amazon Bedrock Evaluation using an LLM-as-a-judge to score both models on live traffic. - C) Route 50% of traffic from the support portal to Bedrock, and 50% to the SageMaker endpoint using API Gateway. - D) Log the production traffic to Amazon S3, and trigger a Bedrock Evaluation job hourly to compare the LLM against the logged data. (Select ONE) |
Correct Answer: A Correct Answer Analysis: - A is Correct: To perform a shadow test against live SageMaker traffic, the candidate model must be deployed as a shadow variant on a SageMaker endpoint. If the candidate is a Bedrock model, it can be wrapped in a SageMaker container (or invoked via a custom inference container) so it can participate natively in the SageMaker Shadow Testing framework. Distractor Analysis: - B is Incorrect: Bedrock Evaluations run as offline jobs against datasets, not as real-time, live-traffic shadow deployments. - C is Incorrect: A 50/50 API Gateway split is an A/B test, which would expose 50% of support agents to the new LLM's responses, violating the constraint. - D is Incorrect: Logging traffic to S3 and evaluating hourly is an offline batch evaluation pattern, failing to test the live concurrency, latency, and payload sizes in real-time. > Key Takeaway: To compare any candidate model against a SageMaker production model using live shadow traffic, the candidate must be deployed as a SageMaker endpoint variant, even if it wraps a Bedrock LLM invocation. |