Skip to content

Skill 2.3.3: Compare the performance of shadow variants to production variants

Question 1 - Shadow Test Deployment Strategy

A financial services company is deploying an updated fraud detection model to a SageMaker AI endpoint. The new model uses a different architecture and the ML team needs to validate its performance against the current production model using real live traffic. However, compliance requirements mandate that end-users must continue receiving predictions exclusively from the current production model during the validation period. The team also needs to collect operational metrics including latency and error rates for both models.

Which approach should the engineering team implement? (Select ONE)

  • A) Deploy the new model as an additional variant on the same SageMaker AI endpoint using a production variant weight of 100% and assign the new model a shadow variant weight of 100%.
  • B) Create a separate SageMaker AI endpoint for the new model and use Amazon Route 53 weighted routing to send 50% of traffic to each endpoint.
  • C) Deploy the new model to a SageMaker AI endpoint with instance scaling policies configured, then invoke it from a custom proxy layer that forwards requests from the production endpoint.
  • D) Use AWS CloudFormation to replace the production endpoint with the new model, relying on CloudWatch alarms to rollback if errors increase.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: SageMaker AI shadow testing allows deploying a shadow variant alongside a production variant on the same endpoint. Live traffic flows to the production variant (which returns responses to callers), while a copy of the requests is asynchronously routed to the shadow variant for offline logging. The production variant maintains 100% visibility to callers while the shadow variant receives mirrored traffic for side-by-side comparison.

Distractor Analysis

  • B is Incorrect: Route 53 weighted routing sends real caller traffic directly to both endpoints (A/B testing), which violates the compliance requirement that callers only receive production predictions. It also introduces unnecessary infrastructure overhead with a second endpoint.
  • C is Incorrect: Building a custom proxy layer introduces significant architectural complexity and operational maintenance, and would require the proxy to synchronously forward requests—defeating the purpose of SageMaker's built-in shadow testing capabilities and risking impact to production latency.
  • D is Incorrect: Replacing the production endpoint directly exposes callers to the new model's predictions before validation is complete, violating the compliance mandate for exclusive production responses during comparison.

Key Takeaway: SageMaker AI shadow testing is the purpose-built native solution for safely comparing a candidate variant against production using mirrored live traffic without affecting caller responses.

Question 2 - Shadow Test Logging with Data Capture

A retail ML team is running a shadow test on a SageMaker AI endpoint to compare a new product recommendation model against the current production variant. The candidate model receives a mirrored copy of production traffic, and the team needs to capture both the incoming request payloads and the shadow variant's outputs for offline evaluation. Compliance rules require that captured inference data be encrypted at rest and the solution must have minimal operational overhead.

Which TWO configurations should be used? (Select TWO)

  • A) Enable SageMaker AI Data Capture on the endpoint, specifying an Amazon S3 destination bucket with SSE-KMS encryption enabled for data at rest.
  • B) Write a custom logging middleware inside the model container to push request/response data to Amazon OpenSearch Service using the AWS SDK.
  • C) Configure enable_capture=True in the SageMaker AI SDK when creating the shadow test, directing captured data to CloudWatch Logs.
  • D) Define the shadow traffic routing to invoke a Lambda function that serializes the request/response pairs into a DynamoDB table.
  • E) Use SageMaker AI Model Monitor to periodically batch-transform historical requests and log predictions to an encrypted S3 bucket.
Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: SageMaker AI Data Capture is the native, serverless mechanism for logging real-time inference requests and responses to Amazon S3. Enabling SSE-KMS on the destination bucket satisfies the encryption-at-rest compliance requirement with minimal operational burden.
  • C is Correct: The SageMaker AI SDK provides a built-in enable_capture parameter for shadow tests. This automatically routes the captured shadow variant traffic for logging, eliminating custom code.

Distractor Analysis

  • B is Incorrect: Building custom logging middleware inside the container adds significant code maintenance, potential performance overhead on the hot path, and requires upfront capacity provisioning for OpenSearch. It violates the minimal operational overhead constraint.
  • D is Incorrect: While Lambda + DynamoDB can log requests, this introduces unnecessary custom architecture compared to SageMaker's native Data Capture. It adds operational complexity and potential latency on the shadow request path.
  • E is Incorrect: SageMaker AI Model Monitor is designed for detecting drift in production data distributions and prediction quality using baseline statistics from historical/captured data, not for real-time shadow test logging. Batch transform would not capture live mirrored requests.

Key Takeaway: SageMaker AI Data Capture with S3 + KMS encryption provides zero-code, compliant request/response logging for shadow tests; avoid custom middleware or batch-style Model Monitor for live capture needs.

Question 3 - Shadow Test Comparison Metrics

A media company is running a shadow test on SageMaker AI comparing a new content moderation LLM against their current production variant. Both variants process the same live traffic. The team needs to determine whether the candidate model should be promoted. The evaluation must include a combination of automated metrics and operational telemetry.

Which metrics should the team examine to make a promotion decision? (Select TWO)

  • A) Compare invocation latency percentiles (p50, p99) between the production and shadow variants using Amazon CloudWatch.
  • B) Compare the exact-match accuracy of both variants against caller-reported feedback forms.
  • C) Compare the semantic embedding similarity between both variants' outputs without reference to any ground truth or business criteria.
  • D) Compare error rate and 5xx response counts emitted by each variant on the same endpoint over the same rolling time window.
  • E) Use the total GPU utilization of the training cluster to decide whether the shadow endpoint has enough capacity.
Click to reveal Answer & Explanation

Correct Answers: A and D

Correct Options Analysis

  • A is Correct: Comparing latency percentiles from CloudWatch metrics is a critical operational validation. Shadow tests are designed to detect real-world traffic impacts (such as increased response time) that offline evaluations would never reveal.
  • D is Correct: Error rates and 5xx counts are core operational indicators captured per-variant in SageMaker AI endpoint CloudWatch metrics. Consistent parity or improvement in errors is required before promoting a candidate.

Distractor Analysis

  • B is Incorrect: Exact-match accuracy is unsuitable for text generation tasks like moderation outputs. Valid responses can differ in wording while being equally correct; requiring exact-match would misleadingly penalize the shadow variant.
  • C is Incorrect: Semantic similarity alone, without business criteria like moderation correctness, fairness, or toxicity labels, is not sufficient for content moderation evaluation. It captures fluency/meaning but not whether the moderation decision was actually correct or aligned with policy.
  • E is Incorrect: While capacity is important for availability, training cluster GPU utilization is completely disconnected from endpoint serving performance. The shadow test evaluates inference speed/errors on deployed infrastructure, not where the model was trained.

Key Takeaway: Shadow tests shine for operational validation—use CloudWatch latency, error, and throughput metrics on the endpoint; do not rely on exact-match or similarity-only measures for GenAI output quality.

Question 4 - Shadow Test Requirement: Same Endpoint

An engineering team at a healthcare company wants to use SageMaker AI shadow tests to validate a new medical coding assistant model before rollout. The team needs to ensure the shadow test is configured correctly so the candidate receives real live traffic while the production responses remain unchanged for the callers.

Which configuration meets this requirement natively in SageMaker AI? (Select ONE)

  • A) Deploy both the production model and the candidate model as separate production variants on the same endpoint, and set shadow_variant_weight=100 on the candidate.
  • B) Deploy the candidate model to a different endpoint in the same VPC, and create a peering connection to mirror traffic from the production endpoint's security group.
  • C) Deploy the candidate model to an AWS Inferentia endpoint and route live traffic using Elastic Load Balancing with a weighted target group.
  • D) Deploy both models as separate endpoints behind CloudFront, and use Lambda@Edge to fork requests to the candidate endpoint.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: SageMaker AI shadow testing requires both models to be deployed on the same SageMaker AI endpoint. The candidate (shadow) variant has a shadow_variant_weight (e.g., 100%) that instructs SageMaker to send a mirrored copy of traffic to the shadow variant while the production variant remains the sole responder to callers.

Distractor Analysis

  • B is Incorrect: SageMaker AI shadow testing cannot work across separate endpoints; the native mechanism requires co-location on a single endpoint. VPC peering is for networking connectivity between VPCs and is not a valid traffic mirroring solution for model comparison.
  • C is Incorrect: While Inferentia instances can host models and ELB can support weighted routing, this describes A/B or canary testing across endpoints that directly exposes callers to both model versions. It does not mirror traffic invisibly to a shadow.
  • D is Incorrect: Lambda@Edge and CloudFront add massive architectural complexity. SageMaker AI supports shadow variants natively on a single endpoint; building an edge-level traffic forking layer is unnecessary and introduces latency.

Key Takeaway: A shadow version requires same-endpoint deployment as a variant with shadow weight configuration—you cannot achieve native shadow testing across separate endpoints or independent infrastructure.

Question 5 - Shadow Test vs. Offline Baseline Evaluation

An e-commerce company is running a recommendation system on SageMaker AI. The data science team has already run offline evaluations on an A/B test using historical data and found that Candidate Model X performs 4% better on Mean Reciprocal Rank (MRR) than the current production model. Before promoting Model X, the team wants to validate that it performs similarly under real traffic patterns, including cold-start requests and malformed payloads.

Which approach provides the strongest evidence for safe promotion? (Select ONE)

  • A) Run an Amazon Bedrock evaluation job on Model X using the same holdout dataset but with ROUGE-L as the metric.
  • B) Deploy Model X as a shadow variant on the production endpoint and monitor CloudWatch latency, error rate, and prediction drift vs. the current production variant over a two-week live traffic window.
  • C) Increase the confidence in the original offline result by retraining Model X with 5× more historical data.
  • D) Use SageMaker AI Clarify to compare the model's SHAP explanations against those from the production model.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: Shadow testing is specifically designed to validate a model under real production traffic patterns (including malformed inputs, concurrency, cold-start, and payload anomalies) with zero risk to callers. Comparing latency, error rate, and response distribution on live traffic provides the strongest operational evidence before promotion.

Distractor Analysis

  • A is Incorrect: Amazon Bedrock evaluations are for foundation model outputs and use precompiled offline datasets, not for comparing SageMaker AI endpoint variants under live traffic. Also, ROUGE-L is a text-generation metric, not appropriate for recommendation ranking.
  • C is Incorrect: Retraining with more historical data only strengthens the offline evaluation, not the live validation. Historical data can still miss cold-start behaviors, concurrency spikes, and payload anomalies.
  • D is Incorrect: SageMaker AI Clarify provides bias detection and model explainability (SHAP), not live traffic validation of endpoint variants. Model explanations do not answer questions about real-time operational health.

Key Takeaway: Structuring real-time shadow tests on the production endpoint complements offline metrics by catching live-traffic issues offline evaluation misses—promote only after both evaluations align.

Question 6 - Shadow Test Metric Comparison in CloudWatch

A fraud detection team runs a shadow test on their SageMaker AI endpoint. The production variant serves all caller traffic and the shadow variant receives mirrored traffic. They want to set up automated monitoring so that an alert is raised when the shadow variant's error rate exceeds the production variant's error rate by more than 1% for 5 minutes. They want minimal operational overhead and near-real-time detection.

Which configuration should they use? (Select ONE)

  • A) Schedule a daily AWS Glue job to scan S3 query result exports and compute a diff between shadow and production error rates.
  • B) Create a CloudWatch composite alarm comparing the Invocation5XXErrors metric from the production model variant and the shadow variant dimensions on the same endpoint, triggering when the difference exceeds 1 percentage point.
  • C) Enable GuardDuty anomaly detection on the SageMaker AI endpoint to detect unusual error counts from the candidate variant.
  • D) Use AWS Trusted Advisor recommendations to identify when the shadow variant has failing requests.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: CloudWatch Metrics automatically publish per-variant endpoint metrics including Invocation5XXErrors, Invocation4XXErrors, ModelLatency, and OverheadLatency. A CloudWatch composite alarm (or Metric Math on per-variant dimensions) can monitor the delta between shadow and production error rates in near real-time and trigger notifications via SNS, all with no custom infrastructure.

Distractor Analysis

  • A is Incorrect: A daily Glue job introduces up to 24 hours of detection lag and unnecessary Apache Spark infrastructure for a metric that CloudWatch already reports in real time. This violates the near-real-time and minimal overhead requirements.
  • C is Incorrect: GuardDuty is a security monitoring and threat detection service—not a model performance or endpoint error monitoring tool. It would not provide model variant error-rate comparisons.
  • D is Incorrect: Trusted Advisor is a cloud optimization and best-practices advisor for cost, security, performance, and fault tolerance—not an operational monitoring tool for model endpoint performance comparisons.

Key Takeaway: CloudWatch native metric dimensions for each endpoint variant make real-time shadow-vs-production comparison simple—no need for batch ETL or security tools.

Question 7 - Understanding Shadow Test Limitations

An ML engineer at a logistics company wants to run a shadow test on a candidate model using SageMaker AI. However, when attempting to configure the shadow test, they notice that certain endpoint configurations do not support shadow testing. They need to choose an endpoint setup that is guaranteed to support shadow variants.

Which endpoint configurations must be avoided because shadow tests are not supported? (Select TWO)

  • A) Single-model endpoints with an ML compute instance type such as ml.m5.xlarge.
  • B) Serverless inference endpoints.
  • C) Multi-model endpoints.
  • D) Multi-container endpoints hosting inference pipelines.
  • E) GPU-backed single-model endpoints using ml.g5.xlarge.
Click to reveal Answer & Explanation

Correct Answers: B and C

Correct Options Analysis

  • B is Correct: Shadow testing is not supported on SageMaker Serverless Inference endpoints. Serverless endpoints are designed for bursty, intermittent traffic and do not support the variant-level traffic mirroring infrastructure required for shadow tests.
  • C is Correct: Multi-model endpoints (where multiple models are deployed to the same instance and are invoked by target model identifier) are not supported for shadow testing because traffic routing to individual model copies within the same instance is not compatible with the shadow mirroring infrastructure.

Distractor Analysis: - A is Incorrect: Single-model endpoints on standard ML compute instances (such as ml.m5.xlarge) are fully supported for shadow testing. This is the typical foundation for a shadow deployment. - D is Incorrect: Multi-container endpoints (which host inference pipelines with different containers for preprocessing, model inference, and postprocessing) are supported for shadow testing in SageMaker AI when configured with a production and shadow variant. - E is Incorrect: GPU-backed single-model endpoints such as ml.g5.xlarge are valid targets for shadow tests—shadow testing support depends on endpoint type and variant configuration, not on whether the instance is CPU or GPU.

Key Takeaway: Avoid serverless inference and multi-model endpoints when planning shadow tests; they are incompatible with SageMaker AI's shadow variant architecture.

Question 8 - Shadow Test Baseline and Representation Analysis

A banking ML team deploys a new credit-scoring model as a shadow variant and compares it against the production variant over a one-week period. After analyzing the logs, they notice the shadow model's output distribution differs substantially from the training data distribution, but the shadow model's validation metrics on the held-out test set were strong.

What is the most appropriate next step? (Select ONE)

  • A) Immediately promote the shadow model because its offline accuracy was strong.
  • B) Ignore the discrepancy because shadow tests do not measure distribution shift.
  • C) Investigate the discrepancy by comparing the incoming live traffic distribution with the training data using distribution shift detection (e.g., SageMaker Model Monitor), and reconsider promotion based on findings.
  • D) Adjust the training data to match the shadow output distribution and retrain.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

  • C is Correct: A discrepancy between the shadow model's live output distribution and the training data can indicate data drift (input distribution shift) or model bias that offline evaluation did not surface. Using SageMaker Model Monitor (or similar techniques) to quantify the drift and understand whether the model is misbehaving on unseen live distributions is the responsible next step before promotion.

Distractor Analysis

  • A is Incorrect: Promoting solely based on offline validation metrics ignores the value of shadow testing, which exists precisely to catch live-traffic issues like input drift, cold-start behavior, or infrastructure anomalies that are invisible offline.
  • B is Incorrect: Shadow tests are explicitly designed to reveal live performance differences; dismissing output distribution discrepancies defeats the purpose. Model quality issues due to data drift should be investigated, not ignored.
  • D is Incorrect: Retraining to force the model to match a potentially anomalous observed output distribution is backwards. The goal is not to mimic the distribution; it is to understand why the live data differs, and to fix the model or pipeline only after determining the root cause.

Key Takeaway: Shadow test output discrepancies are fingerprints of real-world distribution shift—dig into them with drift detection before making promotion decisions.

Question 9 - Shadow Test Frequency and Continuous Validation

A technology company is committed to continuous deployment for their ML services. They plan to update their recommendation model weekly using automated pipelines. They want to use shadow testing before every production promotion. However, the team is concerned about cost, because shadow testing effectively doubles endpoint compute expenses during validation.

Which strategy minimizes cost while still enabling shadow validation? (Select ONE)

  • A) Always keep shadow variants running 24/7 so they are always ready to begin the next deployment cycle.
  • B) Run shadow tests only during off-peak hours, because traffic is lower and compute is cheaper.
  • C) Deploy shadow variants for a limited validation window (e.g., 2–4 days) sufficient to gather enough live traffic, then delete the shadow variant after analysis to avoid indefinite double-spend.
  • D) Use synthetic traffic generated by a separate batch transform job instead of live shadow tests; this completely avoids doubling endpoint costs.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

  • C is Correct: Shadow testing should be scoped to a finite validation window—long enough to collect a statistically meaningful sample of real traffic (e.g., a few days during normal peak and off-peak cycles) but not indefinite. Deleting the shadow variant after evaluation avoids permanent double compute cost while still reaping the benefits of live request validation.

Distractor Analysis

  • A is Incorrect: Keeping the shadow variant permanently running is a costly anti-pattern—it doubles endpoint compute cost at all times and provides no value when no new candidate model is being validated.
  • B is Incorrect: Running shadow tests only during off-peak hours restricts the traffic mix. If your traffic patterns vary significantly between day/night (e.g., e-commerce), validating only on sparse off-peak traffic will create an unrepresentative sample and may poison your promotion decision.
  • D is Incorrect: Batch transform processes static pre-generated requests, not live traffic. This eliminates the core benefit of shadow testing (validating against real, fluctuating, potentially noisy production payloads) and therefore does not satisfy the requirement of using shadow testing.

Key Takeaway: Shadow tests are powerful but should be time-boxed. Deploy for the minimum window needed for statistical confidence, then delete the shadow variant to stop double spending.

Question 10 - Comparing Shadow Test Metrics From Production Variant

An ML engineer has configured a shadow test on their SageMaker AI endpoint, deploying model version v2 as the shadow variant and keeping v1 as the production variant. They want to programmatically retrieve the latency percentiles of the shadow variant from CloudWatch to compare against the production variant in a single script.

Which dimensions should the script specify to isolate metrics for only the shadow variant? (Select ONE)

  • A) EndpointName=<endpoint-name>, VariantName=<variant-name>
  • B) EndpointName=<endpoint-name>, ModelName=All
  • C) EndpointName=<endpoint-name>, InstanceType=ml.m5.xlarge
  • D) EndpointName=<endpoint-name>, ContainerName=<container-name>
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: SageMaker AI publishes CloudWatch metrics for each production or shadow variant under the dimensions EndpointName and VariantName. Specifying both dimensions precisely isolates metrics (e.g., ModelLatency, OverheadLatency) for the shadow variant alone.

Distractor Analysis

  • B is Incorrect: There is no valid CloudWatch dimension called ModelName for SageMaker AI endpoint metrics. The correct dimension pair is EndpointName and VariantName.
  • C is Incorrect: InstanceType is not a CloudWatch dimension for SageMaker endpoint metrics. Instance type influences capacity and performance but cannot be used to filter variant-specific metrics.
  • D is Incorrect: While container metrics can exist, SageMaker endpoint variant metrics (latency, invocations, errors) are not published under a ContainerName dimension. The native isolation mechanism is via VariantName.

Key Takeaway: Use the EndpointName + VariantName CloudWatch dimensions to programmatically compare operational metrics for shadow vs. production variants.