Skip to content

Skill 4.1.4: Monitor model performance in production by using A/B testing

Question 1 - Configuring SageMaker Production Variants for A/B Testing

A financial services company has deployed a credit risk classification model on Amazon SageMaker. The team has trained a new model version using refined feature engineering and wants to validate its performance against the current production model using live traffic. The team needs to gradually shift traffic to the new model while ensuring both models can be compared on identical evaluation metrics over the same time window. The solution should minimize operational complexity and use native SageMaker capabilities.

Which configuration should the engineering team implement to meet these requirements?

  • A) Deploy both model versions to separate SageMaker endpoints, and use Amazon Route 53 weighted routing to split traffic between the two endpoints.
  • B) Configure a single SageMaker endpoint with two production variants, setting initial traffic distribution weights (e.g., 90% current model, 10% candidate model).
  • C) Deploy both models behind an Application Load Balancer with target groups pointing to separate endpoints, using listener rules to split traffic.
  • D) Deploy the candidate model to a separate endpoint and manually run offline evaluation using recorded historical traffic before exposing it to any production traffic.
  • E) Create two SageMaker batch transform jobs that process the same historical dataset offline, then compare the outputs.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: SageMaker endpoints natively support multiple production variants behind a single endpoint with configurable traffic distribution weights. This allows live A/B testing where traffic is split proportionally (e.g., 90% to current variant, 10% to candidate variant) while both variants emit comparable CloudWatch metrics over the same time window.

Distractor Analysis

  • A is Incorrect: Route 53 weighted routing operates at the DNS level and introduces unnecessary complexity for model A/B testing. It also lacks native integration with SageMaker's variant-level metrics, making apples-to-apples comparison difficult.
  • C is Incorrect: Application Load Balancer target groups are designed for traditional web workloads, not SageMaker endpoints. This architecture requires managing EC2-based reverse proxies and introduces significant operational overhead.
  • D is Incorrect: Manual offline evaluation using historical traffic does not satisfy the requirement to test with live production traffic and measure both versions concurrently.
  • E is Incorrect: Batch transform jobs process offline data, not live traffic. This approach validates model quality on historical data but does not measure production performance concurrently with live users.

Key Takeaway: SageMaker production variants on a single endpoint are the native mechanism for A/B testing—simply set traffic distribution weights and compare variant-level metrics.

Question 2 - A/B Testing with Shadow Variants for High-Risk Models

A healthcare analytics company has developed a new version of a patient readmission prediction model that uses a novel neural network architecture. While offline evaluation shows promising results, the clinical operations team is concerned about the risk of exposing patients to incorrect predictions during validation. The team must validate the candidate model against real production traffic without returning the candidate model's predictions to end users. Both models must be monitored on the same metrics for comparison.

Which approach should the ML engineer implement to satisfy these requirements?

  • A) Deploy the candidate model as a shadow variant on the same SageMaker endpoint, configure it to receive a copy of all incoming traffic, log its responses, and return only the incumbent model's predictions to the caller.
  • B) Deploy the candidate model to a separate stage environment, replay production traffic through it offline, and compare results with the production model's historical outputs manually.
  • C) Set production variant weights to 0% for the candidate model, enabling it to learn from production traffic while serving no predictions.
  • D) Deploy the candidate model as a second variant with 5% traffic weight so a small number of patients receive the new predictions, minimizing risk exposure.
  • E) Disable the incumbent model and run the candidate model exclusively in production for a one-hour trial window during low-traffic hours.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: A shadow variant receives a copy of production traffic without having its responses returned to users. This validates the candidate model's serving performance and prediction quality against real traffic patterns while the incumbent model's outputs are what users actually receive. Both variants can be monitored via CloudWatch metrics during this process.

Distractor Analysis

  • B is Incorrect: Replaying traffic offline does not test the model in true production conditions. While this can validate model quality, it does not measure production serving behavior, latency, or resource utilization under real concurrent loads.
  • C is Incorrect: Setting the candidate variant weight to 0% prevents it from receiving any traffic at all—SageMaker does not "learn" from production traffic without explicitly invoking the endpoint variant.
  • D is Incorrect: This exposes real patients to potentially incorrect predictions from an unvalidated model, violating the safety constraint of not returning candidate predictions to end users.
  • E is Incorrect: Exclusive deployment during low-traffic hours still exposes users to candidate predictions and does not allow concurrent comparison between the two model versions.

Key Takeaway: Shadow variants receive real traffic copies without serving responses to users, validating candidate models safely when live exposure carries unacceptable risk.

Question 3 - Evaluating A/B Test Results Using SageMaker Clarify

An e-commerce company has deployed two recommendation models on a single SageMaker endpoint as production variants (Variant A: current matrix factorization model at 80% traffic, Variant B: new deep learning model at 20% traffic). After running the A/B test for two weeks, the team notices Variant B shows significantly lower click-through rates. Before rolling back, the team suspects the metric difference may be explained by differences in traffic composition rather than model quality.

Which TWO actions should the team take to properly evaluate whether the performance difference is statistically valid? (Select TWO)

  • A) Use Amazon SageMaker Clarify with a bias report to compare whether Variant B is receiving disproportionately different user demographics compared to Variant A, which might confound the comparison.
  • B) Immediately roll back Variant B without further investigation since click-through rate is a definitive business metric and lower performance always indicates model degradation.
  • C) Increase Variant B's traffic weight to 50% and run the test for one additional day to gather more data quickly.
  • D) Generate an explainability report using SageMaker Clarify on Variant B to understand whether the model is using unintended input features that explain the performance gap.
  • E) Create a custom CloudWatch Dashboard that displays both variants' invocation success rates and p99 latency, then configure automatic rollback if Variant B's p99 latency exceeds 500 milliseconds.
Click to reveal Answer & Explanation

Correct Answers: A and D

Correct Options Analysis

  • A is Correct: SageMaker Clarify provides bias detection and data distribution analysis capabilities. Comparing the traffic composition between variants helps identify whether confounding variables (e.g., different user demographics routed by region, device type, or session characteristics) are driving the performance difference rather than model quality.
  • D is Correct: SageMaker Clarify's explainability reports reveal feature attributions for the candidate model. If the deep learning model relies on spurious correlations or unintended features, this explains why it underperforms despite promising offline validation.

Distractor Analysis

  • B is Incorrect: Immediately rolling back without investigating the root cause fails to distinguish between genuine model degradation and confounding factors. A statistically invalid comparison wastes the entire test effort.
  • C is Incorrect: Increasing traffic to 50% and shortening the test duration amplifies business risk and further skews the comparison—it does not address the underlying question of whether current differences are statistically valid.
  • E is Incorrect: While monitoring invocation success rates and latency is valuable, these metrics are about serving health—not recommendation quality. Automatic rollback based on latency alone does not resolve the click-through rate question. Additionally, SageMaker does not support natively triggering automatic rollback from a CloudWatch Dashboard; this requires an alarm and an EventBridge/Lambda integration.

Key Takeaway: When A/B test results show performance gaps, use Clarify to check for traffic bias (distribution skew) and feature attribution issues before attributing the difference to model quality.

Question 4 - CloudWatch Metrics for Variant-Level A/B Test Monitoring

A media streaming platform is running an A/B test on a SageMaker multi-variant endpoint comparing the current content ranking model (Variant A) against a new transformer-based ranking model (Variant B). The ML team needs to configure monitoring that enables them to compare both variants on model invocation errors, latency, and invocation counts in a single view. The solution must have minimal operational overhead and integrate natively with SageMaker.

Which TWO configuration choices would fulfill these monitoring requirements? (Select TWO)

  • A) Configure an Amazon CloudWatch dashboard with metrics from the AWS/SageMaker namespace, filtering by both EndpointName and VariantName dimensions.
  • B) Export raw inference logs from both variants to separate Amazon S3 buckets, then run a custom ETL job using AWS Glue to compute metrics and visualize them with Amazon QuickSight.
  • C) Enable SageMaker Model Monitor with a data quality baseline for both production variants to collect distribution statistics on incoming feature vectors.
  • D) Set up CloudWatch alarms for the ModelInvocationErrorRate metric filtered by VariantName=All, then use CloudWatch Logs Insights to break down errors by variant manually.
  • E) Create a custom metric using Amazon Managed Service for Prometheus that scrapes SageMaker endpoints through an exporter sidecar.
Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: SageMaker endpoints automatically publish metrics (e.g., InvocationCount, ModelInvocationErrors, ModelLatency) to the AWS/SageMaker namespace, with dimensions including EndpointName and VariantName. A CloudWatch dashboard using these dimensions provides native, per-variant comparison with minimal operational overhead.
  • C is Correct: SageMaker Model Monitor for data quality creates a baseline of incoming feature distributions and detects data drift. Monitoring both variants with data quality enables the team to ensure that differences in model performance are not caused by different data distributions being routed to each variant.

Distractor Analysis

  • B is Incorrect: Building a custom ETL pipeline with Glue and QuickSight introduces significant operational overhead and latency in metric computation. The native CloudWatch metrics already provide the required variant-level views without custom infrastructure.
  • D is Incorrect: While CloudWatch alarms can be useful, filtering by VariantName=All aggregates both variants and obscures the comparison. The requirement specifically asks for comparing both variants on the same metrics, which requires per-variant dimension filtering.
  • E is Incorrect: Amazon Managed Service for Prometheus is not natively integrated with SageMaker endpoints. While possible through custom exporters, this approach introduces unnecessary complexity and is not the standard AWS-native pattern for SageMaker monitoring.

Key Takeaway: SageMaker natively publishes variant-level metrics to CloudWatch—build dashboards and alarms using EndpointName and VariantName dimensions for A/B test comparisons.

Question 5 - Traffic Shifting Strategy Based on A/B Test Results

A SaaS company has been A/B testing a new churn prediction model on a SageMaker endpoint for three weeks. The initial configuration was 90% traffic to the incumbent model (Variant A) and 10% to the candidate model (Variant B). The A/B test has shown that Variant B consistently outperforms Variant A on both precision and recall, with statistically significant results. The team wants to progressively move all production traffic to the candidate model while maintaining the ability to roll back quickly if issues emerge.

Which approach should the ML engineer use to implement the traffic shift?

  • A) Delete Variant A from the endpoint configuration and keep Variant B as the sole variant so all traffic routes to the candidate model immediately.
  • B) Update the endpoint configuration to set Variant A's weight to 0% and Variant B's weight to 100%, then wait 24 hours before deleting Variant A.
  • C) Gradually shift traffic by updating production variant weights in phases (e.g., 50% A / 50% B, then 20% A / 80% B, then 0% A / 100% B), monitoring metrics at each phase.
  • D) Deploy Variant B to a brand-new endpoint with a new endpoint name, update the application to point to the new endpoint, and decommission the old endpoint.
  • E) Configure an AWS Step Functions workflow that automatically adjusts production variant weights every 10 minutes until Variant B reaches 100%.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Options Analysis

  • C is Correct: A phased traffic shift (e.g., 50/50, 20/80, 0/100) is the standard best practice for transitioning from an A/B test to full production. Each phase allows the team to monitor for regressions, serving issues, or unexpected behavior before committing more traffic. SageMaker supports weight updates through the UpdateEndpointWeightsAndCapacities API, and the incumbent variant remains available for immediate rollback throughout the transition.

Distractor Analysis

  • A is Incorrect: Immediately deleting Variant A provides no rollback capability. If Variant B fails in production (e.g., serving errors under full load, resource contention), the team would need to redeploy Variant A from scratch, causing significant downtime.
  • B is Incorrect: A single-step 0/100 switch does not allow for gradual validation under increasing load. While keeping Variant A available for rollback is good practice, the abrupt transition defeats the purpose of gradual exposure and may amplify any undiscovered scaling issues.
  • D is Incorrect: Deploying to a new endpoint breaks the A/B comparison continuity, requires application changes to update the endpoint URL, and makes rollback more complex. SageMaker's native variant mechanism is designed precisely to avoid this pattern.
  • E is Incorrect: Automated weight updates every 10 minutes are unnecessarily aggressive and risky. The team should review metrics manually (or via automated guardrails with thresholds) before each traffic shift, not blindly increase traffic on a fixed schedule without checking for issues.

Key Takeaway: Gradual traffic shifts between production variants (phased weight updates) with metric monitoring at each step provide the safest path to full deployment while preserving rollback capability.

Question 6 - Production Variant Capacity Configuration for A/B Testing

A gaming company is running an A/B test comparing two lightweight inference models on a SageMaker endpoint. The current model (Variant A) serves 70% of traffic with 3 instances, and the candidate model (Variant B) serves 30% of traffic with 1 instance. After two weeks, the team decides to shift Variant B to 70% traffic. The team is concerned about latency degradation if Variant B does not have adequate instance count for the new traffic volume.

Which additional configuration should the ML engineer include in the UpdateEndpointWeightsAndCapacities API call?

  • A) Only update the traffic weight of Variant B to 70%; instance count automatically scales based on incoming traffic.
  • B) Update both the traffic weight (Variant B to 70%) and instance capacity (scale Variant B from 1 to 3 instances), while optionally scaling down Variant A's instance count.
  • C) Configure SageMaker automatic scaling for Variant B with a target invocation count of 1000 requests per minute and wait for scaling to occur naturally over the next several days.
  • D) Delete the endpoint and recreate it with Variant B at 70% weight and 3 instances to ensure capacity matches traffic.
  • E) Set both variants to equal weight (50% / 50%) with equal instance counts (2 instances each) and let SageMaker balance capacity automatically.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: When shifting traffic between production variants, both traffic weight and instance capacity need to be updated together to maintain performance. The UpdateEndpointWeightsAndCapacities API allows simultaneous weight and capacity adjustments. Increasing Variant B's instance count from 1 to 3 ensures it can handle 70% of traffic without latency degradation, and reducing Variant A's capacity optimizes cost as its traffic decreases.

Distractor Analysis

  • A is Incorrect: SageMaker endpoints do not automatically scale instance counts based on traffic weight changes. Unless Application Auto Scaling policies are explicitly configured with appropriate metrics and thresholds, the instance count remains static regardless of traffic weight.
  • C is Incorrect: Application Auto Scaling is reactive—it responds to actual invocation metrics, not predicted future traffic. Waiting for autoscaling to "catch up" exposes users to potential latency spikes and throttling during the transition period.
  • D is Incorrect: Deleting and recreating the endpoint causes downtime and loses the A/B test continuity. SageMaker provides a native mechanism (UpdateEndpointWeightsAndCapacities) that achieves the same result without endpoint recreation.
  • E is Incorrect: Equal 50/50 split is not the requested transition (70% to Variant B). Arbitrarily changing both variants' weights and capacities does not align with the stated requirement of shifting 70% traffic to Variant B.

Key Takeaway: Weight changes and capacity changes go together—use UpdateEndpointWeightsAndCapacities to adjust both simultaneously during A/B test transitions.

Question 7 - Statistical Approach for Validating A/B Test Results

A retail analytics company has been running an A/B test with 90% traffic to the current demand forecasting model (Variant A) and 10% to a new gradient-boosting model (Variant B). After four weeks, the team observes that Variant B has a 2% lower Mean Absolute Percentage Error (MAPE). The data science lead wants to determine whether this improvement is statistically significant before making a full transition.

Which approach should the team use to establish statistical validity?

  • A) Use Amazon SageMaker Model Monitor to trigger an alert when the MAPE metric of Variant B exceeds a predefined threshold for three consecutive days.
  • B) Collect per-request prediction errors from both variants over the test period and perform a paired hypothesis test (e.g., t-test or Wilcoxon signed-rank test) comparing the two error distributions.
  • C) Trust the result because the test ran for four weeks, which is always sufficient for any A/B test to reach statistical significance regardless of effect size or sample size.
  • D) Configure an Amazon Bedrock model evaluation job with a judge model to score the quality of both variants and determine which is superior.
  • E) Compare only the aggregate MAPE values from the first and last weeks of the test, discarding intervening data points to simplify the analysis.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: Statistical significance testing requires comparing the observed performance metrics at the individual observation level. A paired hypothesis test (paired t-test for approximately normal differences or Wilcoxon signed-rank test for non-parametric comparisons) directly tests whether the difference in MAPE between variants is unlikely to have occurred by chance.

Distractor Analysis

  • A is Incorrect: Model Monitor can alert on thresholds, but setting arbitrary thresholds and counting consecutive days does not establish statistical significance. This approach confuses monitoring with hypothesis testing.
  • C is Incorrect: Test duration alone does not guarantee statistical significance. Statistical power depends on sample size, effect size, and variance. A small effect with high variance may not reach significance even after months.
  • D is Incorrect: Amazon Bedrock model evaluation is designed for generative AI foundation model evaluations, not for comparing traditional ML regression models on numerical forecasting accuracy.
  • E is Incorrect: Discarding data points from the analysis reduces statistical power and may introduce bias. A valid statistical test uses all observations or an appropriately justified subsampling strategy.

Key Takeaway: A/B test conclusions require formal hypothesis testing on per-observation metrics—aggregate comparisons and arbitrary thresholds do not establish statistical validity.

Question 8 - Multi-Metric Evaluation for A/B Testing with Conflicting Results

A financial fraud detection team has deployed a new model (Variant B) alongside the current model (Variant A) in a 20%/80% traffic split on a SageMaker endpoint. The A/B test results show that Variant B has:

  • Precision: 95% (A: 88%)
  • Recall: 45% (A: 68%)
  • False Positive Rate: 2% (A: 8%)

The compliance team mandates that any new fraud model must not miss more than 50% of fraudulent transactions. Business stakeholders prioritize reducing false positives to lower operational review costs.

Based on these results, what should the ML engineer recommend?

  • A) Promote Variant B to full production immediately because it has higher precision and lower false positive rate.
  • B) Keep Variant A in production because it has higher recall, which is always the most important metric for fraud detection.
  • C) Reject Variant B's promotion because its recall of 45% violates the compliance requirement of detecting at least 50% of fraudulent transactions, despite its precision advantage.
  • D) Continue the A/B test with additional traffic to Variant B to see if recall improves over time with more data.
  • E) Combine both models into a serving ensemble that uses Variant B's predictions only when Variant A also flags a transaction as fraudulent.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Options Analysis

  • C is Correct: The compliance constraint is a hard requirement: the model must not miss more than 50% of fraudulent transactions. With a recall of 45%, Variant B fails this threshold. Even though precision improved (fewer false positives), the model does not meet regulatory requirements and cannot be promoted.

Distractor Analysis

  • A is Incorrect: Precision and low false positive rate cannot override hard compliance constraints. Promoting a model that misses 55% of fraudulent transactions would be a compliance violation regardless of operational cost savings.
  • B is Incorrect: While recall is often critical in fraud detection, this option ignores the compliance threshold and dismisses the precision improvement without analysis. The correct approach is to evaluate against explicit business constraints, not a single metric in isolation.
  • D is Incorrect: Recall is a property of the model and its decision threshold, not something that improves simply by collecting more data in an A/B test. More data may reduce variance but will not fundamentally change a model's threshold-dependent behavior.
  • E is Incorrect: Ensemble serving at inference time introduces unnecessary complexity and operational risk. Additionally, if Variant B only fires when Variant A also flags a transaction, the combined system's recall cannot exceed Variant A's recall—this does not solve the fundamental problem.

Key Takeaway: A/B testing decisions must evaluate all relevant metrics against explicit business and compliance constraints—not just overall performance rankings.

Question 9 - A/B Testing with Amazon Bedrock for Foundation Models

A customer service company is using Amazon Bedrock to power a chatbot that handles support inquiries. They have identified two foundation models—Anthropic Claude and Amazon Titan Text—and want to compare their response quality in production to determine which model should handle the full traffic. The team needs to run the comparison without building a custom routing infrastructure and wants to capture token usage, latency, and error rates for both models.

Which AWS feature enables this comparison with the least operational overhead?

  • A) Deploy both models to a single SageMaker multi-variant endpoint and manually implement Bedrock Runtime API calls to each variant.
  • B) Create two separate Amazon Bedrock model inference profiles and use an AWS Step Functions workflow to route requests randomly between them while logging metrics to CloudWatch.
  • C) Use Amazon Bedrock model evaluation with automatic model evaluation jobs that run offline and compare the two models on a curated test dataset.
  • D) Set up an Amazon CloudWatch generative AI observability dashboard that correlates model invocation metrics from both Bedrock models run under their respective inference profiles.
  • E) Deploy a custom API Gateway with a Lambda authorizer that routes traffic between two Bedrock model endpoints and logs custom CloudWatch metrics.
Click to reveal Answer & Explanation

Correct Answer: D

Correct Options Analysis

  • D is Correct: Amazon Bedrock model invocation metrics (token usage, latency, error rates, throttling) are published to CloudWatch under the AWS/Bedrock namespace with dimensions including ModelId and InferenceProfileId. The CloudWatch generative AI observability dashboards provide pre-built views that compare these metrics across models. To run live traffic comparison between Bedrock models, the application would need to route traffic between different model IDs, but CloudWatch natively aggregates the comparison metrics.

Distractor Analysis

  • A is Incorrect: SageMaker multi-variant endpoints are for SageMaker-hosted models, not Amazon Bedrock foundation models. Bedrock models are invoked via the Bedrock Runtime API natively, and you cannot deploy Bedrock FMs as SageMaker production variants.
  • B is Incorrect: Using Step Functions to route Bedrock requests introduces unnecessary orchestration latency, custom logging logic, and operational overhead. Bedrock publishes invocation metrics natively without requiring custom routing infrastructure.
  • C is Incorrect: Bedrock model evaluation with automatic jobs runs offline on curated test datasets. While valuable for initial model selection, it does not measure live production traffic performance or serve real user requests concurrently.
  • E is Incorrect: Building custom API Gateway + Lambda routing adds operational complexity (Lambda cold starts, custom metric emission, error handling) when CloudWatch and Bedrock already provide the necessary metrics natively.

Key Takeaway: For comparing Bedrock foundation models in production, leverage CloudWatch generative AI observability with Bedrock's native per-model metrics instead of building custom routing layers.

Question 10 - Rollback Strategy After Failed A/B Test

An online advertising company shifted 80% of traffic to a new click-prediction model (Variant B) after a seemingly successful A/B test. Two hours after the weight update, the team notices a 35% drop in click-through rate and a spike in ModelLatency p99 for Variant B from 50ms to 800ms. The team must restore service quality immediately with minimal disruption.

Which action sequence should the on-call ML engineer execute?

  • A) Wait for the model to auto-correct through SageMaker's dynamic learning behavior, since production traffic will help it adapt over the next few hours.
  • B) Immediately call UpdateEndpointWeightsAndCapacities to shift traffic back to Variant A (e.g., 95% A / 5% B), then investigate whether the latency spike is caused by insufficient capacity or model-serving issues.
  • C) Delete Variant B from the endpoint configuration to remove it from the serving path entirely, then recreate the endpoint with only Variant A.
  • D) Simply increase Variant B's instance count to address the latency spike and shift 100% of traffic to it once latency returns to normal.
  • E) Configure a Step Functions workflow that gradually reduces Variant B's traffic weight over a 30-minute period to avoid abrupt traffic changes.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: The fastest and safest rollback is to adjust production variant weights back to the incumbent model (Variant A). This immediately routes 95% of traffic to the known-good model while retaining 5% on Variant B for diagnosis. The team can then investigate the root cause (capacity shortfall, model architecture issue, data pipeline problem) without disrupting users.

Distractor Analysis

  • A is Incorrect: SageMaker models do not automatically "learn" from production traffic. Once deployed, model weights are fixed. Latency spikes and quality drops do not resolve themselves without intervention.
  • C is Incorrect: Deleting Variant B removes the ability to diagnose the issue. Endpoint recreation introduces downtime and distinct serving infrastructure changes that complicate rollback. The gradual weight adjustment approach preserves diagnostic capability.
  • D is Incorrect: While capacity may be a contributing factor to latency, throwing more compute at the problem without understanding the root cause does not address the 35% drop in click-through rate. Additionally, shifting 100% of traffic to a failing model amplifies business impact.
  • E is Incorrect: A 30-minute gradual transition delays service restoration far too long when business metrics have already dropped by 35%. Emergency rollback requires immediate action, and the Step Functions orchestration adds unnecessary complexity during an incident.

Key Takeaway: Emergency rollback from a failing A/B test means immediately shifting traffic weight back to the known-good variant—retain a small traffic share on the failing variant for post-incident diagnosis.

Exam Tips for Skill 4.1.4

SageMaker Multi-Variant Endpoints: The core mechanism for A/B testing on SageMaker—one endpoint, multiple variants, traffic distributed by weight ratios.

Shadow Variants: Use when real traffic exposure is too risky—the candidate receives traffic copies but never returns responses to users.

Weight and Capacity Go Together: When shifting traffic, always update both variant weights and instance capacities using UpdateEndpointWeightsAndCapacities.

Variant-Level CloudWatch Metrics: Native metrics (InvocationCount, ModelLatency, ModelInvocationErrors) are published with VariantName dimensions—use them for A/B test dashboards and alarms.

Statistical Validity: Formal hypothesis testing (not arbitrary thresholds) determines whether A/B test differences are significant.

Bedrock vs SageMaker A/B: Bedrock FMs are compared using CloudWatch generative AI observability and Bedrock evaluation jobs; SageMaker handles custom model A/B testing via production variants.