Skill 3.3.1: Implement automated deployment strategies and rollback actions
| Question | Answer & Explanation |
|---|---|
| ### Question 1 - SageMaker Canary Deployment & Auto-Rollback A machine learning team is deploying a new version of a sentiment analysis model to an Amazon SageMaker real-time endpoint. They want to route 10% of traffic to the new model first, hold it for 5 minutes, and automatically roll back if the error rate exceeds 2%. The solution must minimize operational overhead and use native AWS features. Which deployment strategy and configuration should the team implement? - A) Canary traffic shift with AutoRollbackConfig monitoring Invocation5XXErrors. - B) Linear traffic shift with AutoRollbackConfig monitoring CPUUtilization. - C) Blue/Green deployment with AWS CodeDeploy managing EC2 instances. - D) All-at-once traffic shift with manual monitoring of CloudWatch metrics. |
Correct Answer: A Correct Answer Analysis: - A is Correct: A canary deployment shifts a small percentage of traffic (e.g., 10%) to the new model fleet, holds it for a baking period, and then shifts the remainder. SageMaker Deployment Guardrails natively support AutoRollbackConfig, which can monitor a CloudWatch Alarm on Invocation5XXErrors to automatically revert traffic to the old fleet if errors exceed the threshold.Distractor Analysis: - B is Incorrect: A linear traffic shift moves traffic in evenly stepped increments, not a single small slice followed by the remainder. Additionally, monitoring CPUUtilization indicates compute load, not model invocation errors.- C is Incorrect: Using AWS CodeDeploy on EC2 instances introduces massive operational overhead compared to using SageMaker's native managed deployment guardrails for endpoints. - D is Incorrect: All-at-once traffic shifting exposes 100% of traffic immediately, risking outages, and manual monitoring violates the automated rollback requirement. > Exam Tip / AWS Best Practice: Use SageMaker Deployment Guardrails with a Canary traffic shift and CloudWatch Alarms on invocation errors (5XX) for automated, low-risk model rollouts. |
| ### Question 2 - Linear Traffic Shifting & Baking Periods An e-commerce company is updating its recommendation model hosted on an Amazon SageMaker real-time endpoint. The deployment must gradually shift traffic to the new model in 20% increments every 10 minutes. If any alarms trip during the baking periods, traffic must automatically revert to the original model. Operational overhead must be kept to a minimum. Which TWO actions must be taken to meet these requirements? (Select TWO) - A) Configure a linear traffic shift in the SageMaker endpoint deployment. - B) Deploy a second SageMaker endpoint and use a Route 53 weighted routing policy. - C) Configure AutoRollbackConfig with Amazon CloudWatch Alarms for ModelLatency and Invocation5XXErrors. - D) Use AWS CodeDeploy to manage the traffic shifting between the endpoints. - E) Write a custom AWS Lambda function triggered by EventBridge to monitor and shift traffic. |
Correct Answers: A and C Correct Answer Analysis: - A is Correct: A linear traffic shifting strategy shifts traffic in equal percentage increments (e.g., 20%) over defined time intervals, meeting the gradual rollout requirement natively within SageMaker. - C is Correct: AutoRollbackConfig is the native SageMaker feature that links CloudWatch Alarms to the deployment baking periods. If an alarm trips, SageMaker automatically handles the rollback.Distractor Analysis: - B is Incorrect: Using a second endpoint and Route 53 weighted policies introduces unnecessary infrastructure and manual routing complexity, violating the least operational overhead constraint. - D is Incorrect: AWS CodeDeploy is designed for EC2, ECS, and Lambda deployments, not native SageMaker endpoint traffic shifting. - E is Incorrect: Building custom Lambda functions to manage traffic shifting reinvents a feature already natively supported by SageMaker Deployment Guardrails. > Exam Tip / AWS Best Practice: SageMaker Deployment Guardrails handle traffic shifting and automated rollbacks natively, eliminating the need for external DNS routing or custom monitoring scripts. |
| ### Question 3 - Troubleshooting Missing Rollback Signals A data science team deployed a new model version using a canary deployment strategy on Amazon SageMaker. During the canary baking period, the new model started returning 500 errors for 50% of requests. However, SageMaker marked the deployment as successful, shifted 100% of traffic, and terminated the old fleet. The team needs to identify the root cause to prevent this in future deployments. What is the MOST likely cause of this issue? - A) The traffic shifting step size was set too small. - B) No CloudWatch Alarm was configured in the AutoRollbackConfig to monitor Invocation5XXErrors. - C) The endpoint was deployed without an Application Auto Scaling policy. - D) The baking period was configured to be 0 minutes. |
Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker Deployment Guardrails rely entirely on Amazon CloudWatch Alarms to trigger an automated rollback during a baking period. If no alarm is configured for Invocation5XXErrors (or similar metrics), SageMaker has no signal that the deployment is failing. Once the baking period expires, it assumes success and completes the rollout.Distractor Analysis: - A is Incorrect: The step size (e.g., 10% vs 5%) controls the pace of the rollout, but an absence of alarms means no rollback will occur regardless of step size. - C is Incorrect: Application Auto Scaling policies adjust instance counts based on traffic load; they do not trigger deployment rollbacks or monitor model invocation errors. - D is Incorrect: If the baking period was 0 minutes, the deployment would skip the canary phase entirely and go straight to 100%, but the question states the deployment was marked successful after the baking period. > Exam Tip / AWS Best Practice: A managed rollout without a linked CloudWatch Alarm is a blind deployment—SageMaker will complete a bad deployment as confidently as a good one. |
| ### Question 4 - Choosing a Rolling Update Strategy An AI startup is deploying a fine-tuned Foundation Model to a SageMaker real-time endpoint. Due to the large size of the model artifacts, the team wants a deployment strategy that replaces instances in batches rather than shifting traffic percentages. The old instances should only be terminated after the new batches pass their health checks. They want to use the least operational overhead possible. Which native SageMaker deployment strategy meets this requirement? - A) Canary - B) Linear - C) Rolling - D) All-at-once |
Correct Answer: C Correct Answer Analysis: - C is Correct: A rolling update strategy brings up new instances in batches, moves traffic to them, and then terminates the old instances in batches. This is highly efficient for large model artifacts as it maintains capacity and updates the fleet without the need to run a full duplicate fleet simultaneously for traffic shifting. Distractor Analysis: - A is Incorrect: Canary deployments shift a small percentage of traffic to a full new green fleet, hold, then shift the rest. It does not replace instances in batches. - B is Incorrect: Linear deployments shift traffic in even, incremental steps to a full new green fleet, rather than replacing instances in batches. - D is Incorrect: All-at-once shifts 100% of traffic to the new fleet immediately, offering no gradual rollout or batching safety. > Exam Tip / AWS Best Practice: Use a Rolling deployment when you want to update instances in batches (useful for large artifacts) versus Canary/Linear which require bringing up a complete parallel fleet to shift traffic to. |
| ### Question 5 - Configuring AutoRollbackConfig for Latency An MLOps engineer is configuring a deployment for a SageMaker real-time endpoint. They need to ensure that if the new model version causes latency to spike during the first 10 minutes of a canary deployment, all traffic is shifted back to the old model automatically. The old model's instances must be retained to handle the reverted traffic. How should the engineer configure this in SageMaker? - A) Create a CloudWatch Alarm on ModelLatency and attach it to the AutoRollbackConfig of the endpoint deployment. - B) Configure an Application Auto Scaling policy on the endpoint that scales in when ModelLatency increases. - C) Set up an EventBridge rule to monitor CloudWatch Alarms and trigger a Lambda function to update the endpoint weights. - D) Use AWS CodeDeploy to manage the SageMaker endpoint and configure a rollback alarm. |
Correct Answer: A Correct Answer Analysis: - A is Correct: AutoRollbackConfig natively monitors specified CloudWatch Alarms during the baking period of a deployment. If ModelLatency crosses the alarm threshold, SageMaker automatically routes traffic back to the old fleet (blue) and preserves those instances to handle the reverted load.Distractor Analysis: - B is Incorrect: Application Auto Scaling adjusts the number of instances based on metrics like CPU or concurrent invocations, but it cannot trigger a model rollback or revert traffic routing. - C is Incorrect: Using EventBridge and Lambda to manually update endpoint weights is a custom, high-overhead solution that reinvents SageMaker's native AutoRollbackConfig feature.- D is Incorrect: AWS CodeDeploy does not natively integrate with SageMaker endpoint deployment traffic shifting or AutoRollbackConfig for managed ML models.> Exam Tip / AWS Best Practice: AutoRollbackConfig is the native SageMaker feature that pairs with CloudWatch Alarms to automatically preserve the old fleet and revert traffic during a failed deployment. |
| ### Question 6 - Traffic Shifting & Fleet Lifecycle An organization is deploying a critical credit scoring model. They configure a linear deployment strategy to shift 25% of traffic to the new model, wait 10 minutes, shift another 25%, wait 10 minutes, and so on until 100% is reached. If no alarms trip during the entire process, the deployment is marked as successful. What is the final state of the old and new instances at the conclusion of this successful deployment? - A) The old instances are kept as a backup for 24 hours before termination. - B) The old instances are terminated immediately after the final traffic shift completes successfully. - C) The old instances are converted to the new model version in place. - D) Both the old and new instances run concurrently to balance the load. |
Correct Answer: B Correct Answer Analysis: - B is Correct: In a SageMaker blue/green deployment using a linear traffic shift, the old fleet (blue) is only kept alive to handle traffic if a rollback is triggered. Once 100% of traffic is successfully shifted to the new fleet (green) and the baking period passes without an alarm, the old fleet instances are terminated. Distractor Analysis: - A is Incorrect: SageMaker does not retain the old fleet as a backup automatically after a successful deployment; they are terminated to stop incurring costs. - C is Incorrect: Blue/green deployments bring up an entirely new fleet with the new model. They do not update the model on the existing instances in place. - D is Incorrect: Running both fleets indefinitely would double the cost. The old fleet is removed once the shift is complete. > Exam Tip / AWS Best Practice: In a successful SageMaker deployment guardrails rollout, the blue (old) fleet is terminated to free up resources once the green (new) fleet reaches 100% traffic. |
| ### Question 7 - Selecting Metrics for Rollback Alarms A healthcare company uses Amazon SageMaker to host a medical imaging classification model. The MLOps team is deploying a new version and wants to configure a rollback alarm. The new model is generally accurate but occasionally experiences high latency because it requires more compute per invocation than the previous version, which breaks the application's strict SLA. Which CloudWatch metric is the MOST appropriate to monitor in the AutoRollbackConfig to automatically trigger a rollback? - A) CPUUtilization - B) Invocation5XXErrors - C) ModelLatency - D) DiskUtilization |
Correct Answer: C Correct Answer Analysis: - C is Correct: ModelLatency measures the time it takes for the model to process an invocation and return a response. If the new model is taking too long and breaking the application's SLA, an alarm on ModelLatency will trigger the automated rollback to the faster, previous version.Distractor Analysis: - A is Incorrect: CPUUtilization measures the compute load on the instance, but a model could be computationally efficient yet structurally slow, or the instance could simply be maxing out CPU without necessarily violating latency SLAs if it is queuing properly.- B is Incorrect: Invocation5XXErrors tracks hard failures (like internal server errors or model timeouts), not performance degradation or high latency.- D is Incorrect: DiskUtilization tracks storage usage on the instance, which is irrelevant to model processing speed or SLA latency thresholds.> Exam Tip / AWS Best Practice: Match the CloudWatch Alarm metric to the failure mode: use Invocation5XXErrors for hard crashes, and ModelLatency for performance/SLA degradation. |
| ### Question 8 - Comparing Canary and Linear Strategies A retail company's ML team is updating their SageMaker real-time endpoint. The deployment must ensure that exactly 5% of traffic is routed to the new model for a 5-minute period. If the new model performs without errors, the remaining 95% of traffic should be shifted immediately. The team wants to automate this process entirely within SageMaker with zero custom code. Which configuration correctly implements this requirement? - A) Configure a linear deployment with 5% steps and a 5-minute interval. - B) Configure a canary deployment with 5% traffic for 5 minutes, followed by the remaining 95%. - C) Configure an all-at-once deployment and rely on CloudWatch Alarms to manually shift traffic back. - D) Configure a rolling deployment and use Route 53 to shift 5% of traffic. |
Correct Answer: B Correct Answer Analysis: - B is Correct: A canary deployment specifically handles the "small slice, then the rest" pattern. It routes a small percentage of traffic (e.g., 5%) to the new fleet, waits for a baking period (e.g., 5 minutes), and if no alarms trip, shifts the remaining 95% in one final step. Distractor Analysis: - A is Incorrect: A linear deployment shifts traffic in even increments (e.g., 5%, then 10%, then 15%, etc.), which does not match the requirement of shifting the remaining 95% all at once after the initial test slice. - C is Incorrect: All-at-once shifts 100% of traffic immediately, offering no initial test slice and violating the zero custom code constraint by relying on manual intervention. - D is Incorrect: A rolling deployment replaces instances in batches and does not utilize percentage-based traffic shifting in the way described. > Exam Tip / AWS Best Practice: Canary = test a small slice, then shift the rest. Linear = shift in even, steady increments over time. |
| ### Question 9 - CI/CD Integration for Model Rollbacks You are an MLOps engineer designing a CI/CD pipeline for an NLP model. The pipeline builds the model, registers it in the SageMaker Model Registry, and deploys it to a real-time endpoint. The deployment must not drop any traffic and must automatically roll back if the Invocation4XXErrors metric exceeds 5% during the baking period. The team requires the MOST operationally efficient solution.Which approach meets these requirements? - A) Deploy the new model to a new endpoint and use a Route 53 weighted record set to shift traffic, using CloudWatch to manually trigger a rollback. - B) Update the existing SageMaker real-time endpoint with a canary deployment strategy, specifying a CloudWatch Alarm for Invocation4XXErrors in the AutoRollbackConfig.- C) Deploy the new model using a rolling update strategy and rely on the instance health checks to automatically revert bad models. - D) Use AWS CodeDeploy to deploy the model to a fleet of EC2 instances behind an Application Load Balancer to utilize CodeDeploy's rollback features. |
Correct Answer: B Correct Answer Analysis: - B is Correct: Updating the existing SageMaker endpoint natively handles blue/green traffic shifting without dropping traffic. Using a canary strategy with AutoRollbackConfig monitoring Invocation4XXErrors fulfills all requirements with the least operational overhead natively in SageMaker.Distractor Analysis: - A is Incorrect: Using Route 53 and manual rollbacks introduces significant operational overhead and custom logic, failing the efficiency requirement. - C is Incorrect: Rolling updates replace instances in batches but do not utilize traffic shifting or AutoRollbackConfig based on CloudWatch metrics like Invocation4XXErrors to protect live traffic.- D is Incorrect: Managing EC2 instances and an ALB for model serving reinvents SageMaker's managed infrastructure, drastically increasing operational overhead. > Exam Tip / AWS Best Practice: Leverage SageMaker's native UpdateEndpoint API with deployment guardrails to handle blue/green traffic shifting and automated rollbacks without external infrastructure. |
| ### Question 10 - Automated Response to Alarm Triggers A global media company is deploying a recommendation model to a SageMaker real-time endpoint. The engineering team uses a linear deployment strategy, shifting traffic in 20% increments with 10-minute baking periods. During the second shift (40% traffic), a CloudWatch Alarm trips because of a high Invocation5XXErrors rate.What is the IMMEDIATE automated response by SageMaker when the alarm trips during this linear deployment? - A) SageMaker pauses the deployment and waits for an engineer to manually approve the next shift. - B) SageMaker terminates the new instances and immediately scales up the old instances to 100%. - C) SageMaker automatically rolls back the traffic to the previous healthy state (old fleet) and terminates the new instances. - D) SageMaker completes the deployment to 100% and then triggers the rollback. |
Correct Answer: C Correct Answer Analysis: - C is Correct: When a CloudWatch Alarm linked in AutoRollbackConfig trips during a baking period, SageMaker immediately halts the deployment, routes 100% of traffic back to the old (blue) fleet to ensure stability, and subsequently terminates the failing new (green) instances.Distractor Analysis: - A is Incorrect: SageMaker Deployment Guardrails are designed for fully automated rollbacks; they do not pause and wait for human intervention when an alarm trips. - B is Incorrect: SageMaker does not need to "scale up" the old instances because the old fleet is kept fully provisioned and running during the entire baking period precisely to handle a rollback. - D is Incorrect: Continuing the deployment to 100% despite a tripped alarm would exacerbate the error rate, defeating the purpose of the guardrail. > Exam Tip / AWS Best Practice: When an alarm trips during a baking period, SageMaker immediately reverts traffic to the blue fleet and terminates the green fleet—no manual intervention is required. |