Skill 3.3.1: Implement automated deployment strategies and rollback actions

Question 1 - SageMaker Deployment Guardrails with Canary Rollout and Automatic Rollback

A financial services company is deploying a fraud detection model to a SageMaker real-time endpoint. The new model version shows improved precision in offline evaluation, but the reliability team is concerned about potential performance degradation under production traffic patterns. The MLOps team must release the new model so that only 10% of traffic goes to it initially, with automatic reversion to the current model if the error rate exceeds 2% at any point during the first 30 minutes after deployment begins.

Which configuration achieves this with the least operational overhead? (Select ONE)

A) Use SageMaker Deployment Guardrails with a canary traffic shifting strategy, setting the canary percentage to 10%, configuring a CloudWatch Alarm on Invocation5XXErrors with a threshold of 2%, and specifying a baking period of 30 minutes.
B) Deploy the new model to a second SageMaker endpoint, implement application-level routing logic to send 10% of requests to the new endpoint, and write a custom Lambda function that monitors error metrics and switches traffic back if errors exceed 2%.
C) Use CodeDeploy with a canary deployment group targeting the SageMaker endpoint, specifying a 10% traffic shift and an automatic rollback alarm on the error metric.
D) Deploy the new model to the same endpoint using a rolling update strategy, configure a CloudWatch Alarm on ModelLatency, and manually monitor error rates with a dashboard.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

A is Correct: SageMaker Deployment Guardrails provides native blue/green deployment management with configurable traffic shifting strategies (canary, linear, all-at-once), a baking period for validating the new fleet, and automated rollback triggered by CloudWatch Alarms. This requires no custom application code or additional services, minimizing operational overhead.

Distractor Analysis

B is Incorrect: Building a second endpoint with application-level traffic routing introduces significant operational complexity—you must manage two endpoints, write and maintain custom routing code, and build a custom monitoring and rollback mechanism. SageMaker Deployment Guardrails handles all of this natively.
C is Incorrect: CodeDeploy is designed for application code deployment (EC2, Lambda, ECS, on-premises), not for SageMaker model deployments. Using CodeDeploy for SageMaker endpoints is not a supported or valid architecture.
D is Incorrect: A rolling update strategy replaces capacity in batches rather than shifting a small percentage of traffic to a new fleet for validation, and manual monitoring removes the automated rollback protection requirement. Additionally, the alarm targets latency, not the error rate constraint specified.

Key Takeaway: SageMaker Deployment Guardrails with a canary shift and a baking period watched by a CloudWatch Alarm is the native, low-overhead mechanism for gradual rollout with automatic rollback.

Question 2 - Troubleshooting Failed Automatic Rollback

A retail company deployed a new recommendation model version using SageMaker Deployment Guardrails with a linear traffic shifting strategy. The deployment configuration included a CloudWatch Alarm on Invocation4XXErrors and a 45-minute baking period. During the rollout, the error rate exceeded the configured threshold, but the deployment completed successfully and traffic was not rolled back to the old model. The on-call engineer must diagnose why the automated rollback did not trigger.

What is the most likely explanation for this behavior? (Select ONE)

A) The alarm was configured on the wrong metric; automatic rollback requires monitoring a 5XX-class error metric rather than a 4XX-class error metric.
B) The baking period was too long, causing SageMaker to ignore the alarm and complete the deployment after 45 minutes.
C) SageMaker Deployment Guardrails only supports automatic rollback with canary traffic shifting, not linear traffic shifting.
D) The old fleet was terminated before the traffic shift began, so there was no target to roll back to.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

A is Correct: While SageMaker Deployment Guardrails can technically watch any CloudWatch Alarm, Invocation4XXErrors represents client-side errors (e.g., bad payloads, malformed requests). In production ML deployments, 4XX errors are often caused by data quality issues or client misconfigurations unrelated to model quality. If the threshold was set expecting model-specific failures (e.g., 5XX errors from runtime exceptions), the alarm may have triggered on unrelated client errors while the actual model serving infrastructure remained healthy—or the monitoring logic was misinterpreted. The most likely root cause is that the wrong error metric was selected for rollback signaling.

Distractor Analysis

B is Incorrect: The baking period is a validation window during which alarms are actively monitored. A longer baking period does not cause SageMaker to ignore alarms—if an alarm trips during the window, rollback triggers regardless of whether it happens at minute 1 or minute 44.
C is Incorrect: SageMaker Deployment Guardrails supports automatic rollback with all supported traffic shifting strategies (all-at-once, canary, and linear). The shifting shape determines how traffic moves, not whether rollback is available.
D is Incorrect: In SageMaker Deployment Guardrails, the blue fleet (old model) remains active until the baking period completes successfully. It is not terminated before traffic shifting begins, so a rollback target always exists during the deployment.

Key Takeaway: Automatic rollback in SageMaker depends entirely on correctly configured CloudWatch Alarms; verifying the alarm metric, threshold, and evaluation window is the first troubleshooting step when a rollback doesn't trigger.

Question 3 - CodePipeline Failure Stage Diagnosis

An MLOps team has built a CI/CD pipeline using AWS CodePipeline with three stages: Source (AWSCodeCommit), Build (AWSCodeBuild running model training and evaluation), and Deploy (AWSCodeDeploy to a SageMaker endpoint). A recent pipeline execution failed. The error log indicates that the training job completed, but the pipeline stopped before the model was registered to the Model Registry. The team needs to identify which pipeline stage and component to investigate first.

Which pipeline component should the team troubleshoot based on the failure description? (Select ONE)

A) The Build stage's buildspec.yml actions, specifically the step that registers the trained model into SageMaker Model Registry after evaluation.
B) The Source stage's CodeCommit repository, specifically the branch configuration.
C) The Deploy stage's CodeDeploy deployment group, specifically the endpoint configuration.
D) The CodePipeline service role's IAM permissions, specifically the s3:GetObject action.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

A is Correct: The failure occurred after training completed but before the model was registered to the Model Registry. This indicates the failure happened inside the Build stage—specifically in the buildspec.yml commands responsible for evaluating the trained model and calling the RegisterModel API to place it into the SageMaker Model Registry. The Build stage encompasses all commands that run in the build container.

Distractor Analysis

B is Incorrect: The Source stage only clones the repository and passes source artifacts downstream. Since the build stage clearly executed (training job ran), the Source stage completed successfully.
C is Incorrect: The Deploy stage runs after the Build stage. The failure occurred before the model was registered, meaning the pipeline never reached the Deploy stage.
D is Incorrect: If the CodePipeline service role lacked s3:GetObject permissions, the pipeline would fail at the very beginning when trying to retrieve the source artifact from S3 or CodeCommit. The training job running proves this permission is already in place.

Key Takeaway: When diagnosing a pipeline failure, identify exactly where the execution stopped—before troubleshooting, map the failed action to its specific stage (Source, Build, Test, or Deploy).

Question 4 - Event-Driven Retraining Pipeline with Conditional Registration

A healthcare organization processes medical imaging data for a diagnostic model. New labeled data is uploaded to an S3 bucket at irregular intervals whenever hospital partners complete their data-sharing agreements. The MLOps team must ensure the model is retrained automatically when new data arrives, but the retrained model must wait for a human review step before any production deployment.

Which TWO architectural components should the team implement to meet these requirements? (Select TWO)

A) An S3 Event Notification that triggers an AWS Lambda function to call StartPipelineExecution on a SageMaker Pipeline.
B) A fixed hourly EventBridge schedule that executes the SageMaker Pipeline to check for new data.
C) A Condition step in the SageMaker Pipeline that requires manual approval before registering the model.
D) A model registration step in the pipeline that outputs to the SageMaker Model Registry with ModelApprovalStatus set to PendingManualApproval.
E) An AWS Glue crawler that scans for new data and immediately deploys the model with a canary shift.

Click to reveal Answer & Explanation

Correct Answers: A and D

Correct Options Analysis

A is Correct: S3 Event Notifications (via Amazon EventBridge) provide an event-driven trigger for irregular data arrival. When new labeled data is uploaded, S3 emits an event, Lambda receives it, and invokes StartPipelineExecution. This ensures retraining happens exactly when new data arrives—no wasted compute, no missed data.
D is Correct: Registering the model with ModelApprovalStatus set to PendingManualApproval places it in the SageMaker Model Registry in a state that requires human review. A separate deployment pipeline or step that gates on Approved status ensures the model never reaches production without sign-off.

Distractor Analysis

B is Incorrect: A fixed hourly schedule contradicts the irregular data arrival pattern. Hourly checks would execute the pipeline even when no new data exists (wasted compute) and could delay retraining by up to an hour. Event-driven triggering is more efficient for irregular workloads.
C is Incorrect: SageMaker Pipelines does not have a native "Condition step" that supports manual approval. Manual approval is implemented through external mechanisms such as CodePipeline approval actions or by setting the Model Registry approval status to PendingManualApproval and requiring a separate approval action.
E is Incorrect: AWS Glue crawlers are used for data discovery and cataloging, not for triggering model deployments. Additionally, immediate deployment with a canary shift violates the requirement for human approval before production.

Key Takeaway: For irregular data arrival, use event-driven triggers; separate training from deployment by having pipelines end at Model Registry registration with PendingManualApproval.

Question 5 - Automated Testing Gate for Traditional ML Models

A manufacturing company runs a SageMaker Pipeline that trains a predictive maintenance model. The pipeline automatically retrains weekly, but the reliability team has identified that model quality occasionally degrades silently after a new dataset is loaded. The MLOps team must add an automated quality gate that prevents model registration when the trained model's performance drops below a defined threshold on a held-out validation set.

Which configuration should the team implement inside the SageMaker Pipeline? (Select ONE)

A) Add a Condition step after the evaluation step that compares the model's ROC-AUC score against a minimum threshold and routes to registration only when the condition passes.
B) Add an assertion inside the buildspec.yml file that fails the build when the model's evaluation metrics are not in the expected format.
C) Deploy the model to a shadow endpoint, run a batch of synthetic traffic through it, and manually review the results before registration.
D) Use Amazon Bedrock Model Evaluation to automatically score the model against industry benchmarks and block registration if the score falls below 80%.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

A is Correct: SageMaker Pipelines provides a native Condition Step that evaluates the output of an evaluation step (e.g., MetricData from a processing or training job) against a specified threshold. If the model's ROC-AUC exceeds the minimum, the pipeline proceeds to the RegisterModel step; otherwise, it fails and blocks registration. This is the idiomatic way to gate a traditional ML model on a numeric metric threshold.

Distractor Analysis

B is Incorrect: buildspec.yml assertions are part of the CodeBuild stage, which validates code syntax and unit tests. While CodeBuild may run the training job, the model quality gate belongs in the SageMaker Pipeline's condition logic, where it can directly control registration.
C is Incorrect: Shadow deployments and manual reviews involve operational overhead and human intervention, which defeats the purpose of automated quality gating. It also doesn't prevent registration—it only defers the decision.
D is Incorrect: Amazon Bedrock Model Evaluation is designed for foundation model evaluation (generative AI workloads), not for evaluating traditional ML model metrics like ROC-AUC on a predictive maintenance model. It is the wrong service for this use case.

Key Takeaway: For traditional ML pipelines, use SageMaker Pipeline Condition Steps to gate registration on numeric evaluation metrics—this keeps quality enforcement automated and deterministic.

Question 6 - Managing Model Versions with Lineage for Audit

A financial institution must comply with regulatory requirements that mandate full reproducibility of any model that has ever served production traffic. An auditor asks the MLOps team to provide the exact training dataset location, hyperparameters, and evaluation metrics for a model version that was deployed 6 months ago. The team needs a solution that captures this information automatically without manual bookkeeping.

Which TWO AWS services or features should the team use to meet this compliance requirement? (Select TWO)

A) Amazon SageMaker Model Registry to store model versions with their recorded lineage and metadata.
B) MLflow on SageMaker AI to track experiment runs, parameters, and metrics, with the model registry integration recording the lineage from run to deployment.
C) AWS CloudTrail to record every inference request served by the endpoint during the past 6 months.
D) AMAZON SageMaker Model Cards to document the intended use cases and ethical considerations of each model.
E) AWS X-Ray with a service map that traces all downstream API calls from the endpoint.

Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

A is Correct: SageMaker Model Registry catalogs model versions within a model group, storing immutable metadata and lineage details (training job ARN, input S3 URIs, hyperparameters, evaluation metrics) for each registered version. This allows auditors to trace from a deployed model package to its origin.
B is Correct: MLflow on SageMaker AI (managed MLflow) provides experiment tracking that records training runs, parameters, metrics, and artifacts. When integrated with the SageMaker Model Registry or MLflow Registry, it automatically links a registered model version back to the specific MLflow run that produced it, preserving full reproducibility.

Distractor Analysis

C is Incorrect: AWS CloudTrail records API activity (management and data events) but does not capture model training lineage details like hyperparameters, dataset URIs, or evaluation metrics. It is an audit logging service, not an MLOps metadata repository.
D is Incorrect: SageMaker Model Cards (now part of SageMaker Model Registry documentation features) provide governance documentation for model use cases and ethical considerations. They do not automatically capture training lineage or technical metadata.
E is Incorrect: AWS X-Ray provides distributed tracing for request flows through microservices and APIs. It helps debug performance issues but does not record model training lineage or artifact provenance.

Key Takeaway: For audit and reproducibility, always register trained models to a registry and use MLflow experiment tracking to preserve the link between training runs and deployed model versions.

Question 7 - Prompt Versioning and Deployment in CI/CD

A company is building a customer support chatbot using Amazon Bedrock and a foundation model. The team is iterating on prompt design to improve response quality. They need to release prompt changes through the same CI/CD pipeline as their application code, so each prompt version is tested and deployed with full traceability, and the application code refers to prompts by reference rather than embedded text.

Which solution meets these requirements with the least operational overhead? (Select ONE)

A) Use Amazon Bedrock Prompt Management to create versioned prompts with variables, test variants in parallel, and reference prompt versions by ARN in the application code.
B) Store prompts as JSON files in a Git repository, parse them at build time, and inject them into the application as environment variables.
C) Create a Lambda layer containing all prompt text, version it with the application, and reference prompts by function handles.
D) Store prompts in an S3 bucket, use a Lambda function to fetch them at runtime, and track allowed changes with S3 versioning.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

A is Correct: Amazon Bedrock Prompt Management is the AWS-native solution for managing prompts as versioned resources. It allows teams to create prompt versions with variables, test them as comparable variants (A/B testing), track changes, and reference prompts by their ARN in application code. Integrating with CodePipeline or SDK deployments allows prompt version releases to flow through the same CI/CD process with full auditability. No custom infrastructure is required.

Distractor Analysis

B is Incorrect: Storing prompts as Git-tracked JSON files addresses version control but requires the team to build and maintain parsing logic, templating systems, and variable substitution at build time. This adds custom code and reduces integration with Bedrock-native invocation patterns.
C is Incorrect: Lambda layers are designed for shared code libraries and dependencies, not for prompt management. Versioning prompts via a Lambda layer mixes concerns and lacks native prompt testing features.
D is Incorrect: S3 versioning provides file-level version history, but it does not offer structured prompt versions with variables, comparison testing, or integration with Bedrock invocation. Runtime fetching also adds unnecessary latency and API calls.

Key Takeaway: Amazon Bedrock Prompt Management is the native service for versioning, testing, and releasing prompts in AI workflows—prefer it over homegrown Git/S3 solutions when using Bedrock.

Question 8 - Foundation Model Fine-Tuning Deployment Automation

A company fine-tunes a foundation model in Amazon Bedrock for domain-specific document summarization. After fine-tuning, the model produces a unique model ID that must be used by inference applications. The MLOps team needs to automate the deployment pipeline so each fine-tuned model version is versioned, tested, and promoted to production by immutable identifier rather than editing configuration in place.

Which TWO steps should the team include in their CI/CD pipeline to achieve this? (Select TWO)

A) Store the fine-tuned model ID in a parameter store or S3 artifact within the version's pipeline output, and retrieve that ID at deployment time.
B) Use SageMaker Model Registry to track each fine-tuned Bedrock model as a versioned entry, recording the Bedrock model ID and lineage metadata.
C) Store the fine-tuned model ID in a shared environment variable that all application versions read at runtime, updating it manually after each fine-tuning run.
D) Automate the inference application's model reference to point to a stable alias or latest version tag that Bedrock automatically resolves.
E) Include an evaluation step in the pipeline that tests the fine-tuned model on a held-out summarization dataset before allowing the version to progress to production.

Click to reveal Answer & Explanation

Correct Answers: B and E

Correct Options Analysis

B is Correct: Though SageMaker Model Registry is typically used for models on SageMaker infrastructure, its versioning and metadata capabilities can be extended to track Bedrock fine-tuned model IDs. Alternatively, teams can use a custom registry. In the AWS ecosystem, recording the Bedrock model ID, lineage (base model, training data, hyperparameters), and approval status in a versioned registry ensures that deployments reference immutable identifiers. The key point is that the model ID must be versioned and traced.
E is Correct: An evaluation step in the CI/CD pipeline is essential for automated quality gating. Fine-tuned foundation models should be tested on a representative held-out dataset (e.g., ROUGE scores for summarization) compared to the previous version. Only versions that pass the quality threshold proceed to production.

Distractor Analysis

A is Incorrect: Storing model IDs in parameter stores is operationally acceptable but incomplete—parameter stores do not inherently provide the lineage tracking, approval workflows, or version comparison features that a model registry offers. It doesn't establish an immutable, versioned record suitable for audits.
C is Incorrect: Using a shared environment variable updated manually introduces human error risk and prevents reproducibility. It also violates the "immutable identifier" requirement since the same variable reference could point to different models at different times, breaking auditability.
D is Incorrect: Amazon Bedrock does not automatically resolve a "latest" alias for fine-tuned models. The inference application must explicitly specify the model ID or provisioned throughput ARN. This option assumes a feature that doesn't exist.

Key Takeaway: Fine-tuned foundation models need versioned registry tracking and automated evaluation gates just like traditional ML models; deployments should refer to immutable model IDs rather than mutable configurations.

Question 9 - RAG System Update Pipeline with Incremental Resync

A company maintains a retrieval-augmented generation (RAG) system using Amazon Bedrock Knowledge Bases with an S3 data source. Their document corpus updates daily as team members edit internal documentation pages. The MLOps team needs to implement a pipeline that keeps the vector index current with minimal embedding costs and operational overhead, reflecting only documents that have changed since the last update.

Which solution best meets these requirements? (Select ONE)

A) Trigger a Bedrock Knowledge Bases ingestion job (StartIngestionJob) via S3 Event Notifications for object changes, allowing the service to incrementally sync only changed documents.
B) Run a daily scheduled Lambda function that fetches all documents from S3, deletes the vector index, and rebuilds it from scratch using a custom embedding pipeline.
C) Configure an EventBridge event that copies the entire S3 bucket to a new location and runs a full re-indexing job on the copy.
D) Use a SageMaker Pipeline with a processing step that reads all documents, chunks them, and writes vectors to OpenSearch, with an S3 event trigger to run the pipeline hourly.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

A is Correct: Amazon Bedrock Knowledge Bases supports incremental ingestion (resync) where the service processes only documents that have been added, modified, or deleted since the last ingestion run. Triggering StartIngestionJob via S3 Event Notifications (routed through EventBridge or Lambda) ensures timely updates with minimal embedding costs—only changed documents are re-parsed and re-embedded.

Distractor Analysis

B is Incorrect: Deleting the entire vector index and rebuilding from scratch on every daily run wastes significant compute and embedding API costs. It also introduces downtime or degraded retrieval quality during the rebuild window, and requires custom embedding pipeline maintenance.
C is Incorrect: Copying the entire S3 bucket to a new location and running a full re-index replicates the most expensive possible approach. It provides no incremental benefit and creates unnecessary S3 storage costs.
D is Incorrect: Using a custom SageMaker Pipeline for document processing requires building and maintaining custom chunking, embedding, and indexing code. Bedrock Knowledge Bases handles all of this natively. Running hourly also introduces unnecessary compute even when no documents changed.

Key Takeaway: For RAG systems using Bedrock Knowledge Bases, prefer event-driven incremental ingestion jobs—they automatically sync only changed documents, minimizing cost and operational overhead.

Question 10 - CodeDeploy Rollback Configuration for SageMaker Endpoint

A company runs a machine learning inference application packaged as a Docker container deployed to Amazon ECS. The MLOps team uses AWS CodeDeploy to manage blue/green deployments with automatic rollback. However, a recent deployment to ECS completed with a new model version even though the ModelLatency metric exceeded acceptable thresholds during the rollout.

What is the most likely cause of this failed rollback? (Select ONE)

A) CodeDeploy automatic rollback requires a CloudWatch Alarm to be configured and associated with the deployment group; the alarm was missing or misconfigured.
B) CodeDeploy only supports manual rollbacks for ECS blue/green deployments.
C) The ModelLatency metric is not available in Amazon CloudWatch for custom application metrics.
D) Automatic rollback is only supported for EC2/On-Premises deployment targets, not ECS.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

A is Correct: AWS CodeDeploy supports automatic rollback for blue/green deployments to ECS (and Lambda/EC2) when CloudWatch Alarms are configured and associated with the deployment group. If no alarm is configured—or the alarm is misconfigured (wrong threshold, insufficient evaluation period, wrong metric name)—CodeDeploy has no signal to detect the degradation and will complete the deployment despite the elevated latency.

Distractor Analysis

B is Incorrect: CodeDeploy supports both automatic rollback (triggered by CloudWatch Alarms during the deployment's traffic shifting window) and manual rollback for ECS blue/green deployments.
C is Incorrect: Custom application metrics like ModelLatency can absolutely be published to CloudWatch using the PutMetricData API or CloudWatch agent. Latency is a common application performance metric available in CloudWatch for monitoring.
D is Incorrect: Automatic rollback with CloudWatch Alarms is supported for ECS blue/green deployments, as well as Lambda and EC2/On-Premises. The target platform does not determine automatic rollback availability.

Key Takeaway: Automatic rollback in CodeDeploy—like SageMaker Deployment Guardrails—depends on properly configured and linked CloudWatch Alarms; without them, degraded deployments complete without intervention.

Summary Table: Key Concepts for Skill 3.3.1

Concept	Key Points
Traffic Shifting Strategies	Canary (small slice first), Linear (even steps), All-at-once (immediate full shift)
Baking Period	Validation window where alarms are actively monitored before old fleet removal
Automatic Rollback Trigger	CloudWatch Alarm must be configured and linked; missing alarm = no rollback
Troubleshooting Pipeline Failures	Identify the exact stage/action where execution stopped before investigating
Model Registry	Stores versioned models with lineage, metadata, and approval status
Prompt Management	Version prompts as resources with variables; reference by ARN/version ID
RAG Update Pipelines	Incremental resync processes only changed documents; full re-index for embedding/chunking changes
Monitoring Metrics for Rollback	Prefer model-specific metrics (5XX errors, ModelLatency, Invocation4XX) aligned with actual failure modes