Skip to content

Skill 3.1.1: Implement automated deployment strategies and rollback actions

Task 3.3: Deployment and Orchestration of ML and AI Workflows - Implement automated orchestration and continuous integration and continuous delivery (CI/CD) pipelines for MLOps and AI workloads

Core Concept & Objective

Automated deployment replaces a model/prompt/agent in production with zero or minimal downtime, using a controlled traffic shift. Rollback automatically reverts to the last known good version when a defined signal (alarm, test failure, approval rejection) indicates the new version is unhealthy.
Primary AWS objective: repeatable, auditable, reversible releases for ML and AI workloads.

Deployment Strategies – SageMaker Endpoints

Strategy Traffic Behavior Best For Rollback Support
All-at-once 100% shift immediately Dev/test, low-risk updates Blue/green auto-rollback if alarms configured
Canary Small % first, then remainder after wait High-risk production models Auto-rollback via CloudWatch alarms
Linear Equal increments over time (e.g., 10% every 5 min) Predictable, gradual risk reduction Auto-rollback via CloudWatch alarms
Rolling update Replaces instances in batches Capacity-constrained, no extra fleet Limited auto-rollback; verify support
Shadow variant Copy of traffic, responses not returned Test new model without user impact No user impact; not a rollback mechanism
A/B test (multi-variant) Split traffic between variants Compare business metrics Manual analysis; no automatic rollback
  • Blue/green is the foundation: new fleet (green) is provisioned alongside old (blue). Traffic shifts according to policy. After baking period, old fleet is terminated if no alarm trips.
  • Deployment guardrails (SageMaker) enable canary/linear/all-at-once for real-time endpoints. Configure via DeploymentConfig in UpdateEndpoint.
  • Baking period = TerminationWaitInSeconds. Alarms are evaluated during this window. Too short = missed failures.
  • Auto-rollback requires AutoRollbackConfiguration with one or more CloudWatch alarms. No alarm = no rollback signal.
  • Shadow variants receive a copy of production traffic but their responses are discarded. Good for offline validation, not for rollback.
  • Multi-variant endpoints support A/B testing with explicit traffic distribution. Rollback is manual.

Exam Gotcha: Managed rollout shapes do not apply to every hosting option (e.g., serverless, async). Confirm the endpoint type supports canary/linear before designing the release.

Rollback Actions – Mechanisms & Triggers

Service Rollback Trigger Mechanism
SageMaker Deployment Guardrails CloudWatch alarm enters ALARM during baking Traffic returns to old fleet; old fleet retained
CodeDeploy (Lambda/ECS/EC2) CloudWatch alarm, deployment failure, manual Redeploys last known good revision; for Lambda shifts alias back
SageMaker Model Registry Approval status = Rejected Pipeline conditional step blocks deployment
Bedrock Agent Aliases Manual or automated decision Update alias to point to previous agent version
Bedrock Prompt Management Manual or automated decision Application reverts to previous prompt version
CodePipeline Manual approval rejection or failed stage Re-run previous pipeline execution or trigger CodeDeploy rollback
  • CodeDeploy rollback is configured at deployment group level: RollbackConfiguration with AlarmConfiguration. Rollback on alarm, failure, or manual.
  • Lambda traffic shifting uses aliases. Canary/linear configs shift a percentage of invocations. Rollback redeploys previous Lambda version.
  • ECS blue/green uses load balancer listener rules. Rollback switches listener back to original target group.
  • EC2 supports in-place (downtime possible) or blue/green (zero downtime). In-place rollback re-deploys previous revision.
  • Model Registry is the audit handoff: each model package version records lineage, metadata, and approval status (PendingManualApproval, Approved, Rejected). Pipeline reads status before deployment.

Exam Gotcha: An automatic rollback is only as good as the alarm that triggers it. A rollout with no alarm configured will complete a bad deployment as confidently as a good one.

CI/CD Pipeline Integration – Key Services & Handoffs

Service Role in Deployment/Rollback
AWS CodePipeline Orchestrates stages (Source, Build, Test, Deploy). Manual approval action gates production. Invoke Lambda to call SageMaker UpdateEndpoint.
AWS CodeBuild Runs unit tests, container builds, evaluation scripts. Failed build stops pipeline.
AWS CodeDeploy Executes rollout and rollback for EC2/ECS/Lambda. Manages traffic shifting and alarm monitoring.
AWS CodeCommit Hosts Git repositories inside AWS. Source for pipeline.
AWS CodeConnections Secure connection to external Git (GitHub, GitLab, Bitbucket) without storing credentials.
Amazon EventBridge Triggers pipeline on model approval, new data, schedule, or training completion.
SageMaker Pipelines Orchestrates ML steps (processing, training, evaluation, registration). Ends at Model Registry, not deployment.
SageMaker Model Registry Stores versioned models with approval status. Deployment pipeline looks up approved version.
Amazon Bedrock Prompt Management Versioned prompts with variables. Release prompt version, not inline text.
Amazon Bedrock Agents Agent versions + aliases. Alias points to a version; rollback = update alias.
  • Separation of concerns: Training pipeline produces a model and registers it. Deployment pipeline consumes an approved model and deploys it. This allows a human or test gate between training and production.
  • Automated testing gates: Code tests (deterministic) fail build. Traditional ML models gate on metric threshold vs held-out set. Generative AI gates on evaluation scores (e.g., LLM-as-a-judge, faithfulness, toxicity) compared to baseline.
  • Retraining triggers: Schedule (fixed cadence), Event (new data lands, metric threshold crossed), Manual (while building trust). Pipeline should end at registration.

Security, Compliance & Audit

  • IAM roles: Separate execution roles for SageMaker training, inference, CodePipeline, CodeDeploy. Least privilege.
  • Encryption: KMS for model artifacts in S3, ECR images, endpoint volumes. TLS for data in transit.
  • Network: VPC endpoints for SageMaker, S3, ECR, CloudWatch to keep traffic private.
  • Audit: Model Registry lineage, CloudTrail for API calls, CloudWatch Logs for endpoint invocations, CodeDeploy deployment history.
  • Compliance: Approval status in Model Registry provides evidence of who approved what and when.

Decision Matrix – When to Use X vs Y

Requirement Recommended Strategy Why
Zero downtime + easy rollback Blue/green (canary/linear) New fleet ready; rollback shifts traffic back
Minimize blast radius Canary Small % first; alarm triggers rollback
Predictable gradual shift Linear Even increments; controlled exposure
Fastest deployment, low risk tolerance All-at-once Immediate; use only with alarms and low-risk
No extra capacity during update Rolling update Replaces instances in batches
Test new model silently Shadow variant Copy traffic, responses discarded
Compare business metrics A/B test (multi-variant) Split traffic, analyze results
Gate on model quality Model Registry approval Human/automated approval before deploy
Gate on code quality CodeBuild tests Unit/integration tests in pipeline
Gate on generative quality Bedrock Model Evaluation Score prompts vs baseline; block regression

Anti-Patterns & Exam Gotchas

No CloudWatch alarm = no automatic rollback. SageMaker and CodeDeploy require alarms to trigger rollback.

Baking period too short = failures missed before old fleet terminated.

Manual approval is not a rollback strategy. It gates deployment but does not automatically revert a bad release.

Deploying without Model Registry = no lineage, no audit trail, no versioned rollback target.

Ignoring prompt/agent versioning = behavior changes cannot be explained or rolled back.

Using All-at-once in production without alarms = high blast radius; a single bad model affects all users.

Assuming deployment guardrails work everywhere = serverless/async endpoints may not support canary/linear. Verify first.

Confusing retraining with redeployment. Retraining produces a new model; deployment releases it. Keep pipelines separate.