Skip to content

3.3

The study notes below focus exclusively on Task 3.3 (Use automated orchestration tools to set up CI/CD pipelines). They synthesize the official task knowledge/skills list with the exam-review transcript. All hands-on exercises are deliberately designed for near-zero ongoing cost: every resource is created, exercised once (or for <5 minutes), and immediately deleted. No free-tier assumptions are made; the cheapest possible compute sizes, shortest timeouts, and S3/Lambda-centric patterns are used. Total expected bill for a careful student who cleans up is well under $1–2.

1. Core Concepts You Must Master

MLOps + CI/CD mapping

  • Version control → source stage
  • Continuous integration = compile + unit/integration tests (CodeBuild)
  • Continuous delivery = produce deployable artifact
  • Continuous deployment = automatic promotion (CodeDeploy / SageMaker endpoints / Step Functions)
  • Model governance = approval gates, model registry, EventBridge-triggered retrain

Primary AWS services & quotas (memorize the “why” and the hard limits)

  • CodePipeline: orchestrates stages (Source → Build → Test → Deploy). 30-day free pipeline then ~$1/pipeline/month + $0.002–0.005 per action execution. Soft limit 1000 pipelines/account.
  • CodeBuild: runs buildspec.yml. Use BUILD_GENERAL1_SMALL only; 60-minute max timeout (set to 5–10 min).
  • CodeDeploy: deployment strategies (AllAtOnce, Blue/Green, Canary, Linear, Rolling). Hooks (BeforeAllowTraffic, AfterAllowTraffic) + CloudWatch alarms for automatic rollback.
  • SageMaker Pipelines / Model Building Pipelines: native ML steps (Processing, Training, Transform, RegisterModel, Condition, Callback). Integrates with CodePipeline or runs standalone.
  • EventBridge: watches SageMaker Model Package State Change, Training Job State Change, etc., to trigger retrain or inference.
  • Step Functions / MWAA (Airflow): long-running or nested orchestration; prefer Step Functions when Lambda would sit idle.
  • Supporting: CodeCommit/GitHub/GitLab (source), S3 (artifacts), SAM/CloudFormation (infra-as-code), SageMaker Model Registry.

Deployment strategies (exam loves these)

  • Serverless/Lambda: All-at-once, Canary (10 %/5 min then 100 %), Linear (10 % every n minutes).
  • EC2/ECS: Blue/Green, Rolling, Rolling with extra batch, Immutable.
  • Always pair with alarms + automatic rollback and pre-/post-traffic hooks.

Git-flow patterns

  • GitHub Flow / GitFlow → feature branches → PR → CodeBuild validation → merge to main → pipeline.
  • Manual approval actions or CodePipeline “approval” stage before prod.

Automated tests inside the pipeline

  • Unit → CodeBuild buildspec
  • Integration / contract → same CodeBuild or separate stage
  • End-to-end / load → optional CodeBuild + Lambda or Glue job
  • Model quality gates → SageMaker Model Monitor or custom processing step that fails the pipeline.

Retrain loops

  • EventBridge rule on data arrival (S3 event) or model drift → start SageMaker Pipeline or CodePipeline.
  • SageMaker Pipelines can contain a Condition step that decides “retrain or skip”.

2. Exam Traps & Tips (directly from transcript + task list)

Trap Reality / Tip
“Just use Lambda for orchestration” Idle time = cost; move long-running or nested logic to Step Functions.
Confusing CodePipeline stages with Step Functions states CodePipeline = release pipeline; Step Functions = workflow inside a stage or standalone ML pipeline.
Forgetting rollback CodeDeploy Canary/Linear + CloudWatch alarm = automatic rollback; know the hook names.
Thinking SageMaker Pipelines replace CodePipeline They complement: SageMaker Pipelines for ML steps, CodePipeline for overall CI/CD + infra.
Ignoring quotas CodeBuild concurrent builds, CodePipeline parallel executions, SageMaker training job limits—know they exist.
Manual “git push” as the only trigger Prefer EventBridge, S3 event, or scheduled rule for data-driven retrain.
No tests before deploy Always insert a CodeBuild test action or SageMaker Condition step; exam questions punish missing gates.
Blue/Green only for EC2 Also native for Lambda (via CodeDeploy) and ECS; serverless questions expect Canary/Linear.
Data ingestion forgotten AWS Data Pipeline / Data Exchange / Glue → S3 → pipeline trigger is a classic pattern.

3. Hands-on Exercises (Minimize Cost)

Exercise A – Minimal CodePipeline + CodeBuild + S3 (CI portion + artifact)

Goal: Experience source → build → test stages, buildspec.yml, artifacts, and a manual approval gate. Relates directly to “configuring CodePipeline stages”, “unit tests”, “GitFlow-style promotion”.

Cost controls: S3 source (no CodeCommit), BUILD_GENERAL1_SMALL, 5-minute timeout, single execution, delete everything at end. Expected cost < $0.10.

Steps

  1. Create an S3 bucket ml-cicd-source-<accountid> and upload a tiny zip: buildspec.yml src/hello.py # simple def add(a,b): return a+b tests/test_hello.py # assert add(2,3)==5 buildspec.yml content: ```yaml version: 0.2 phases: install: runtime-versions: python: 3.11 build: commands: - pip install pytest - pytest tests/ -v - echo "Build succeeded" > build.txt artifacts: files:

    • build.txt
    • src/** ```

  2. Console → CodePipeline → Create pipeline

  3. Source: Amazon S3, bucket + object key of the zip, change detection = none (we will start manually).
  4. Build: new CodeBuild project, managed image aws/codebuild/standard:7.0, environment BUILD_GENERAL1_SMALL, timeout 5 min, no VPC, no cache.
  5. Deploy: skip for now (or add a second S3 deploy action that just copies the artifact).
  6. Add a Manual approval stage between Build and Deploy.
  7. Release change (start pipeline). Watch the Build stage; confirm pytest ran and artifact appeared in the pipeline’s S3 artifact bucket.
  8. Approve the manual gate (or reject to practice failure path).
  9. Immediately delete: pipeline, CodeBuild project, both S3 buckets (empty first).

Expected result

  • Green Build stage with pytest output in logs.
  • Artifact zip containing build.txt.
  • Pipeline stops at approval until you act.

Exam linkage

  • Trap: “forgot unit tests” → you just ran them inside CodeBuild.
  • Tip: Manual approval = governance gate before model promotion.
  • Quotas: you stayed on the smallest compute; know the 60-min hard limit exists.

Exercise B – SageMaker Pipelines “Hello-ML” + EventBridge trigger (ML-specific CI/CD)

Goal: Define a SageMaker Pipeline with Processing → Training (dummy) → Condition → RegisterModel, execute once, then wire an EventBridge rule that could retrain. Covers “SageMaker Pipelines”, “automated retrain”, “EventBridge”, “model registry”.

Cost controls: Use the tiniest possible instances (ml.m5.large or even better ml.t3.medium if available for processing), 1–2 minute jobs, one execution, delete pipeline + model package group + any endpoints. Expected cost < $0.50 if you stop immediately.

Steps

  1. (Optional but recommended) Create a SageMaker domain with a single user profile only if you don’t already have one; otherwise use the SageMaker console “Pipelines” UI or a temporary Studio notebook that you shut down.

  2. In a short-lived Studio notebook or CloudShell, run (or paste into Pipeline Designer): python import sagemaker from sagemaker.workflow.pipeline import Pipeline from sagemaker.workflow.steps import ProcessingStep, TrainingStep from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo from sagemaker.workflow.condition_step import ConditionStep from sagemaker.workflow.functions import JsonGet from sagemaker.processing import ScriptProcessor from sagemaker.sklearn.processing import SKLearnProcessor from sagemaker.inputs import TrainingInput from sagemaker.estimator import Estimator from sagemaker.workflow.step_collections import RegisterModel # \ldots define a 10-line SKLearnProcessor that writes a metrics.json with "accuracy": 0.91 # \ldots a dummy TrainingStep (or skip real training and just use a ProcessingStep that “pretends”) # \ldots ConditionStep that checks accuracy >= 0.9 # \ldots RegisterModel step that registers into a Model Package Group pipeline = Pipeline(name="ml-cicd-demo", steps=[\ldots], sagemaker_session=sess) pipeline.upsert(role_arn=role) execution = pipeline.start() execution.wait(delay=30, max_attempts=20) # should finish in < 5 min

  3. Console → EventBridge → Rules → Create rule

  4. Event pattern: SageMaker → Model Package State Change (or Training Job State Change).
  5. Target: a tiny Lambda that just prints the event (create Lambda with 128 MB, 10 s timeout, Python 3.12).
  6. Manually transition a model package status (or re-run the pipeline) and confirm the Lambda is invoked (check CloudWatch Logs).
  7. Cleanup (mandatory):
  8. Delete the pipeline execution and the pipeline itself.
  9. Delete the Model Package Group and any model packages.
  10. Delete the Lambda and EventBridge rule.
  11. Shut down any Studio apps / kernel gateways.
  12. Delete the SageMaker domain if you created one solely for this exercise.

Expected result

  • Pipeline execution graph shows Processing → Condition (True) → RegisterModel.
  • Model appears in Model Registry with status “PendingManualApproval” or “Approved”.
  • EventBridge rule fires and Lambda log shows the event.

Exam linkage

  • Trap: “use Lambda for the whole orchestration” → you used SageMaker Pipelines + EventBridge instead.
  • Tip: Condition step = quality gate before registration (continuous testing).
  • Retrain mechanism: EventBridge on data/model events is the standard pattern.
  • Deployment strategies later attach to the registered model (blue/green endpoint update).

Exercise C – CodeDeploy Canary for Lambda (deployment strategies + rollback)

Goal: Practice Canary deployment, alarms, hooks, and automatic rollback—exactly the serverless strategies called out in the transcript.

Cost controls: Pure Lambda + CodeDeploy; no EC2. One canary run lasting < 10 minutes, then delete. Cost ≈ $0.

Steps

  1. Create two simple Lambda functions (or one function with two versions):
  2. prod-alias points at version 1 (stable).
  3. Version 2 contains a deliberate bug (e.g., raise Exception("boom")) so the canary fails.
  4. Console → CodeDeploy → Applications → Create (compute platform = Lambda).
  5. Create Deployment Group:
  6. Traffic shifting = Canary (10 % for 5 minutes, then 100 %).
  7. Add a CloudWatch alarm on Errors metric of the Lambda (threshold > 0).
  8. Enable automatic rollback on alarm or hook failure.
  9. (Optional hooks) Add a tiny Lambda for BeforeAllowTraffic / AfterAllowTraffic that just returns success.
  10. Create a deployment that shifts from v1 → v2. Watch:
  11. 10 % traffic moves.
  12. Alarm fires (or hook fails).
  13. CodeDeploy automatically rolls back to v1.
  14. Delete the deployment group, application, Lambdas, and alarm.

Expected result

  • Deployment status becomes “Failed” and “Rolled back”.
  • Alias still points at the healthy version.
  • CloudWatch shows the alarm state change.

Exam linkage

  • Direct hit on “deployment strategies (blue/green, canary, linear)” and “rollback actions”.
  • Trap: forgetting alarms/hooks → you just saw why they matter.
  • Tip: same pattern works for SageMaker endpoint variants (all-traffic vs canary).

4. Quick Reference Cheat-Sheet for the Exam

  • Pipeline skeleton: Source (CodeCommit/S3/GitHub) → CodeBuild (test + package) → Approval → CodeDeploy/SageMaker → EventBridge feedback loop.
  • Prefer SageMaker Pipelines for pure ML steps; wrap them in CodePipeline when you also need infra or multi-account promotion.
  • Always surface model metrics to a Condition step or CodeBuild test so a bad model never reaches production.
  • Data arrival (S3/Data Exchange) → EventBridge → pipeline start = automated retrain.
  • Clean up every lab resource; the exam will never ask you to leave expensive instances running.

Complete the three exercises above, delete everything, and you will have muscle memory for every knowledge and skill bullet in Task 3.3 while spending well under a couple of dollars. Good luck!