Task 3.3 - Implement automated orchestration and continuous integration and continuous delivery (CI/CD) pipelines

Overview

Task 3.3, Implement automated orchestration and continuous integration and continuous delivery (CI/CD) pipelines for MLOps and AI workloads, is about making a deployment repeatable, testable, and reversible.

It covers the rollout shape and the rollback that guards it, the pipeline services that carry a change from source to production, the tests that gate the change, the mechanism that retrains a model, and the versioning that lets an auditor reconstruct what shipped and when.

As you review these lessons for Task 3.3, check that you understand how to do the following.

Implement automated deployment strategies and rollback actions.
Configure and troubleshoot AWS CodeBuild, AWS CodeCommit, AWS CodeDeploy, AWS CodePipeline, and AWS CodeConnections.
Configure training and inference jobs.
Configure automated testing strategies within CI/CD pipelines for traditional ML and AI workloads.
Build and integrate mechanisms to re-train models.
Manage model versions for repeatability and audits (for example, SageMaker Model Registry, MLflow on SageMaker AI).
Manage prompts (for example, Amazon Bedrock Prompt Management).
Implement automated agent deployment pipelines and agent version management.
Implement AI model testing frameworks, including prompt testing.
Configure FM deployment automation with fine-tuned model versioning.
Configure AI-specific pipeline orchestration for RAG system updates and knowledge base refresh cycles.

AWS services overview

The services here divide into three jobs: orchestrating machine learning steps, carrying a code change through build and release stages, and recording the versioned artifacts a pipeline produces. Notice that the machine learning orchestration and the software release pipeline are separate tools that meet at a handoff, rather than one tool doing both. Review each service to learn more.

AWS Service	Description
Amazon SageMaker Pipelines	Orchestrates the machine learning steps themselves, from processing and training through evaluation and registration, on infrastructure the service provisions and releases for each run.
Amazon SageMaker Model Registry	Catalogs trained models as versions inside a model group, records metadata and lineage with each one, and carries the approval status a deployment step reads.
MLflow on Amazon SageMaker AI	Tracks experiment runs, parameters, and metrics through a managed tracking server, and can register a chosen run automatically into the model registry.
AWS CodePipeline	Models the release process as stages and moves a change through them, which is where a machine learning deployment gets its approval gates and ordering.
AWS CodeBuild	Runs the build and test commands for a stage, which is where unit tests, container builds, and evaluation scripts execute.
AWS CodeDeploy	Deploys an application to compute targets and manages the rollout and rollback behavior for that deployment.
AWS CodeCommit	Hosts Git repositories inside the account, so source control and the pipeline that reads it live in the same place.
AWS CodeConnections	Holds a named connection to a repository hosted outside AWS, so a pipeline can use that repository as a source without storing credentials in the pipeline.
Amazon EventBridge	Routes events, which is how a completed training run, a registry approval, or a schedule triggers the next thing without anyone starting it.
Amazon Bedrock Prompt Management	Stores prompts as versioned resources with variables and comparable variants, so a prompt change is released rather than edited in application code.

Knowledge Check

A pipeline must use a Git repository that is hosted outside AWS as its source, without credentials being stored in the pipeline definition. Which service provides that link? (Choose 1)

A. AWS CodeConnections
B. AWS CodeCommit
C. AWS CodeBuild
D. AWS CodeDeploy

Answer & Explanation

Correct Answer: A. AWS CodeConnections

Explanation:
AWS CodeConnections (formerly AWS CodeStar Connections) provides managed, secure connection resources between AWS services (like AWS CodePipeline) and third-party code repositories hosted outside AWS—such as GitHub, GitHub Enterprise Server, GitLab, and Bitbucket Cloud. It establishes an OAuth or GitHub App-based connection handshake, allowing CodePipeline to detect commits and fetch source code without embedding static Personal Access Tokens (PATs) or credentials inside pipeline definitions or Infrastructure as Code (CloudFormation/CDK) templates.

Why the other options are incorrect: * B. AWS CodeCommit: AWS CodeCommit is a Git repository service hosted inside AWS itself, rather than a connection link to external, third-party Git repositories. * C. AWS CodeBuild: AWS CodeBuild is a managed build service that executes build specs, runs unit tests, and packages build artifacts; it does not manage secure third-party repository integrations. * D. AWS CodeDeploy: AWS CodeDeploy automates application deployments to compute targets (e.g., EC2, ECS, Lambda, on-premises instances) during the deployment stage of a CI/CD pipeline.

AWS skills overview

In this lesson, you review the skills behind Task 3.3 as a light refresher. Each task groups a handful of related skills into a few topics. Read through them and notice which ones already feel familiar, and which ones feel shaky enough that you want to come back to them before exam day.

Deployment strategies and rollback actions

In practice, this group of skills lets the machine learning engineer replace a model in production without an outage, and undo the replacement automatically when the new version misbehaves.

Be comfortable doing the following:

Choosing a rollout shape, and separating a shift that moves all traffic at once from a canary that moves a small slice first, a linear shift that moves it in even steps, and a rolling update that replaces capacity in batches.
Configuring the rollback that guards the rollout, which means naming the alarms that are watched and the window during which they are watched.
Configuring and troubleshooting the pipeline services that carry the change, and recognizing which stage a failure belongs to before changing anything.

The mechanism behind every one of these shapes is the same, and it is the part worth remembering: a new fleet is brought up alongside the old one, traffic moves to it according to the shape you chose, and a baking period follows during which specified alarms are watched. If no alarm trips, the old fleet is removed. If an alarm trips, traffic returns to the old fleet, which is why an automatic rollback is only as good as the alarm that triggers it. A rollout with no alarm configured has no rollback signal, and it will complete a bad deployment as confidently as a good one. Managed rollout shapes also do not apply to every hosting option, so confirm the option supports them before designing the release around them.

Knowledge Check

A managed rollout completed successfully even though the new model version returned errors at a high rate. What explains that outcome? (Choose 1)

A. The rollout used a canary shape rather than a linear shape
B. No alarm was configured for the rollout to watch during the baking period
C. The old fleet was removed before traffic shifted
D. The endpoint had a scaling policy attached

Answer & Explanation

Correct Answer: B. No alarm was configured for the rollout to watch during the baking period

Explanation:
SageMaker AI Deployment Guardrails (blue/green deployments) use a baking period to validate new model versions before terminating the old fleet. During this time, SageMaker relies on Amazon CloudWatch Alarms (e.g., monitoring Invocation5XXErrors or ModelLatency) to trigger an automated rollback if errors occur.

If no CloudWatch Alarm is configured and linked to the deployment guardrail's AutoRollbackConfig, SageMaker has no signal to detect model failures. Once the baking period expires without a tripped alarm, SageMaker completes the rollout, shifts 100% of traffic to the green fleet, and terminates the blue fleet—marking the deployment as "successful" despite the high error rate.

Why the other options are incorrect: * A. The rollout used a canary shape rather than a linear shape: Traffic shifting strategies (canary vs. linear) control the rate at which traffic moves to the new fleet, but neither strategy automatically triggers rollbacks without a linked CloudWatch Alarm. * C. The old fleet was removed before traffic shifted: In SageMaker deployment guardrails, traffic is shifted incrementally to the new green fleet before the old blue fleet is terminated. * D. The endpoint had a scaling policy attached: Application Auto Scaling policies manage compute instance counts based on utilization metrics and have no impact on blue/green deployment validation or rollback mechanisms.

Configuring jobs and retraining mechanisms

In practice, this group of skills lets the machine learning engineer define the jobs a pipeline runs and arrange for a model to be rebuilt without anyone remembering to start it.

Be comfortable doing the following:

Configuring a training job and an inference job, including the container image, the instance configuration, the input and output locations, and the role the job assumes.
Building a retraining mechanism, which means a pipeline that can run unattended plus a trigger that starts it.
Integrating that mechanism with the rest of the workflow, so a retrained model lands in a registry and waits for approval rather than deploying itself.

There are three honest triggers for retraining, and choosing between them is a design decision rather than a preference. A schedule retrains on a fixed cadence, which is simple and wastes work when nothing has changed. An event retrains when something specific happens, such as a new data batch landing or a monitored metric crossing a threshold, which is more efficient and requires that the event actually exists and is emitted. A manual run remains the right answer while a pipeline is still being trusted. Whichever trigger you use, the pipeline should end at registration rather than at deployment, because that separation is what lets a human or a test gate stand between a fresh model and production traffic.

Knowledge Check

A team wants a model rebuilt only when new labeled data arrives, and the arrival is irregular. Which retraining trigger fits that requirement? (Choose 1)

A. A fixed daily schedule for the training pipeline
B. An event emitted when the new data lands, routed to start the pipeline
C. A manual pipeline run whenever someone notices the new data
D. A scaling policy on the training instances

Answer & Explanation

Correct Answer: B. An event emitted when the new data lands, routed to start the pipeline

Explanation:
An event-driven architecture is the ideal solution for irregular workloads. When new labeled training data is uploaded to Amazon S3, S3 generates an event notification (e.g., ObjectCreated) that Amazon EventBridge captures. EventBridge then evaluates a rule and automatically invokes the Amazon SageMaker Pipeline execution API (StartPipelineExecution). This guarantees immediate execution when data lands without wasting compute during idle stretches.

Why the other options are incorrect: * A. A fixed daily schedule for the training pipeline: Schedule-based triggers (e.g., EventBridge cron expressions) are designed for predictable, periodic runs. Running on a schedule when data arrives irregularly leads to either redundant pipeline executions when no new data has landed or long delays waiting for the next scheduled run. * C. A manual pipeline run whenever someone notices the new data: Manual triggers introduce operational overhead, human error, and unnecessary delays, violating the principles of automated MLOps CI/CD pipelines. * D. A scaling policy on the training instances: Application Auto Scaling policies manage runtime compute capacity for active endpoints based on traffic metrics; they do not orchestrate or trigger batch workflow executions.

Automated testing in ML and AI pipelines

In practice, this group of skills lets the machine learning engineer put real gates in a pipeline, including gates for behavior that has no single correct answer.

Be comfortable doing the following:

Configuring automated testing inside a pipeline, and separating tests of the code from tests of the model that the code produced.
Gating a deployment on an evaluation result, so a model that fails a quality threshold stops at the gate instead of reaching an endpoint.
Testing generative AI behavior, including comparing prompt variants and evaluating responses against criteria rather than against an exact expected string.

The distinction that matters here is that a traditional pipeline can assert an exact result and a generative one usually cannot. Code tests are deterministic, so a build stage can fail on them outright. A traditional model can be gated on a metric threshold computed over a held-out dataset, which is still a numeric comparison. A generative response varies between runs, so the test becomes an evaluation: run a fixed set of prompts, score the responses against stated criteria or against a reference, and compare the aggregate to the previous version. That is also why prompts belong under version control alongside the model, since a prompt change alters behavior exactly as a weight change does, and a pipeline that versions one but not the other cannot explain a regression.

Knowledge Check

A pipeline must stop a generative AI change from shipping when response quality drops, and responses vary between runs. Which testing approach fits? (Choose 1)

A. Run a fixed prompt set through an evaluation and compare scores against the previous version
B. Assert that each response matches a stored expected string
C. Run the existing unit tests for the application code
D. Deploy with a canary shift and watch endpoint latency

Answer & Explanation

Correct Answer: A. Run a fixed prompt set through an evaluation and compare scores against the previous version

Explanation:
Generative AI models are non-deterministic, meaning responses vary across runs even for identical prompts. To test response quality in a automated CI/CD pipeline (such as via Amazon Bedrock Model Evaluation or SageMaker Clarify), you run a curated, representative "golden prompt dataset" through both the candidate and baseline model versions. The pipeline evaluates semantic quality using LLM-as-a-judge or task-specific metrics (e.g., faithfulness, answer relevance, toxicity, or ROUGE/BERTScore) and blocks the deployment stage if the candidate version's score drops below the threshold set by the current version.

Why the other options are incorrect: * B. Assert that each response matches a stored expected string: Because generative AI responses naturally vary between runs, exact string assertions will cause false negatives and flaky pipeline builds. * C. Run the existing unit tests for the application code: Application unit tests verify code syntax, API routing, and data structures, but cannot evaluate semantic response quality or model hallucination rates. * D. Deploy with a canary shift and watch endpoint latency: Monitoring endpoint latency checks infrastructure responsiveness and throughput, but does not detect drops in response accuracy, correctness, or generation quality.

Versioning models, prompts, and agents

In practice, this group of skills lets the machine learning engineer answer the audit question: what exactly was serving traffic last month, and what produced it.

Be comfortable doing the following:

Managing model versions for repeatability and audits, using a registry that groups versions of one model, records the metadata and lineage behind each version, and carries an approval status a pipeline can read.
Tracking the experiments those versions came from, and registering a chosen run into the registry rather than copying artifacts by hand.
Versioning prompts and agents the same way, and automating a foundation model deployment so a customized or fine-tuned version is released by identifier rather than by editing configuration in place.

The habit to carry away is that every artifact that changes behavior needs a version, not just the model weights. A registry gives a trained model a version, a group to sit in, and an approval status, which is what turns a deployment step into a lookup rather than a decision. Experiment tracking sits upstream of that and answers where a version came from, including the parameters and metrics of the run, and it can register the winning run automatically so the lineage stays intact. Prompts are the artifact teams most often forget: a stored prompt with variables can be tested as comparable variants and then released as a version, so an application refers to a version rather than embedding text. Agents need the same treatment, because an agent's behavior comes from its configuration and its tools as much as from the model behind it.

Knowledge Check

An investigation must establish which model version served traffic during a past incident and which training run produced it. Which combination supports that? (Choose 1)

A. A model registry version with its recorded lineage back to the tracked training run
B. The container image tag used by the endpoint
C. The endpoint's scaling history
D. The pipeline definition as it exists today

Answer & Explanation

Correct Answer: A. A model registry version with its recorded lineage back to the tracked training run

Explanation:
Amazon SageMaker Model Registry maintains end-to-end lineage tracking and metadata for deployed models. When a model version is registered, it creates an immutable record linking the model package directly to the exact SageMaker training job run, input S3 dataset URIs, hyperparameters, and evaluation metrics. In combination with deployment history, this enables engineers to trace backward from a past incident to the exact model package and the training pipeline run that produced it.

Why the other options are incorrect: * B. The container image tag used by the endpoint: Container image tags track the serving environment or framework version (e.g., PyTorch/Triton inference server image), not the specific trained model weights or their original training job. * C. The endpoint's scaling history: Scaling history records CloudWatch metrics for instance counts and request volumes over time, but provides no details on model versioning or artifact lineage. * D. The pipeline definition as it exists today: Present pipeline code represents current workflow definitions—which may have drifted or changed since the incident—and does not store historical execution artifacts or runtime lineage.

Orchestrating retrieval system updates

In practice, this group of skills lets the machine learning engineer keep a retrieval application current without rebuilding it on every document change.

Be comfortable doing the following:

Configuring pipeline orchestration for retrieval updates, so a document change leads to an ingestion run that refreshes the index.
Choosing between a scheduled refresh and an event-driven refresh, and recognizing that ingestion processes only documents that were added, changed, or removed since the last run.
Separating a routine refresh from a change that requires reindexing, because they cost very different amounts of work.

The cost difference is the point. A resync is incremental: unchanged documents are skipped, changed documents are parsed, chunked, embedded, and indexed again, and deleted documents are removed from the store. That makes a frequent refresh affordable, and it makes an event-driven trigger on document changes a reasonable design. Changing the embedding model, the chunking strategy, or the index dimensions is a different class of change, because every existing vector was produced under the old settings and has to be regenerated. Copying an index does not regenerate the vectors, unless the copy is routed through an embedding pipeline as part of the operation, so treating a copy as a refresh leaves stale vectors in place while looking like progress.

Knowledge Check

A retrieval application must reflect edits to a handful of documents in a large corpus, and the team wants the refresh to stay inexpensive. What does an ingestion resync do? (Choose 1)

A. Regenerates every vector in the index
B. Processes only the documents that were added, changed, or removed since the last run
C. Copies the existing index to a new one
D. Replaces the vector store with a different service

Answer & Explanation

Correct Answer: B. Processes only the documents that were added, changed, or removed since the last run

Explanation:
In Amazon Bedrock Knowledge Bases and RAG data ingestion pipelines, an ingestion resync (or incremental synchronization) performs delta reconciliation between the data source (e.g., an Amazon S3 bucket) and the vector index. Rather than re-processing the entire document corpus, the ingestion job checks metadata timestamps and object hashes to isolate changes:

New files: Parsed, chunked, embedded, and added to the index.
Modified files: Re-parsed, re-chunked, re-embedded, and updated in place.
Deleted files: Corresponding vector embeddings and chunks are removed from the vector store.

This incremental approach minimizes embedding model API calls and compute costs, keeping frequent corpus updates fast and inexpensive.

Why the other options are incorrect: * A. Regenerates every vector in the index: Performing a full re-index re-evaluates every document regardless of whether it changed, incurring significant latency and high unnecessary embedding costs. * C. Copies the existing index to a new one: Duplicating an index creates a snapshot or backup, but does not parse or apply document edits from the data source. * D. Replaces the vector store with a different service: Changing vector database backends is an infrastructure migration, not a data ingestion or synchronization process.

Knowledge Check

A team updates a production model and wants a small slice of traffic to reach the new version ahead of the rest, with an automatic return to the previous version when error rates rise. Which configuration achieves that? (Choose 1)

A. A canary traffic shift with a baking period watched by an alarm on the error metric
B. A shift of all traffic at once, followed by a manual check of the metrics
C. A second endpoint running the new version, with the application choosing between them
D. A scaling policy that adds instances when the error metric rises

Answer & Explanation

Correct Answer: A. A canary traffic shift with a baking period watched by an alarm on the error metric

Explanation:
Amazon SageMaker Deployment Guardrails support canary traffic shifting, where a small percentage of production traffic (e.g., 10%) is routed to the new model version (green fleet) while the remaining traffic stays on the baseline version (blue fleet).

During the specified baking period, SageMaker monitors linked Amazon CloudWatch Alarms (such as Invocation5XXErrors). If the alarm trips during this window, SageMaker automatically triggers a rollback, rerouting 100% of traffic back to the original blue fleet and preserving system stability without manual intervention.

Why the other options are incorrect: * B. A shift of all traffic at once, followed by a manual check of the metrics: Shifting 100% of traffic at once exposes all users to potential failures immediately, and relying on manual checks removes the automated rollback protection. * C. A second endpoint running the new version, with the application choosing between them: Managing client-side routing across two separate endpoints adds operational complexity and custom application logic, whereas deployment guardrails handle blue/green traffic shifting natively at the SageMaker endpoint level. * D. A scaling policy that adds instances when the error metric rises: Application Auto Scaling adds compute capacity based on traffic load or resource utilization; it cannot revert a bad model deployment or manage blue/green traffic shifts.