Skill 3.3.6: Manage Model Versions for Repeatability and Audits
Task 3.3: Deployment and Orchestration of ML and AI Workflows – Implement automated orchestration and CI/CD pipelines for MLOps and AI workloads.
Core objective: Every artifact that changes behavior—model weights, prompts, agents, indexes—must have an immutable version, lineage, and approval status. CI/CD pipelines write versions; deployment reads approval.
Core Versioning Concepts
| Concept | What It Is | Exam Takeaway |
|---|---|---|
| Model Group | Logical container for versions of one model | One group per model/use case |
| Model Version | Immutable package: artifacts + metadata + lineage + approval | Never edit in place; create new version |
| Approval Status | PendingManualApproval, Approved, Rejected |
Deployment gates on Approved only |
| Lineage | Training job, data S3 URIs, hyperparameters, metrics, container image, pipeline execution, git commit | Required for audits and incident forensics |
| Repeatability | Same version ID + metadata + artifacts → reproducible deployment | Registry version is the source of truth |
| Audit Trail | Who approved, when, what training run, what data, what metrics | CloudTrail + Model Registry approval history |
SageMaker Model Registry
| Feature | Detail | Use |
|---|---|---|
| Model Groups | Organize versions | Separate groups by model/use case |
| Model Packages | Versioned artifacts | Increment on every registration |
| Metadata | Customer metadata, inference spec, metrics, validation spec | Enables search and audit |
| Approval Workflow | Pending → Approved/Rejected | Pipeline registers as Pending; deployment reads Approved |
| Lineage | Links to training job, pipeline execution, data, artifacts | Answers “what produced this version?” |
| EventBridge Events | Model Package State Change |
Trigger deployment, notification, or approval automation |
| Pipelines Integration | RegisterModel step |
Automated registration from SageMaker Pipelines |
| Cross-Account | Resource policies where supported | Deployment role must access registry + S3 artifacts |
Gotcha: Model Registry approval is a gate, not a rollback. Rollback requires CloudWatch alarms on the endpoint.
MLflow on SageMaker AI
| Capability | Detail | Exam Takeaway |
|---|---|---|
| Managed Tracking Server | SageMaker-provisioned MLflow | No infrastructure to manage |
| Experiment Tracking | Runs, parameters, metrics, artifacts | Upstream of registry; answers “where did this version come from?” |
| Model Registry | MLflow registry or auto-register to SageMaker Model Registry | Chosen run can be promoted automatically |
| Integration | Works with SageMaker Studio and Pipelines | Use for experiment comparison; use SageMaker Model Registry for deployment gates |
Gotcha: MLflow tracking is not a deployment gate. Use SageMaker Model Registry approval status for production decisions.
Versioning Adjacent Artifacts
| Artifact | Service | Versioning Mechanism | CI/CD Action |
|---|---|---|---|
| Prompts | Amazon Bedrock Prompt Management | Versioned prompts with variables/variants | Test variants; release version; app refers to version ARN |
| Agents | Amazon Bedrock Agents | Agent versions + aliases | Immutable config snapshot; deploy by alias |
| Fine-Tuned FMs | Bedrock custom models / SageMaker models | Model ARN/version | Deploy by identifier; never edit in place |
| RAG Index | Bedrock Knowledge Bases | Ingestion jobs | Incremental resync for document changes; full reindex for embedding/chunking changes |
Gotcha: Copying an index does not re-embed vectors. Changing embedding model, chunking, or dimensions requires full reindex.
CI/CD Integration Pattern
| Stage | Service | Purpose |
|---|---|---|
| Source | CodeCommit / CodeConnections | Git repo inside AWS or external without stored credentials |
| Build/Test | CodeBuild | Unit tests, container builds, evaluation scripts |
| Train/Evaluate | SageMaker Pipelines | Processing, Training, Evaluation, Condition steps |
| Register | SageMaker Model Registry | Create version as PendingManualApproval |
| Approve | CodePipeline manual approval / EventBridge + Lambda | Human or automated gate |
| Deploy | SageMaker Deployment Guardrails | Canary/linear traffic shift + CloudWatch alarms |
| Monitor | SageMaker Model Monitor, Clarify, CloudWatch | Drift, bias, quality |
| Retrain | EventBridge schedule/event → StartPipelineExecution | Schedule, event-driven, or manual trigger |
Gotcha: Pipeline should end at registration, not deployment. Separation allows human/test gate before production.
Security, Compliance & Audit
| Area | Implementation |
|---|---|
| IAM | Least-privilege roles for pipelines, training, registry, deployment |
| KMS | CMK for S3 artifacts, ECR images, SageMaker resources where supported |
| VPC | VPC endpoints for S3, ECR, SageMaker, CodePipeline, CodeBuild, Secrets Manager, KMS |
| Audit | CloudTrail, CloudWatch Logs, Model Registry approval history, Model Cards |
| Compliance | Model Cards for governance; immutable version history for audits |
| Secrets | Never store credentials in pipeline definitions; use CodeConnections or Secrets Manager |
Decision Matrix
| Requirement | Best Fit | Why |
|---|---|---|
| Native SageMaker approval + deployment gate | SageMaker Model Registry | Approval status, lineage, Pipelines, EventBridge |
| Experiment tracking + compare runs | MLflow on SageMaker AI | Params/metrics/artifacts, UI, auto-registration |
| Prompt versioning | Bedrock Prompt Management | Versioned prompts, variables, variants |
| Agent versioning | Bedrock Agents versions/aliases | Immutable config, alias routing |
| Fine-tuned FM versioning | Bedrock custom models / SageMaker models | Model ARN/version, deploy by ID |
| RAG index refresh | Bedrock Knowledge Bases ingestion jobs | Incremental resync; full reindex for embedding/chunking changes |
Exam Gotchas & Anti-Patterns
No alarm = no rollback. Deployment guardrails require CloudWatch alarms. A rollout without an alarm completes a bad deployment as confidently as a good one.
Register, don’t deploy directly. Pipeline should create a
PendingManualApprovalversion. Human/test gate before production.Version prompts, agents, and indexes too. Model weights are not the only artifact that changes behavior.
EventBridge is the glue. Use events for retraining triggers, approval notifications, and deployment automation.
| Anti-Pattern | Correct Approach |
|---|---|
| Editing production model in place | Create new immutable version |
| Storing prompts in application code | Version prompts in Bedrock Prompt Management |
| Manual approval via email/chat | Pipeline gate or EventBridge + Lambda |
| Using endpoint latency as quality gate for GenAI | Evaluation against golden prompts + criteria |
| Assuming canary/linear shapes apply to all hosting options | Confirm hosting option supports managed rollout |
| No lineage: only container image tag | Use Model Registry with full lineage back to training run |