Skill 3.3.10: Configure FM Deployment Automation with Fine-Tuned Model Versioning
Task 3.3: Deployment and Orchestration of ML and AI Workflows - Implement automated orchestration and continuous integration and continuous delivery (CI/CD) pipelines for MLOps and AI workloads.
Core Concept: Automate release of fine-tuned foundation models (FMs) as immutable, versioned artifacts with evaluation gates, approval, blue/green deployment, and rollback. Objective: repeatable, auditable, safe FM updates without manual configuration edits.
Service Roles in FM CI/CD
| Service | Role | Key Configuration |
|---|---|---|
| Amazon Bedrock Custom Models | Fine-tune FMs; each job creates immutable custom model ARN | Requires Provisioned Throughput for inference; no on-demand; version via ARN/tags/registry |
| SageMaker Model Registry | Version models; approval status gates deployment | Model groups, model package versions, metadata, lineage; supports audit |
| SageMaker Pipelines | Orchestrate ML steps (fine-tune, evaluate, register) | Processing/Training/Evaluation/RegisterModel steps; Condition steps for gates |
| AWS CodePipeline | Orchestrate release stages (source, build, test, approve, deploy) | Manual approval; integrates with Lambda/Step Functions to call Bedrock/SageMaker |
| AWS CodeBuild | Run unit tests, evaluation scripts, container builds | buildspec; can invoke AWS CLI for FM APIs |
| AWS CodeConnections | Connect external Git repos (GitHub, GitLab, Bitbucket) | No credentials in pipeline; OAuth/GitHub App |
| AWS CodeCommit | Internal Git repos | Source control inside AWS |
| AWS CodeDeploy | Deploy apps to EC2/Lambda/ECS | Not for SageMaker endpoints; use SageMaker deployment guardrails |
| Amazon EventBridge | Trigger pipelines on events (job complete, approval, schedule) | S3 events, Bedrock/SageMaker state changes, cron |
| AWS Step Functions | Orchestrate long-running FM workflows, wait for approval | Call Bedrock/SageMaker APIs, Lambda, human approval |
| Amazon Bedrock Prompt Management | Version prompts as resources | Variables, variants, version ARNs; release prompts not edit in code |
| MLflow on SageMaker AI | Track experiments, register runs | Managed tracking server; auto-register to Model Registry |
Fine-Tuned FM Versioning: Bedrock vs SageMaker
| Aspect | Bedrock Custom Models | SageMaker Fine-Tuned Models |
|---|---|---|
| Artifact | Custom model ARN | Model artifacts in S3 + Model Registry package |
| Versioning | New job = new ARN; use tags/naming/Model Registry for versions | Model Registry model group + versions; metadata/lineage |
| Approval | External gate (CodePipeline manual approval, Model Registry) | Model Registry approval status (Pending/Approved/Rejected) |
| Deployment | Provisioned Throughput (PT) required; create/update PT | SageMaker endpoint; deployment guardrails (canary/linear/all-at-once) |
| Rollback | Switch app to previous ARN/PT; keep old PT during bake | Auto-rollback to blue fleet on CloudWatch alarm |
| Security | IAM roles, KMS, VPC endpoints (PrivateLink) | IAM roles, KMS, VPC endpoints |
| Best For | Managed FM APIs, quick integration, limited customization | Full control, custom containers, any framework, deeper customization |
Automated Deployment Flow (Event-Driven or Pipeline-Driven)
- Source: Commit fine-tuning config/prompt to Git (CodeCommit/CodeConnections).
- Build/Test: CodeBuild runs unit tests, validates config, packages.
- Fine-Tune: Step Functions/SageMaker Pipelines triggers Bedrock model customization job or SageMaker training job.
- Evaluate: Run evaluation on held-out data/golden prompts (Bedrock Model Evaluation, SageMaker Clarify). Compare candidate vs baseline.
- Register: Create new custom model ARN or Model Registry model package version with metadata/lineage.
- Approve: Manual approval (CodePipeline) or auto-approve based on metrics/Model Registry status.
- Deploy: Create/update Provisioned Throughput (Bedrock) or update SageMaker endpoint with blue/green and CloudWatch alarms.
- Monitor & Rollback: Alarms trigger auto-rollback. Keep previous version active during baking period.
- Audit: Tags, Model Registry, MLflow, CloudTrail, pipeline execution ID.
Deployment Strategies & Rollback
| Strategy | Behavior | Use Case | Rollback Signal |
|---|---|---|---|
| All-at-once | 100% traffic to new fleet | Dev/test, low risk | CloudWatch alarm (if configured) |
| Canary | Small slice first (e.g., 10%) | High-risk FM updates | CloudWatch alarm during baking |
| Linear | Even steps (e.g., 10% every N min) | Controlled gradual shift | CloudWatch alarm |
| Rolling | Replace capacity in batches | Capacity replacement | N/A for traffic shift |
Exam Gotcha: A rollout without a CloudWatch alarm has no rollback signal. SageMaker deployment guardrails will complete a bad deployment as confidently as a good one.
Exam Gotcha: Bedrock custom models require Provisioned Throughput for inference. On-demand inference is not available. Plan capacity and cost.
CI/CD Gates & Testing
- Code tests: Deterministic unit tests in CodeBuild; fail build outright.
- Traditional ML: Gate on metric threshold over held-out dataset (accuracy, F1, RMSE).
- Generative AI: Non-deterministic. Use fixed prompt set, LLM-as-a-judge, compare aggregate scores (faithfulness, relevance, toxicity, ROUGE/BERTScore) against baseline. Block if drop.
- Prompt testing: Version prompts via Bedrock Prompt Management; test variants as comparable resources.
- Approval gate: Manual approval in CodePipeline or Model Registry approval status.
- Deployment gate: Canary/linear with baking period and CloudWatch alarms.
Security & Compliance
- IAM roles: Separate least-privilege roles for fine-tuning, evaluation, deployment, pipeline execution.
- KMS: Encrypt S3 artifacts, Model Registry, Bedrock custom models, logs.
- VPC endpoints: PrivateLink for Bedrock, SageMaker, S3, KMS, CodePipeline, EventBridge to avoid public internet.
- Audit: CloudTrail, Model Registry lineage, MLflow tracking, tags, immutable version IDs.
- CodeConnections: External Git without stored credentials.
When to Use X vs Y
| Decision | Use X | Use Y |
|---|---|---|
| ML orchestration | SageMaker Pipelines | CodePipeline |
| ML steps: process, train, evaluate, register | Release stages: source, build, test, approve, deploy | |
| External Git | CodeConnections | CodeCommit (internal) |
| FM hosting | Bedrock (managed, PT) | SageMaker (custom, endpoints) |
| Retraining trigger | EventBridge (irregular data) | Schedule (predictable cadence) |
| Prompt versioning | Bedrock Prompt Management | Application code (anti-pattern) |
| RAG refresh | Incremental resync | Full re-index (embedding/chunking change) |
Anti-Patterns
- Editing production config in place: Changing model ID in app instead of releasing new version.
- Mutable tags: Using "latest" for model versions; cannot audit or rollback reliably.
- No alarm on deployment: No auto-rollback signal.
- Prompts in code: Not versioned; cannot explain regressions.
- Copying index as refresh: Stale vectors; must regenerate if embedding/chunking changes.
- CodeDeploy for SageMaker: Wrong service; use SageMaker deployment guardrails.
- CodeCommit for external GitHub: Wrong; use CodeConnections.
- Bedrock custom models on-demand: Not supported; requires Provisioned Throughput.
Key Exam Takeaway: Separate ML orchestration (SageMaker Pipelines) from release orchestration (CodePipeline). Version every behavior-changing artifact (model, prompt, agent, RAG config). Gate deployment on evaluation and approval. Always configure CloudWatch alarms for auto-rollback.