Skill 3.3.5: Build and integrate mechanisms to re-train models
Task 3.3: Deployment and Orchestration of ML and AI Workflows - Implement automated orchestration and CI/CD pipelines for MLOps and AI workloads
Objective: Automate model refresh so production stays accurate, repeatable, auditable, and reversible. Separate ML orchestration from code CI/CD and model deployment.
Retraining Architecture & Service Roles
| Layer | AWS Service | Responsibility | Exam Focus |
|---|---|---|---|
| ML orchestration | SageMaker Pipelines | DAG: process → train → evaluate → register | Purpose-built for ML steps; parameters, caching, conditions |
| Trigger | Amazon EventBridge | Schedule or event rule → start pipeline | Routes events; does not run training |
| Drift detection | SageMaker Model Monitor + CloudWatch | Data/model quality alarms | Alarm → EventBridge → retrain |
| Versioning | SageMaker Model Registry | Model groups, versions, approval status, lineage | Retraining pipeline ends at registration |
| Experiment tracking | MLflow on SageMaker AI | Runs, params, metrics, auto-register | Upstream lineage |
| Code CI/CD | CodePipeline + CodeBuild + CodeCommit/CodeConnections | Test pipeline code, build containers | Separate from ML runs |
| Model deployment | CodeDeploy / SageMaker endpoints | Rollout + rollback | Auto-rollback requires alarms |
| RAG refresh | Bedrock Knowledge Bases | Ingestion resync | Incremental vs full reindex |
Retraining Trigger Decision Matrix
| Trigger | Use When | Mechanism | Trade-off |
|---|---|---|---|
| Schedule | Predictable cadence; compliance | EventBridge schedule → SageMaker Pipeline | Simple; wastes compute if no change |
| Event-driven | New data lands irregularly; drift detected | S3 object-created event, CloudWatch alarm, Model Monitor → EventBridge → Pipeline | Efficient; requires reliable event source |
| Manual | Initial trust-building; low frequency | Start pipeline manually | Human overhead; not scalable |
| Hybrid | Periodic drift check + data arrival | Schedule drift job + event retrain | Balanced; more components |
Exam gotcha: Retraining pipeline should end at Model Registry registration, not deployment. Approval status (pending manual approval / approved) gates the deployment pipeline.
Standard Retraining Pipeline Stages
| Stage | Purpose | Key Config | Gate/Output |
|---|---|---|---|
| Data validation | Schema, drift, quality | Processing job; baseline stats | Pass/fail condition |
| Data prep | Feature engineering, split | Processing job; instance type; S3 I/O | Train/validation/test datasets |
| Train | Fit model | Training job; container; instance; hyperparams; IAM role | Model artifacts in S3 |
| Evaluate | Metrics, bias, explainability | Processing job; Clarify; baseline comparison | Metric threshold condition |
| Register | Version model | Model Registry; model group; metadata; lineage | Approval status |
| Deploy | Release approved model | CodePipeline/CodeDeploy; endpoint update | Traffic shift + rollback |
CI/CD vs ML Orchestration Handoff
| Change | Pipeline | Output |
|---|---|---|
| Pipeline code / container | CodePipeline + CodeBuild | Updated SageMaker Pipeline |
| Data drift / new data | EventBridge → SageMaker Pipeline | New model version |
| Model approval | EventBridge → CodePipeline/CodeDeploy | Updated endpoint |
| Prompt change | Bedrock Prompt Management | New prompt version |
| RAG document change | Bedrock KB ingestion | Refreshed index |
Key Configuration Nuances
- Training job: container image, instance type/count, input channels, output S3 path, execution role, hyperparameters, stopping conditions, checkpointing.
- Inference job: real-time endpoint for production; batch transform for offline evaluation; serverless/async for intermittent or large payloads.
- Pipeline parameters: data URI, instance types, hyperparameters, model approval status. Makes pipeline reusable and idempotent.
- Step caching: skips unchanged steps based on inputs/parameters. Can avoid wasted retrains on schedule triggers.
- Conditions: condition step for evaluation thresholds; register model step with approval status.
- EventBridge: event pattern for S3, CloudWatch alarm, SageMaker job state change; schedule for cadence.
- Model Monitor: data quality, model quality, bias drift, feature attribution. Alarms trigger retraining.
Monitoring & Retraining Signals
| Signal | Service | Action |
|---|---|---|
| Data drift | Model Monitor | Alarm → EventBridge → retrain |
| Model quality drop | Model Monitor + ground truth | Alarm → retrain |
| New labeled data | S3 event | EventBridge → retrain |
| Schedule | EventBridge schedule | Retrain on cadence |
| Manual | Console/CLI | Retrain while building trust |
Security & Compliance
- IAM roles: Separate roles for pipeline execution, training job, processing job, EventBridge. Least privilege; PassRole only where needed.
- KMS: Encrypt S3, EBS, Model Registry, Pipelines, logs. Customer-managed keys for audit.
- VPC: Private subnets + VPC endpoints for S3, ECR, CloudWatch, SageMaker, KMS. No internet for training.
- Secrets: Secrets Manager for external data credentials. No hardcoded secrets.
- Audit: Model Registry lineage, SageMaker Experiments/MLflow, CloudTrail, CloudWatch Logs.
- Cross-account: Resource policies, KMS grants, S3 bucket policies.
Versioning Beyond Models
- Model Registry: Model group → versions → metadata → lineage → approval status.
- MLflow: Tracks runs, params, metrics; can auto-register chosen run into Model Registry.
- Prompts: Bedrock Prompt Management stores versioned prompts with variables and variants. Prompt change = behavior change.
- Agents: Version configuration and tools; behavior comes from config + tools + model.
- RAG: Ingestion resync processes only added/changed/deleted docs. Changing embedding model/chunking/index dimensions requires full reindex.
- Fine-tuned FMs: Release by model identifier/version, not in-place config edits.
Deployment & Rollback (Retraining Handoff)
| Strategy | Behavior | When to Use |
|---|---|---|
| All-at-once | 100% traffic to new fleet | Only with strong pre-prod testing; no safety |
| Canary | Small slice first, then rest | Safest for risky changes |
| Linear | Even steps over time | Predictable gradual shift |
| Rolling | Replace capacity in batches | Capacity-based updates |
Exam gotcha: Auto-rollback requires linked CloudWatch alarms and a baking period. No alarm = no rollback signal; bad deployment completes as “successful.”
Anti-Patterns
- **Retrain-to-deploy without gate