Skill 3.3.4: Configure automated testing strategies within CI/CD pipelines for traditional ML and AI workloads.
Task 3.3: Deployment and Orchestration of ML and AI Workflows - Implement automated orchestration and continuous integration and continuous delivery (CI/CD) pipelines for MLOps and AI workloads.
Core objective: Every behavior-changing artifact—code, data, model, prompt, agent config, retrieval index—must pass automated tests before production. Traditional ML gates on deterministic/numeric metrics; AI/GenAI gates on evaluation scores vs. baseline. Pipelines must be repeatable, versioned, auditable, and reversible.
Service-to-Test Role Map
| Layer | AWS Service | Automated Testing Role |
|---|---|---|
| Source | CodeCommit / CodeConnections | Internal Git or external GitHub/GitLab/Bitbucket without stored credentials |
| Build/Test | CodeBuild | Runs unit, integration, data validation, container builds, test reports |
| Release orchestration | CodePipeline | Stages, approvals, cross-service handoff, event triggers |
| Deploy/Rollback | CodeDeploy / SageMaker Deployment Guardrails | Canary/linear/all-at-once, baking period, CloudWatch alarm rollback |
| ML workflow | SageMaker Pipelines | Processing, training, evaluation, condition steps, registration |
| Model lineage | SageMaker Model Registry / MLflow | Versions, metadata, approval status, run lineage |
| GenAI prompts | Bedrock Prompt Management | Versioned prompts, variables, variant comparison |
| GenAI eval | Bedrock Model Evaluation | Automatic/human evaluation, LLM-as-a-judge, safety/quality metrics |
| RAG refresh | Bedrock Knowledge Bases | Incremental ingestion resync, retrieval evaluation |
| Triggers | EventBridge / S3 events | Event-driven retraining, pipeline start, approval notifications |
Test Taxonomy & Gate Mechanisms
| Test Layer | Traditional ML | AI/GenAI | Gate Mechanism |
|---|---|---|---|
| Code | Unit, integration, contract | Same + prompt template tests | CodeBuild fails build |
| Data | Schema, completeness, drift, leakage | Prompt dataset integrity, PII/safety scan | Processing/Clarify step; ConditionStep |
| Model quality | Accuracy/F1/AUC, RMSE, slice metrics | Faithfulness, relevance, toxicity, hallucination, semantic similarity | Evaluation step vs. baseline threshold |
| Bias/Explainability | Clarify bias, SHAP | Toxicity, bias, safety | ClarifyCheck; approval status |
| Prompt/Agent | N/A | Prompt regression, tool selection, task completion, safety | Prompt Management version + eval suite |
| RAG retrieval | N/A | Context precision/recall, groundedness, citation accuracy | Retrieval eval before index promotion |
| Deployment | Canary/linear with alarms | Same + guardrail metrics | CloudWatch alarm → auto-rollback |
Pipeline Patterns
- Traditional ML CI/CD: CodePipeline source → CodeBuild (code/data tests) → SageMaker Pipeline (process/train/evaluate/register) → Model Registry approval → CodeDeploy/SageMaker endpoint with canary/linear + alarms.
- GenAI CI/CD: Source → CodeBuild (prompt/safety tests) → Bedrock Model Evaluation → Prompt Management version → deploy FM/agent by version ID → canary with CloudWatch alarms.
- Fine-tuned FM deployment: Version custom/fine-tuned model in Model Registry or Bedrock custom model; deploy by identifier, not in-place config.
- RAG refresh: S3/EventBridge → Knowledge Base ingestion resync (incremental) → retrieval eval → promote index. Full reindex only when embedding model, chunking strategy, or index dimensions change.
- Retraining triggers: Schedule = simple, wasteful if no change; Event = efficient for irregular/new data; Manual = while pipeline is being trusted. End at registration, not deployment.
- SageMaker Pipelines steps: ProcessingStep, TrainingStep, EvaluationStep, ConditionStep, RegisterModel. ConditionStep gates registration/deployment on metric threshold.
Decision Matrix
| Choice | Use When | Avoid When |
|---|---|---|
| CodeBuild vs SageMaker Pipelines | CodeBuild = build/test code/container; SageMaker Pipelines = ML steps | Using CodeBuild for ML orchestration or SageMaker Pipelines for generic code release |
| CodePipeline vs SageMaker Pipelines | CodePipeline = release stages, approvals, cross-service; SageMaker Pipelines = ML workflow | Expecting one tool to do both end-to-end |
| Canary vs Linear vs All-at-once | Canary = small slice first, fastest risk reduction; Linear = even steps; All-at-once = lowest cost | All-at-once for high-risk models; canary when endpoint type unsupported |
| Scheduled vs Event retraining | Schedule for predictable cadence; Event for irregular data arrival | Schedule when data arrives irregularly; Event when no reliable event exists |
| Incremental resync vs Full reindex | Resync for added/changed/deleted docs; Full reindex for embedding/chunking/dimension changes | Treating index copy as refresh; full reindex on every doc edit |
| Manual approval vs Auto-deploy | Manual for regulated/high-risk; Auto when tests+thresholds are trusted | Manual-only with no automated tests; auto-deploy without alarms |
Security & Compliance
- IAM roles: Least privilege per stage; separate roles for CodeBuild, SageMaker, CodeDeploy.
- Secrets: CodeConnections for external repos; Secrets Manager for credentials; no PATs in pipeline definitions.
- Encryption: KMS at rest for S3, ECR, Model Registry, CodePipeline artifacts; TLS in transit.
- Network: VPC endpoints for CodePipeline, CodeBuild, SageMaker, S3, KMS; restrict public egress where required.
- Audit: CloudTrail API logs; Model Registry lineage; MLflow run IDs; prompt/agent versions.
- Approvals: CodePipeline manual approval or Model Registry approval status for production gates.
Exam Gotchas
No alarm = no auto rollback. Deployment Guardrails watch CloudWatch alarms during baking period. Without an alarm, a bad deployment completes successfully.
Deployment Guardrails are not universal. Confirm endpoint hosting option supports canary/linear/rolling and auto-rollback before designing release.
GenAI tests cannot assert exact strings. Use fixed prompt sets + evaluation metrics vs. baseline.
Prompts are artifacts. Version prompts and agent configs like model weights; otherwise regressions are unexplainable.
Retraining pipeline should end at registration. Keep human/test gate between fresh model and production traffic.
Incremental resync ≠ full reindex. Copying an index does not regenerate vectors; embedding/chunking changes require full reindex.
CodePipeline ≠ SageMaker Pipelines. One orchestrates release stages; the other orchestrates ML steps. They meet at handoff.
Test set contamination is fatal. Never train on validation/test data; use time-based splits for drift-prone data.
Anti-Patterns
- Manual-only approvals without automated tests.
- Testing only code and not data/model/prompt/retrieval behavior.
- Hardcoded thresholds not tied to baseline or business KPI.
- No baseline comparison for GenAI evaluation.
- Deploying retrained model directly from training pipeline.
- Ignoring slice metrics (bias/fairness/subgroup performance).
- Using production data for tests without sanitization.
- No lineage between model version, training run, data, prompt, agent config.
- Full reindex on every document change → high cost/latency.
- Storing credentials in pipeline definition instead of CodeConnections/Secrets Manager.