Skill 3.3.9: Implement AI Model Testing Frameworks, Including Prompt Testing
Task 3.3: Deployment and Orchestration of ML and AI Workflows - Implement automated orchestration and CI/CD pipelines for MLOps and AI workloads.
Core Concept
- Business objective: Gate AI/ML changes in CI/CD so bad models, prompts, RAG updates, or agents never reach production; enable repeatable, auditable, reversible releases.
- Traditional ML testing: deterministic metrics on held-out data; gate on numeric threshold.
- Generative AI testing: non-deterministic outputs; evaluate aggregate quality vs baseline using fixed prompt sets and scoring criteria.
- Prompt testing: treat prompts as versioned artifacts; test variants against golden datasets before release.
Testing Layers & Gates
| Layer | What to Test | Gate Type | Primary AWS Services |
|---|---|---|---|
| Code | Unit, integration, lint, container build | Deterministic pass/fail | CodeBuild, CodePipeline |
| Traditional ML | Accuracy, F1, AUC, RMSE on holdout | Metric threshold | SageMaker Pipelines, Clarify |
| Prompt/LLM | Quality, toxicity, bias, faithfulness, relevance | Aggregate score vs baseline | Bedrock Model Evaluation, Clarify |
| RAG | Retrieval recall, context relevance, groundedness | Golden Q/A + faithfulness | Bedrock Knowledge Bases, Bedrock Eval |
| Agents | Task completion, tool selection, trajectory | Success rate + guardrail compliance | Bedrock Agents, Bedrock Eval |
| Online | Latency, errors, drift, quality | Canary + alarms | CloudWatch, Model Monitor, CodeDeploy |
Prompt Testing Framework Design
- Version prompts in Bedrock Prompt Management: variables, variants, comparison, release by version.
- Golden dataset: fixed prompts, references or criteria, edge cases, adversarial inputs, safety probes.
- Scoring: exact match only for deterministic tasks; use semantic similarity, LLM-as-a-judge, or human review for generative tasks.
- Baseline comparison: candidate vs current production version; block if quality drops or safety metric exceeds limit.
- Non-determinism control: fix temperature/top-p where possible; run multiple trials; use statistical significance.
- Human-in-the-loop: calibrate LLM judge; sample high-risk outputs; required for regulated or safety-critical use.
Prompt & AI Evaluation Metrics
| Metric | Measures | Best For | Gotcha |
|---|---|---|---|
| Exact Match | String equality | Extraction, classification | Fails on paraphrase |
| BLEU / ROUGE | N-gram overlap | Translation, summarization | Poor semantic meaning |
| BERTScore | Semantic similarity | General generation | Needs reference |
| Faithfulness / Groundedness | Hallucination vs context | RAG | Requires retrieved context |
| Answer Relevance | On-topic response | QA | Not correctness |
| Toxicity / Bias / Harm | Safety | All GenAI | Threshold tuning needed |
| Task Completion | Agent success | Agents | Needs trajectory review |
| Latency / Cost | Operational efficiency | All | Not a quality metric |
AWS Services for AI Testing & CI/CD
| Service | Role |
|---|---|
| SageMaker Pipelines | Orchestrate processing, training, evaluation, condition steps, registration |
| SageMaker Model Registry | Version models, approval status, lineage to training run |
| SageMaker Clarify | Bias, explainability, FM evaluation |
| SageMaker Model Monitor | Data/quality drift, online monitoring |
| Bedrock Model Evaluation | Automatic, human, LLM-as-a-judge evaluations |
| Bedrock Prompt Management | Versioned prompts, variables, variants |
| Bedrock Guardrails | Safety filters, PII, denied topics |
| CodePipeline / CodeBuild / CodeDeploy | CI/CD stages, build/test, rollout/rollback |
| CodeConnections | External repo link without stored credentials |
| EventBridge | Triggers on data arrival, approval, schedule |
| MLflow on SageMaker | Experiment tracking, auto-register winning run |
| CloudWatch | Alarms for rollback and operational metrics |
CI/CD Pipeline Flow for AI
- Source: Commit model code, prompts, agent config in CodeCommit or external repo via CodeConnections.
- Build/Test: CodeBuild runs unit tests, container builds, lint, security scans.
- Evaluate: SageMaker Pipelines or Bedrock Eval runs golden dataset; compute metrics; compare baseline.
- Gate: Threshold pass → register in Model Registry with approval status.
- Deploy: CodeDeploy or SageMaker endpoint with canary/linear; Bedrock deploy by model/prompt version ID.
- Monitor: CloudWatch alarms, Model Monitor; auto-rollback if alarm trips.
- Retrain/Refresh: EventBridge on data arrival or schedule; RAG incremental sync.
RAG & Agent Testing Specifics
- RAG: Test retrieval recall, context relevance, faithfulness, answer relevance. Incremental sync only processes added/changed/deleted docs. Embedding model or chunking change = full reindex.
- Agents: Test tool selection, argument correctness, task success, guardrail compliance. Version agent config, tools, and prompts.
- Fine-tuned FM: Evaluate against base model; regression tests; version by model ID; deploy by identifier, not in-place edit.
Security & Compliance
- IAM roles: Least privilege for SageMaker, Bedrock, CodePipeline, CodeBuild.
- KMS: Encrypt S3 artifacts, ECR images, CodeBuild artifacts, Bedrock custom models, logs.
- VPC endpoints: Private access to SageMaker, Bedrock, S3, KMS, CodeBuild.
- Secrets: No credentials in prompts or pipeline definitions; use Secrets Manager.
- Logging: CloudTrail, Bedrock invocation logging, Model Monitor.
- Guardrails: PII redaction, toxicity, denied topics; test guardrails in pipeline.
Trade-offs & Anti-Patterns
- Automated vs human eval: Automated is scalable/cheap; human is accurate/nuanced; combine.
- LLM-as-judge vs human: Judge scales but may be biased; calibrate with human labels.
- Offline vs online: Offline safe/cheap; online real-world; canary limits blast radius.
- Canary vs blue/green: Canary limits exposure; blue/green instant switch; both need alarms.
- Full reindex vs incremental: Incremental cheap; full needed for embedding/chunking changes.
- Anti-patterns: Exact-match LLM tests; no baseline; prompts in code; no versioning; manual-only gates; ignoring non-determinism; no rollback alarm; training on eval set; unpinned model versions; single metric.
Exam Gotchas
Prompt changes are model changes. Version prompts and test them like weights. Non-deterministic outputs require aggregate scoring against a golden dataset, not exact-match assertions. Automated rollback requires a CloudWatch alarm; a rollout with no alarm will not roll back. For RAG, incremental sync only processes changed docs; embedding model or chunking changes require full reindex. Bedrock Model Evaluation supports automatic, human, and LLM-as-a-judge; use LLM-as-a-judge for scale, human for calibration. Managed rollout shapes (canary/linear) are not available for all hosting options; confirm support before designing release. CodePipeline/CodeBuild test code; SageMaker Pipelines/Bedrock Eval test models/prompts. Don't confuse the two.