Skill 2.3.2: Create Model Performance Baselines and Implement Drift Detection
Task 2.3: ML Model and Foundation Model (FM) Development - Analyze and evaluate the performance of ML and AI systems
Objective: Establish a known-good reference (baseline) and continuously detect when production inputs, predictions, or GenAI outputs depart from it, so you can retrain, rollback, or alert before business impact.
Core Concepts
- Baseline – Recorded statistics, constraints, and metrics from a known-good period (training data or validated production data). Serves as the comparison point.
- Drift – Statistically significant change in input distribution (data drift), prediction quality (model quality drift), fairness (bias drift), feature importance (explainability drift), or GenAI output quality.
- Business objective – Maintain model trust, compliance, and performance; trigger retraining or rollback; reduce silent degradation.
AWS Services & Roles
| Service | Primary Role for Baselines & Drift |
|---|---|
| SageMaker Model Monitor | Managed drift detection: data quality, model quality, bias, feature attribution. |
| SageMaker Clarify | Bias and explainability baselines; SHAP feature attribution drift. |
| SageMaker MLflow | Track experiments, parameters, metrics, artifacts for reproducible baselines. |
| CloudWatch | Metrics, logs, alarms for drift violations and endpoint health. |
| Bedrock Evaluations | GenAI/FM quality baselines: programmatic, LLM-as-a-judge, human. |
| Bedrock Prompt Management | Versioned prompts and variant comparison to detect prompt drift. |
| Bedrock Knowledge Bases | Retrieval accuracy assessment for RAG systems. |
| S3 | Stores baseline artifacts, data capture, ground truth. |
| IAM/KMS | Access control and encryption for baseline/capture data. |
Baseline Types & Artifacts
| Baseline Type | What It Measures | Input Needed | Output Artifacts | Drift Detected |
|---|---|---|---|---|
| Data quality | Input feature distribution | Training/reference data | statistics.json, constraints.json | Data drift |
| Model quality | Prediction accuracy vs ground truth | Labeled data | Model quality metrics | Model quality drift |
| Bias | Fairness metrics across groups | Labeled data + sensitive attributes | Bias constraints | Bias drift |
| Explainability | Feature attribution (SHAP) | Training data | Explainability constraints | Feature attribution drift |
| GenAI quality | Output quality, faithfulness, relevance | Prompt dataset + references | Evaluation metrics | GenAI quality drift |
Drift Types & Detection Trade-offs
| Drift Type | What Changes | Labels Required? | Detection Latency | Typical Metrics |
|---|---|---|---|---|
| Data drift | Input distribution | No | Immediate | KS, Chi-square, L-infinity |
| Model quality drift | Accuracy/precision/recall | Yes | Delayed | F1, MAE, RMSE |
| Bias drift | Fairness metrics | Yes | Delayed | DPPL, DCA |
| Feature attribution drift | Feature importance | No | Immediate | SHAP values |
| GenAI quality drift | Output quality | Often yes (human/judge) | Varies | ROUGE, BLEU, BERTScore, LLM-judge |
Exam Gotcha: Model Monitor is asynchronous – it detects drift but does not block inference. Use it for monitoring, not real-time prevention.
Exam Gotcha: Data capture must be enabled on the endpoint before Model Monitor can analyze live traffic. No data capture = no drift detection.
Exam Gotcha: Model Monitor requires baseline artifacts. You cannot create a monitoring schedule without a baseline job.
Implementation Workflow
- Establish baseline – Run baseline job on representative known-good data; store statistics/constraints in S3.
- Enable data capture – On the endpoint; captures requests/responses to S3 (can sample).
- Create monitoring schedule – Hourly/daily; compares captured data to baseline.
- Collect ground truth – For model quality/bias drift; join labels with captured data.
- Set CloudWatch alarms – On violation metrics; notify via SNS.
- Act on drift – Retrain, update baseline, or rollback; re-evaluate.
Decision Matrix: Which Approach?
| Requirement | Recommended Approach | Why |
|---|---|---|
| Early warning of input changes | Data quality baseline + Model Monitor | No labels needed; fast. |
| Actual prediction degradation | Model quality baseline + ground truth | Measures real performance. |
| Fairness/compliance monitoring | Clarify bias baseline | Detects bias drift. |
| Explainability drift | Clarify explainability baseline | Tracks feature importance changes. |
| GenAI/FM quality baseline | Bedrock Evaluations | Programmatic, LLM-judge, human. |
| Prompt wording changes | Bedrock Prompt Management | Versioned prompts, variant comparison. |
| RAG retrieval accuracy | Bedrock Knowledge Bases evaluation | Separates retrieval from generation. |
| Reproducible experiments | SageMaker MLflow | Tracks params, metrics, artifacts. |
GenAI/FM Drift Considerations
- Monitor inputs/outputs via Bedrock model invocation logging to CloudWatch/S3.
- Prompt drift – Version prompts; compare variants in Bedrock Prompt Management.
- Quality drift – Periodic Bedrock Evaluations: programmatic (ROUGE/BLEU/BERTScore), LLM-as-a-judge, human.
- RAG drift – Measure retrieval (context precision/recall) separately from generation (faithfulness, answer relevance).
- Human-in-the-loop – Use for high-consequence outputs; treat ratings as data.
- Anti-pattern – Single accuracy figure for text generation; valid answers can have different wording. Use semantic + overlap metrics.
Exam Gotcha: In RAG, a fluent answer that omits source facts is often a retrieval failure, not a generation failure. Measure retrieval and generation separately.
Security & Compliance
- IAM roles – Model Monitor needs S3 read/write, CloudWatch PutMetricData, KMS decrypt/encrypt.
- KMS – Encrypt S3 baseline artifacts, data capture, and CloudWatch Logs.
- VPC endpoints – For S3, CloudWatch, SageMaker if no internet access.
- S3 bucket policies – Restrict access to baseline and capture buckets.
- CloudTrail – Audit Model Monitor and Bedrock evaluation API calls.
Anti-Patterns & Exam Traps
- No baseline – Cannot detect drift.
- Unrepresentative baseline – False alarms or missed drift.
- Monitoring only infrastructure – CPU/latency ≠ model quality.
- Ignoring delayed labels – Model quality drift requires ground truth; set up labeling.
- Expecting Model Monitor to block bad predictions – It is asynchronous.
- Forgetting data capture – Model Monitor has no data to analyze.
- Confusing data drift with model quality drift – Data drift is early warning; quality drift is actual degradation.
- Not setting CloudWatch alarms – Drift detected but no action taken.
- GenAI: single metric – Text generation needs multiple metrics (semantic + overlap + human/judge).
- RAG: measuring only final answer – Isolates neither retrieval nor generation.
Quick Exam Summary
- Baseline = known-good reference (statistics, constraints, metrics).
- Drift = departure from baseline.
- SageMaker Model Monitor = managed drift detection.
- Clarify = bias + explainability.
- CloudWatch = metrics, alarms, logs.
- Bedrock Evaluations = GenAI/FM quality.
- Prompt Management = versioned prompts.
- Knowledge Bases = retrieval accuracy.
- Data capture = prerequisite for Model Monitor.
- Ground truth = prerequisite for model quality/bias drift.