Skill 2.3.6: Apply comprehensive model evaluation techniques for traditional ML and GenAI models.
Task 2.3: ML Model and Foundation Model (FM) Development - Analyze and evaluate the performance of ML and AI systems.
Core objective: Prove a model is production-ready by measuring quality, safety, and drift against a reproducible baseline—before and after deployment. Match the metric to the output type; never use one score for all problems.
Evaluation Landscape: Traditional ML vs GenAI
| Aspect | Traditional ML | GenAI |
|---|---|---|
| Output | Discrete label / continuous value | Free-form text, summaries, answers |
| Primary metrics | Accuracy, F1, RMSE, AUC | BLEU, ROUGE, BERTScore, LLM-as-judge |
| Reference | Ground-truth labels | Reference text or human preference |
| Key challenge | Overfitting, class imbalance | Multiple valid answers, hallucination, bias |
| AWS tools | SageMaker Clarify, Model Monitor, Experiments | Bedrock Evaluations, Prompt Management, Knowledge Bases |
Traditional ML Evaluation Metrics
| Problem | Metrics | When to use / Gotcha |
|---|---|---|
| Classification | Accuracy, Precision, Recall, F1, AUC-ROC, Log Loss, Confusion Matrix | F1 for imbalance; AUC for ranking; Log Loss for calibration. Accuracy misleading on skewed classes. |
| Regression | RMSE, MAE, R2, MAPE | RMSE penalizes large errors; MAE robust; R2 relative fit; MAPE breaks on zeros. |
| Clustering | Silhouette, Davies-Bouldin, Calinski-Harabasz | No labels. Use multiple indices; validate with domain expert. |
| Ranking / Recommendation | NDCG, MRR, MAP, Precision@K, Recall@K | NDCG for graded relevance; MRR for first relevant hit. |
| Anomaly Detection | Precision/Recall at threshold, F1, AUC-PR | AUC-PR better than ROC for rare anomalies. |
| Time Series | RMSE, MAE, MAPE, sMAPE, MASE | MASE scale-independent; backtest on rolling windows. |
Gotcha: Never evaluate on training data. Hold out a test set; use cross-validation for small data. Class imbalance makes accuracy a vanity metric.
GenAI Evaluation Metrics
| Category | Metrics | Purpose / Gotcha |
|---|---|---|
| Overlap-based | BLEU (translation), ROUGE (summarization), METEOR, CIDEr | Fast, cheap, but poor for paraphrases and semantics. |
| Embedding / Semantic | BERTScore, cosine similarity, Sentence-BERT | Captures meaning, not exact words. |
| LLM-as-a-judge | Faithfulness, Answer Relevance, Context Relevance, Correctness, Harmlessness, Helpfulness | Scalable nuance, but judge bias possible. Calibrate with human ratings. |
| Human evaluation | Likert, pairwise preference, rubric-based | Gold standard for nuance; expensive, slow, needs structured rubric. |
| RAG triad | Context Relevance, Faithfulness, Answer Relevance | Evaluate retrieval and generation separately. |
| QA | Exact Match, F1, ROUGE-L | Exact Match strict; F1 partial credit. |
| Summarization | ROUGE-½/L, BERTScore, factual consistency | ROUGE misses factual errors; add consistency check. |
| Translation | BLEU, METEOR, chrF, COMET | COMET correlates better with human judgment. |
Gotcha: A single accuracy figure is useless for GenAI. Combine overlap + semantic + judge + human for high-stakes use cases.
RAG Evaluation: Split Retrieval from Generation
| Component | Metric | Failure mode |
|---|---|---|
| Retrieval | Context Precision, Context Recall, MRR, NDCG | Missing relevant documents |
| Generation | Faithfulness, Answer Relevance, Groundedness | Hallucination, omission of source facts |
| End-to-end | Correctness, Completeness, Helpfulness | Cannot isolate root cause |
Exam Gotcha: A fluent answer that omits source facts is often a retrieval failure. Measure retrieval separately from generation to isolate cause.
Evaluation Modes in Amazon Bedrock
| Mode | How it works | Best for |
|---|---|---|
| Programmatic | Computed metrics (BLEU, ROUGE, BERTScore) on dataset | Repeatable, low cost, objective |
| LLM-as-a-judge | Judge model scores + explains | Scalable nuance, subjective quality |
| Human | Human workers rate in console / workflow | High-stakes, nuanced, preference |
Gotcha: Human-based evaluation jobs created in the console require browser-facing display for prompts/responses. Automated jobs do not.
Baselines, Drift, and Monitoring
- Baseline: Record known-good metrics, input distributions, and operational metrics at launch. Use SageMaker Model Monitor.
- Drift types: Data drift (input distribution), Model quality drift (accuracy/F1), Bias drift (fairness), Feature attribution drift (SHAP).
- CloudWatch: Stores endpoint metrics (latency, errors, invocations) and custom metrics for drift alerts.
- SageMaker Clarify: Detects bias pre-training and post-training, explains predictions with SHAP.
- SageMaker MLflow: Tracks parameters, metrics, artifacts, and traces for GenAI apps.
- Bedrock Prompt Management: Versions prompts and compares variants for repeatable prompt evaluation.
Gotcha: A baseline without input distribution is only half useful. Record inputs + outcomes to distinguish data drift from model degradation.
Comparing Variants: Shadow, A/B, Canary, Blue/Green
| Strategy | Traffic | Caller impact | Use case |
|---|---|---|---|
| Shadow | Copy to shadow, production serves | None | Validate new model/container/instance on live traffic |
| A/B | Split traffic | Users see candidate | Measure business impact |
| Canary | Small % to new | Some users | Gradual rollout, rollback |
| Blue/Green | Switch after validation | None after switch | Zero-downtime release |
Exam Gotcha: Shadow testing routes a copy of live requests to the shadow variant; only production responses go back to callers. Not all endpoint configurations support shadow testing.
Debugging Convergence Issues
| Symptom | Likely cause | Action |
|---|---|---|
| Loss oscillates, never settles | Learning rate too high | Reduce LR, add decay |
| Loss barely moves | LR too low, model capacity too small | Increase LR, add layers/features |
| Train improves, validation worsens | Overfitting | Regularization, dropout, more data, early stopping |
| Both train and validation poor | Underfitting | More capacity, better features, train longer |
| Loss = NaN | Exploding gradients, bad data | Gradient clipping, check data, lower LR |
Security, Compliance, and Operational Guardrails
- IAM roles: Evaluation jobs need S3 read/write, Bedrock access, CloudWatch Logs, KMS decrypt.
- Encryption: KMS for data at rest (S3, EFS), TLS in transit. VPC endpoints for private access to SageMaker/Bedrock.
- Data privacy: Human evaluation must not expose PII. Use redaction, private workforces, or synthetic data.
- Model governance: Bedrock Guardrails for content filtering, PII redaction, denied topics. CloudWatch + CloudTrail for audit.
- Bias detection: SageMaker Clarify for traditional ML; Bedrock evaluations for GenAI bias/toxicity.
Anti-Patterns & Distractor Traps
- Single metric for GenAI: Accuracy or BLEU alone misses meaning, hallucination, bias.
- End-to-end RAG only: Cannot tell if retrieval or generation failed.
- No baseline: Drift detection impossible.
- Offline only: Offline scores disagree with live traffic; use shadow tests.
- Tuning on test set: Inflates performance; use validation set.
- Ignoring class imbalance: Accuracy misleading; use F1, AUC-PR.
- Human eval without rubric: Inconsistent, not reproducible.
- Assuming shadow testing always available: Some endpoint configs excluded.
- Forgetting judge bias: LLM-as-a-judge can favor own outputs; calibrate with human ratings.
Final exam rule: For traditional ML, match metric to problem type. For GenAI, combine overlap, semantic, judge, and human evaluation. For RAG, always measure retrieval and generation separately.