Skill 2.3.8: Apply NLP Evaluation Metrics (BLEU, ROUGE, BERTScore, Semantic Similarity)
Task 2.3: ML Model and Foundation Model (FM) Development – Analyze and evaluate the performance of ML and AI systems
Core Concept: Quantify text generation quality. Choose metric aligned to task: BLEU for translation (precision), ROUGE for summarization (recall), BERTScore/semantic similarity for meaning. AWS objective: automate evaluation via Bedrock Evaluations, track with SageMaker MLflow, monitor drift in CloudWatch, and separate RAG retrieval from generation.
Metric Selection Matrix
| Metric | Measures | Best For | Weakness | Reference Needed? |
|---|---|---|---|---|
| BLEU | n-gram precision + brevity penalty | Machine translation | Ignores meaning, synonyms; poor for long text | Yes |
| ROUGE-N | n-gram recall | Summarization | Ignores meaning, word order | Yes |
| ROUGE-L | Longest common subsequence | Summarization | Same as ROUGE-N | Yes |
| BERTScore | Contextual embedding cosine similarity (P/R/F1) | Any text generation, semantic equivalence | Requires model; less interpretable; compute cost | Yes (or reference-free variant) |
| Semantic Similarity | Cosine similarity of embeddings | RAG, Q&A, retrieval, paraphrase detection | Depends on embedding model; no absolute scale | No (can compare to query/context) |
| METEOR/chrF | Alignment-based, character n-grams | Translation, morphologically rich languages | Less common in AWS exams | Yes |
Exam Gotcha: BLEU = precision-focused; ROUGE = recall-focused. Do not swap them. Neither captures meaning—pair with BERTScore or semantic similarity for open-ended tasks.
AWS Service Mapping
| AWS Service | Role in NLP Evaluation |
|---|---|
| Amazon Bedrock Evaluations | Runs automatic (programmatic) jobs computing BLEU, ROUGE, BERTScore, etc.; LLM-as-a-judge; human worker jobs. |
| Amazon Bedrock Knowledge Bases | RAG retrieval source; evaluate retrieval accuracy (Context Precision/Recall) separately from generation. |
| Amazon Bedrock Prompt Management | Version prompts; compare variants to isolate prompt changes. |
| SageMaker AI with MLflow | Track evaluation metrics, parameters, artifacts across runs; reproduce experiments. |
| SageMaker AI shadow tests | Compare candidate vs production on live traffic copy; production answers caller. |
| Amazon CloudWatch | Monitor metric baselines and drift; alarm on evaluation score degradation. |
RAG Evaluation Split (Critical)
| Layer | Metrics | Failure Symptom |
|---|---|---|
| Retrieval | Context Precision, Context Recall, semantic similarity | Missing facts from source documents |
| Generation | Faithfulness, Answer Relevance, BERTScore | Fluent answer omits retrieved facts; hallucination |
Exam Gotcha: A fluent answer that omits source facts is a retrieval failure, not a generation failure. Always measure retrieval and generation separately.
Human vs Automated Evaluation
| Mode | Pros | Cons | AWS Implementation |
|---|---|---|---|
| Programmatic | Fast, cheap, reproducible | No semantic judgment; metric gaming | Bedrock Evaluations automatic; SageMaker MLflow |
| LLM-as-a-judge | Scalable, explains score, handles open-ended | Model bias, cost, needs calibration | Bedrock Evaluations with judge model |
| Human | Gold standard for subjective quality, bias, tone | Slow, expensive, non-reproducible | Bedrock Evaluations human jobs (console requires browser display) |
Exam Gotcha: Bedrock human evaluation jobs created in console require a browser-facing display; automated jobs do not.
Decision Matrix: When to Use X vs Y
| Scenario | Recommended Metric(s) |
|---|---|
| Machine translation | BLEU (primary), METEOR, BERTScore |
| Summarization | ROUGE-N, ROUGE-L, BERTScore |
| Open-ended generation / chat | BERTScore, semantic similarity, LLM-as-a-judge, human |
| RAG Q&A | Context Precision/Recall (retrieval) + Faithfulness/Answer Relevance (generation) |
| Retrieval only | Context Precision/Recall, semantic similarity |
| Classification | Accuracy, F1, precision/recall (not BLEU/ROUGE) |
| Bias detection | Human evaluation, LLM-as-a-judge with bias prompts, Bedrock Guardrails |
Implementation Nuances & Anti-Patterns
- Tokenization sensitivity: BLEU/ROUGE depend on tokenizer, casing, stemming. Use consistent preprocessing.
- Corpus vs sentence: BLEU/ROUGE often corpus-level; BERTScore per sentence. Align with evaluation goal.
- Embedding model consistency: Semantic similarity and BERTScore require same model for fair comparison. Model drift changes scores.
- Reference quality: BLEU/ROUGE/BERTScore need high-quality references. Poor references invalidate metrics.
- Single metric trap: Never rely on one metric. Combine overlap, semantic, and human/LLM judgment.
- Anti-pattern: Using BLEU for summarization or ROUGE for translation without understanding precision/recall.
- Anti-pattern: Evaluating only final answer in RAG without isolating retrieval.
- Anti-pattern: Ignoring prompt/dataset versioning; use Bedrock Prompt Management and MLflow.
Security & Compliance
- IAM roles: Bedrock evaluation jobs, SageMaker MLflow, S3 access require least-privilege roles.
- KMS encryption: Encrypt evaluation datasets, human review data, MLflow artifacts at rest.
- VPC endpoints: Use PrivateLink for Bedrock, SageMaker, S3 to keep evaluation data private.
- Human review: Sensitive data may require private workforce or internal employees; ensure data handling compliance.
- Bias & fairness: Evaluate outputs for bias; use Bedrock Guardrails or custom metrics.
Key Configurations
- Bedrock Evaluations: Provide prompt dataset in S3; choose automatic (BLEU, ROUGE, BERTScore, etc.) or human; for human, configure work team and UI.
- SageMaker MLflow: Log metrics (bleu, rouge, bertscore) as metrics; parameters; artifacts.
- CloudWatch: Set alarms on evaluation score thresholds; monitor drift from baseline.
- Bedrock Knowledge Bases: Evaluate retrieval using context precision/recall; sync data sources.
- Shadow tests: Route copy of live requests to shadow variant; log responses; production answers caller.
Exam Gotcha: Shadow testing is not A/B testing. A/B splits traffic and exposes users to candidate; shadow copies traffic and only production answers.