Skip to content

Skill 2.3.7: Implement integrated human evaluation frameworks

Task 2.3: ML Model and Foundation Model (FM) Development – Analyze and evaluate the performance of ML and AI systems

Core objective: Combine automated metrics + LLM-as-a-judge + human judgment to validate subjective GenAI quality, safety, bias, and high-stakes outputs. Human evaluation is the ground truth for open-ended text; use it to calibrate automated metrics and gate deployments.

When Human Evaluation Is Required vs Optional

Condition Use Human Eval? Why
Subjective quality: tone, helpfulness, creativity, safety Yes Multiple correct wordings; no single automated metric captures quality
High-stakes decisions: health, finance, legal, hiring Yes + HITL review Consequence requires audit trail and escalation
Objective classification/regression No Use accuracy, F1, RMSE, AUC
RAG factuality Hybrid Retrieval metrics + generation metrics + human spot-check for omissions
Safety/bias/toxicity Hybrid Guardrails/Clarify + human review for context
Calibrating LLM-as-judge Yes Human ratings anchor judge reliability

AWS Service Selection for Human Evaluation

Service Primary Use Key Config / Exam Notes
Amazon Bedrock Evaluations Managed model/KB evaluation: automatic, LLM-as-judge, human Human jobs require console-created work team; automated can be programmatic. Supports pairwise/rating scales. Evaluates models and RAG sources.
Amazon SageMaker Ground Truth Labeling + human evaluation datasets, RLHF preference ranking Private/vendor/Mechanical Turk workforce. Use for custom rubrics, pairwise comparisons, ranking. Output to S3.
Amazon Augmented AI (A2I) Human review of ML predictions in production Integrates with Rekognition, Textract, custom SageMaker endpoints. Trigger on low confidence. Not a labeling service.
Amazon SageMaker Clarify Bias detection & explainability Use with human review for sensitive attributes; not human eval itself.
Amazon Bedrock Guardrails Automated safety/PII/grounding controls Reduce human review volume; human eval validates residual risk.
Amazon CloudWatch / Model Monitor Operational metrics & drift Baselines, alarms; human feedback can be a custom metric.

Human-in-the-Loop (HITL) Patterns

  • Review low-confidence predictions: A2I or custom workflow; route uncertain cases to humans; feed corrections back to training.
  • RLHF / preference tuning: Ground Truth pairwise ranking; humans choose better response; data used for reward model/fine-tuning.
  • Content moderation: Guardrails first; human review for appeals/edge cases.
  • RAG answer validation: Human reviews retrieved passages + final answer for faithfulness/completeness/citation.
  • Continuous feedback: In-app thumbs up/down, comments; store in S3/CloudWatch; use for eval sets, not direct training without curation.

Exam Gotcha: A2I reviews predictions; Ground Truth labels data. Do not confuse them. Bedrock human evaluation is for model/KB evaluation jobs, not live production review.

Text Generation Quality Assessment

Metric Type Examples Use / Trap
Overlap BLEU, ROUGE, METEOR, chrF Fast, reference-based; poor for paraphrase/meaning. BLEU for MT, ROUGE for summarization.
Embedding/Semantic BERTScore, semantic similarity, MoverScore Captures meaning; needs embedding model; not a substitute for factuality.
LLM-as-a-Judge Bedrock Evaluations, custom judge Scalable; watch position bias, self-preference, verbosity bias. Calibrate with humans.
Human Likert, pairwise, ranking, rubric Gold standard for subjective quality; expensive, needs IAA.
RAG Triad Faithfulness, Answer Relevance, Context Relevance Split retrieval vs generation to isolate failures.

Hard Rule: For open-ended generation, no single accuracy/BLEU score proves quality. Combine overlap + semantic + judge + human for high-stakes.

Designing Human Evaluation Rubrics

  • Define dimensions: Correctness, completeness, faithfulness/groundedness, relevance, fluency, coherence, safety, bias, toxicity, tone.
  • Scale: Likert 1–5, binary pass/fail, pairwise preference, ranking.
  • Controls: Blind model identity, randomize order, gold questions, attention checks, multiple annotators per item.
  • Reliability: Measure inter-annotator agreement (Cohen’s/Fleiss’ kappa, Krippendorff’s alpha). Adjudicate disagreements.
  • Sampling: Stratify by use case, risk, language, demographic. Ensure enough items for confidence intervals.
  • Workforce: Private workforce for sensitive data; vendor for scale; SMEs for domain-specific.

Integrated Evaluation Pipeline (Conceptual)

  1. Offline automated: Bedrock Evaluations / SageMaker metrics on holdout set.
  2. LLM-as-judge: Score at scale with rubric; compare to human calibration set.
  3. Human evaluation: Ground Truth/A2I/Bedrock human jobs for subjective/high-risk samples.
  4. RAG split: Measure retrieval (context precision/recall) separately from generation (faithfulness/relevance).
  5. Production monitoring: CloudWatch/Model Monitor for drift; capture user feedback; re-evaluate.
  6. Gate deployment: Promote only if automated, judge, and human thresholds pass.

Security, Compliance, and Privacy for Human Review

  • IAM: Least-privilege roles for Bedrock, SageMaker, S3, Ground Truth, A2I. Separate execution role from reviewer access.
  • Encryption: KMS CMK at rest for S3, Bedrock, SageMaker. TLS in transit.
  • Network: VPC endpoints for Bedrock/SageMaker/S3 to avoid public internet. Private workforces in VPC.
  • Data handling: Anonymize/PII redact before human review. Use private workforce for sensitive data. Set retention and audit logs.
  • Compliance: Human review logs, annotation provenance, model cards, audit trails for regulated use.

Common Anti-Patterns / Distractor Traps

  • Using only BLEU/ROUGE for GenAI: Misses meaning and factuality. Add semantic/LLM/human.
  • Using only LLM-as-judge: Can be biased; always calibrate with human ratings.
  • Using A2I for data labeling: Wrong service; use Ground Truth.
  • Using Ground Truth for live production review: Ground Truth is batch labeling; use A2I for real-time/low-confidence review.
  • Skipping retrieval metrics in RAG: Fluent answer omitting source facts may be retrieval failure, not generation.
  • Ignoring inter-annotator agreement: Human ratings without reliability are anecdotal.
  • Assuming Bedrock human eval is programmatic: Human jobs require console/work team setup; automated can be API-driven.
  • No baseline: Human eval without baseline or control is not actionable.

Quick Decision Matrix

Scenario Best Approach
Compare two FM responses for subjective quality Bedrock human eval pairwise or Ground Truth ranking
Review low-confidence ML predictions in production A2I
Label data for RLHF preference tuning Ground Truth
Evaluate RAG retrieval accuracy Bedrock KB evaluation retrieval metrics
Evaluate RAG answer faithfulness Bedrock KB evaluation generation metrics + human spot-check
Detect bias in model outputs Clarify + human review
Monitor production drift CloudWatch/Model Monitor + human feedback loop