Skill 2.3.7: Implement integrated human evaluation frameworks (for example, human-in-the-loop workflows, text generation quality assessment)

Question 1 - Human Review Workflow Integration

A financial services company is building a document summarization system using Amazon Bedrock with Claude 3 models. The system processes quarterly earnings reports and generates executive summaries. Compliance officers require that summaries containing forward-looking statements or risk disclosures must be reviewed by a human compliance specialist before being distributed to stakeholders. The solution must integrate human review directly into the processing workflow while minimizing architectural complexity.

Which AWS service combination should the ML engineer use to implement this human-in-the-loop review workflow? (Select ONE)

A) Amazon Bedrock Guardrails with an auto-reject policy trigger an Amazon EventBridge event routed to a custom review portal hosted on EC2.
B) Amazon Bedrock Knowledge Bases with Bedrock Evaluations in human-review mode.
C) Amazon Bedrock Guardrails with a return control provision and an AWS Step Functions workflow that invokes an Amazon API Gateway endpoint; this routes the summary to a custom SaaS reviewer portal.
D) AWS Glue ETL job with an Amazon Mechanical Turk task embedded in the pipeline.

Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

C is Correct: Amazon Bedrock Guardrails can detect content that requires human review (via denylisted phrases, sensitive topics, or content triggers like forward-looking statements) and return a control signal to Step Functions. Step Functions orchestrates the review flow with a Wait for Callback integration, pausing execution until a reviewer approves through a custom portal exposed via API Gateway. This satisfies the requirement with minimal custom code while keeping the workflow audit-ready.

Distractor Analysis

A is Incorrect: Amazon Bedrock Guardrails uses EventBridge integration, but building a custom reviewer portal on EC2 introduces significant operational overhead with server management. While functionally possible, the architectural complexity exceeds the stated minimal-architecture constraint and unnecessarily diverges from the recommended serverless Step Functions + API Gateway pattern.
B is Incorrect: Bedrock Knowledge Bases evaluates retrieval and generated answer quality, not human-in-the-loop workflow routing. Bedrock Evaluations can execute human-review evaluation jobs, but they are for rating model outputs, not for embedding a governance approval gate inside a production processing pipeline.
D is Incorrect: Mechanical Turk is designed for crowd-sourced, general-purpose tasks and is not suitable for compliance review requiring specialized internal stakeholder qualifications. Additionally, embedding MTurk in a Glue ETL pipeline is architecturally inappropriate because Glue is for batch data transformation rather than interactive review workflow orchestration.

Key Takeaway: Use Amazon Bedrock Guardrails' return control feature with Step Functions' waitForTaskToken to build serverless human-in-the-loop review workflows without managing infrastructure.

Question 2 - Text Generation Quality Assessment Framework

A content generation startup uses Amazon Bedrock to generate product descriptions for e-commerce clients. The quality assurance team needs to assess the generated descriptions along multiple dimensions: readability, factual accuracy against product specifications, tone consistency, and brand compliance. They want automated scoring combined with intermittent human spot-checks. The solution must minimize operational overhead and provide structured, comparable ratings across thousands of generations monthly.

Which approach should the ML engineer recommend? (Select ONE)

A) Run Amazon Bedrock Evaluations using an LLM-as-a-judge model with custom scoring rubrics, and use Amazon SageMaker Ground Truth for periodic human-labeling jobs on a sampled subset.
B) Deploy a custom Python script using ROUGE and BLEU metrics running on an EC2 instance, and use Amazon S3 for storing generated text for manual review.
C) Create an Amazon SageMaker Ground Truth labeling job with a private workforce for all generated descriptions.
D) Use Amazon Comprehend sentiment analysis for automated scoring and Amazon CloudWatch dashboards for monitoring.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

A is Correct: Amazon Bedrock Evaluations supports LLM-as-a-judge evaluation where a second model (e.g., Claude) evaluates generated outputs against custom metrics with defined rubrics. This provides scalable, structured scoring for readability, factual accuracy, tone, and brand compliance. Amazon SageMaker Ground Truth with a private workforce handles intermittent human spot-checks, providing periodic human-labeled validation of the automated scoring while maintaining low operational overhead.

Distractor Analysis

B is Incorrect: Running a custom script on an EC2 instance introduces unnecessary operational overhead (server management, scaling concerns). ROUGE and BLEU are n-gram overlap metrics that cannot accurately evaluate tone, brand compliance, or factual accuracy. This approach requires significant manual setup for thousands of monthly generations.
C is Incorrect: Human review of all generated descriptions via Ground Truth is not feasible or cost-effective for thousands of monthly items. This introduces significant operational overhead, latency, and cost, while ignoring the automation requirement in the scenario.
D is Incorrect: Amazon Comprehend's sentiment analysis evaluates emotional sentiment (positive, negative, neutral, mixed), which is insufficient for assessing readability, factual accuracy, tone consistency, or brand compliance. CloudWatch dashboards are for monitoring metrics, not for evaluating text generation quality.

Key Takeaway: Bedrock Evaluations with LLM-as-a-judge provides scalable, rubric-based automated assessment across multiple quality dimensions, while Ground Truth private workforces handle the spot-check requirement.

Question 3 - Socratic Evaluation with Amazon Bedrock

An ML team at a healthcare technology company is developing an AI system for generating clinical documentation summaries using Amazon Bedrock. The system must minimize hallucination risk, align with established medical guidelines, and adhere to a strict zero-tolerance policy for fabricated medical claims. The compliance team requires structured, programmatic evaluation with a documented rationale behind each quality score. The solution must not require training a custom model and must minimize operational overhead.

Which evaluation method should the ML engineer implement? (Select ONE)

A) Compute ROUGE-L scores comparing generated summaries against physician-written reference summaries.
B) Use Amazon Bedrock Evaluations with a judge model that evaluates answer correctness, grounding, and faithfulness using an integrated Socratic grading methodology.
C) Deploy a BERT-based classifier via Amazon SageMaker to detect hallucinations in generated clinical summaries.
D) Track model outputs in Amazon SageMaker MLflow and manually review the top 100 highest-token summaries quarterly.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: Amazon Bedrock Evaluations with a judge model (LLM-as-a-judge) can evaluate generative AI outputs against multiple quality dimensions including grounding (whether the answer is rooted in source material), faithfulness (whether it correctly represents source content), and correctness. The Socratic grading feature in Bedrock Evaluations provides step-by-step reasoning behind each score. This directly addresses the requirement for documented rationale without needing custom model training, while maintaining minimal operational overhead.

Distractor Analysis

A is Incorrect: ROUGE-L is an n-gram overlap metric that measures surface-level similarity between generated and reference text. It cannot detect fabricated medical claims or assess factual alignment with medical guidelines. Two summaries can share many overlapping words while still containing dangerous hallucinations.
C is Incorrect: Deploying a custom BERT classifier on SageMaker requires significant operational overhead including model training, tuning, endpoint management, and monitoring, while providing no inherent alignment with medical guideline compliance. This violates the "no custom model training" constraint.
D is Incorrect: Tracking outputs in MLflow is for experiment management, not for evaluation. Manually reviewing quarterly introduces unacceptable latency for a zero-tolerance compliance environment and is not a scalable or defensible evaluation framework.

Key Takeaway: Bedrock Evaluations with Socratic grading provides rubric-based, explainable evaluation that satisfies compliance documentation requirements without custom model development.

Question 4 - Step Functions Human Review Pattern

A legal tech company uses Amazon Bedrock to draft contract clauses. The legal operations team requires that clauses involving high-risk terms (e.g., indemnification, liability waivers) must be approved by a licensed attorney before being presented to clients. The workflow must be resilient to reviewer availability gaps, maintain an audit trail of all approvals and rejections, and minimize operational overhead. The engineering team is proficient in AWS Step Functions.

Which architecture should the ML engineer implement for this human review workflow? (Select ONE)

A) Use Amazon Bedrock Guardrails with custom word filters; when triggered, invoke an Amazon SNS topic that emails a queue of attorneys; attorneys reply via email; a custom email parser (Lambda) triggers continuation.
B) Use AWS Step Functions with a waitForTaskToken callback integration, invoking an Amazon API Gateway REST API for the reviewer portal; include Timeout and Heartbeat settings on the review task state.
C) Use Amazon EventBridge to route flagged clauses to Amazon Simple Queue Service (SQS); a fleet of EC2 instances runs a web portal where attorneys review clauses.
D) Store all flagged clauses in Amazon S3; use Amazon Mechanical Turk to distribute clauses to external legal reviewers; use Amazon Lake Formation for audit logging.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: Step Functions is architecturally ideal for human-in-the-loop workflows. The waitForTaskToken integration pattern pauses workflow execution and waits for a callback from an external system (e.g., a reviewer portal exposed via API Gateway). The Timeout and Heartbeat settings ensure the workflow can handle reviewer unavailability gracefully, while Step Functions provides inherent state management and the execution history serves as an audit trail. This minimizes operational overhead via serverless architecture.

Distractor Analysis

A is Incorrect: An email-based review system with a custom parser introduces significant fragility and operational complexity due to email latency, response format inconsistencies, and spam/security concerns. Building a custom SNS→email→parser→continuation pipeline is architecturally inappropriate when Step Functions waitForTaskToken exists for this exact pattern.
C is Incorrect: Running a fleet of EC2 instances for a reviewer portal introduces heavy operational overhead (server management, patching, scaling, availability). The SQS + EC2 pattern requires constant server maintenance which contradicts the minimal overhead constraint.
D is Incorrect: Amazon Mechanical Turk is for crowd-sourced tasks with generalist workers, not for legally binding review requiring licensed attorneys. This approach cannot guarantee reviewer qualification and would not satisfy the legal compliance requirement. Lake Formation manages data permissions but does not inherently provide workflow audit trails.

Key Takeaway: Step Functions waitForTaskToken with Timeout and Heartbeat is the canonical serverless pattern for human-in-the-loop workflows that require audit trails and resilience to reviewer unavailability.

Question 5 - Multi-Dimensional Human Evaluation Framework

A media company is evaluating an Amazon Bedrock text generation model for article headline creation. The editorial team must rate headlines on three dimensions: accuracy (does it match the article?), engagement (would a reader click?), and style consistency (does it match brand voice?). They need to run weekly evaluation cycles, track reviewer agreement metrics (Cohen's Kappa), and monitor trend changes over time. The solution must provide structured data export for BI integration.

Which AWS services should the ML engineer combine? (Select TWO)

A) Amazon Bedrock Evaluations in human-review mode with rating instructions covering the three dimensions.
B) Amazon SageMaker Ground Truth with a private workforce and custom UI templates for multi-dimensional ratings.
C) Amazon CloudWatch dashboards for visualizing reviewer inter-rater agreement metrics.
D) Amazon Redshift with Amazon QuickSight for storing and analyzing inter-annotator agreement (Kappa scores) and for trend reporting via a BI dashboard.

Click to reveal Answer & Explanation

Correct Answers: B and D

Correct Options Analysis

B is Correct: Amazon SageMaker Ground Truth supports multi-dimensional custom labeling task interfaces and private workforces. You can configure custom UI templates for editors to rate headlines on accuracy, engagement, and style consistency, while Ground Truth manages reviewer assignment, task queueing, and output consolidation in structured JSON format.
D is Correct: Ground Truth outputs structured JSON data that can be loaded into Amazon Redshift. QuickSight provides the BI dashboard for trend visualization, enabling the team to view inter-annotator agreement metrics (Cohen's Kappa can be computed and stored), quality trends over time, and reviewer performance analytics. This satisfies the requirement for structured data export for BI integration.

Distractor Analysis

A is Incorrect: Amazon Bedrock Evaluations with human reviewers provides rating functionality, but its console interface is designed for simple evaluation datasets rather than complex multi-dimensional rating workflows with custom UI templates. It may not provide sufficient flexibility for custom rating schemas needed here, and the question asks specifically about Ground Truth's strength in custom multi-dimensional labeling tasks.
C is Incorrect: Amazon CloudWatch is for operational metrics like latency, error rates, and resource utilization—not for visualizing inter-rater agreement statistics (Cohen's Kappa) of human evaluations. Ground Truth provides built-in statistical analysis and Quality Insights, while redshift + QuickSight handles the BI reporting requirement.

Key Takeaway: For multi-dimensional human evaluation with BI reporting needs, Ground Truth custom UI templates + private workforce + Redshift/QuickSight analytics chain provides the flexibility and structured output required.

Question 6 - Hybrid Automated + Human Evaluation Design

An e-commerce company uses Amazon Bedrock to generate customer service response drafts. The customer experience team wants a hybrid evaluation pipeline where automated LLM-as-a-judge scoring handles all responses, but human escalation occurs only when the judge model flags responses above a risk threshold or score below a confidence threshold. The solution must be event-driven, serverless, and minimize human review burden.

Which AWS service chain should the ML engineer implement? (Select ONE)

A) Run Bedrock Evaluations with a judge model; route low-scoring responses to Amazon SNS to notify team leads for manual review via email.
B) Use AWS Step Functions with a Choice state after Bedrock judge model evaluation; low-confidence or high-risk responses proceed to an API Gateway reviewer portal via waitForTaskToken, while passing responses continue automatically.
C) Route all responses through a Bedrock Evaluation job; use Amazon EventBridge to trigger a Lambda that writes low-scoring responses to a DynamoDB table read by a custom review application.
D) Use Amazon SageMaker Processing Jobs to score responses and Amazon SQS queue for human reviewers with a fleet of EC2 worker instances.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: Step Functions with a Choice state enables conditional branching, where the judge model's output (a score and risk flag) gates whether a response proceeds automatically or enters human review. The waitForTaskToken integration provides the serverless mechanism for human escalation via the API Gateway reviewer portal. This minimizes human review burden while maintaining event-driven architecture.

Distractor Analysis

A is Incorrect: Amazon SNS email notifications are not a structured human review workflow mechanism. There is no state tracking, no timeout handling, no audit trail, and no way to programmatically continue the pipeline after review, whereas Step Functions provides all of these natively.
C is Incorrect: While writing to a DynamoDB table for a custom review application is architecturally plausible, it requires building and maintaining a separate review workflow application rather than using Step Functions' native state management, waitForTaskToken integration, and audit trail, violating the minimize-burden constraint.
D is Incorrect: Using SageMaker Processing Jobs and EC2 instances for the review portal introduces significant operational overhead. The question specifies a serverless requirement, and the EC2 fleet requires manual server management and scaling considerations.

Key Takeaway: Step Functions Choice states combined with waitForTaskToken create granular conditional human-review gates that keep humans out of already-confident model decisions, minimizing review burden.

Question 7 - RAG System Human Review Monitoring

A pharmaceutical company uses Amazon Bedrock Knowledge Bases for a RAG system that answers drug interaction questions from internal clinical trial documents. Patient safety is paramount. The team wants continuous monitoring of both retrieval quality and answer quality, with human review triggered when retrieval confidence is low or when the generated answer fails faithfulness checks. The system must generate weekly evaluation reports showing retrieval accuracy and answer safety trends.

Which combination of AWS services should the ML engineer implement? (Select TWO)

A) Use Amazon Bedrock Evaluations in programmatic mode (with metrics like ROUGE and BERTScore) against a golden dataset.
B) Use Amazon Bedrock Evaluations in retrieval-evaluation mode to measure context precision and context recall.
C) Use Amazon SageMaker Model Monitor to trigger Step Functions human-review workflows when evaluation metrics drop below thresholds.
D) Use Amazon CloudWatch custom metrics and alarms to detect when quality metrics cross thresholds, and trigger an Amazon EventBridge rule that invokes a Step Functions review workflow.

Click to reveal Answer & Explanation

Correct Answers: B and D

Correct Options Analysis

B is Correct: Amazon Bedrock Evaluations supports retrieval evaluation mode where the focus is on whether retrieval returns relevant context for the query. Conceptually, metrics like context precision (are retrieved documents relevant?) and context recall (did retrieval miss any relevant documents?) are central to assessing RAG pipeline retrieval accuracy. This addresses the requirement to monitor retrieval quality independently from answer quality.
D is Correct: CloudWatch custom metrics and alarms provide the operational monitoring layer. When retrieval confidence or faithfulness scores fall below thresholds, CloudWatch alarms trigger EventBridge rules, which invoke Step Functions to orchestrate the human review workflow. This creates a closed-loop monitoring system that is serverless and integrated with the evaluation outputs.

Distractor Analysis

A is Incorrect: Programmatic evaluation with ROUGE and BERTScore evaluates generation similarity against reference answers but does not distinguish between retrieval quality and generation quality in a RAG system, and would not meaningfully assess whether the retrieved context was correct or complete.
C is Incorrect: Amazon SageMaker Model Monitor is for monitoring traditional machine learning models deployed on SageMaker endpoints for data drift, model drift, and bias, not for orchestrating GenAI evaluation pipelines on Bedrock Knowledge Bases RAG systems. It cannot directly monitor Bedrock evaluation metrics or trigger Bedrock-based human review workflows.

Key Takeaway: Evaluate RAG retrieval (context precision/recall) and generation (faithfulness) separately, and use CloudWatch + EventBridge + Step Functions for threshold-based human review escalation.

Question 8 - Inter-Rater Agreement Analysis

A governance team at a multinational bank uses Amazon SageMaker Ground Truth with a private workforce of compliance analysts to evaluate generated financial advise from Amazon Bedrock. Each output is rated by three analysts for regulatory compliance on a 1-5 scale. Leadership wants to know if analysts are consistently agreeing with each other before trusting the evaluation data. The team needs automated computation of statistical agreement metrics without building custom infrastructure.

Which approach should the ML engineer use? (Select ONE)

A) Download Ground Truth output manifests and compute Cohen's Kappa or Fleiss' Kappa using a custom Python script on an EC2 instance.
B) Use Amazon SageMaker Ground Truth's built-in worker quality insights, which provide inter-annotator agreement metrics and worker performance statistics.
C) Use Amazon QuickSight with ground truth label data from S3 to configure interactive-based agreement visualizations using calculated fields.
D) Use Amazon Bedrock Evaluations human-review mode and read agreement statistics from the evaluation job console output.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: Amazon SageMaker Ground Truth automatically computes worker quality metrics, including inter-annotator agreement (using metrics conceptually similar to Cohen's Kappa and Fleiss' Kappa) and worker consistency over time. These metrics are directly visible in the Ground Truth console and via API without requiring custom computation or infrastructure. This directly satisfies the requirement for automated computation of statistical agreement metrics.

Distractor Analysis

A is Incorrect: While technically possible, running a custom Python script on an EC2 instance introduces unnecessary operational overhead including server provisioning, instance lifecycle management, and security patching. This violates the constraint of avoiding custom infrastructure and would not provide automated, scheduled computation.
C is Incorrect: QuickSight is for visualization of data and would require first computing the Kappa statistics externally and loading them into a QuickSight data source. It cannot calculate Fleiss' Kappa or Cohen's Kappa without manually configuring complex calculated fields, which still requires understanding and re-implementing the formula, assuming QuickSight even supports the mathematical operations needed.
D is Incorrect: Amazon Bedrock Evaluations in human-review mode provides basic rating input and basic anonymized worker statistics, but is not designed to compute complex inter-annotator agreement metrics like Fleiss' Kappa for multi-reviewer scenario, nor does it offer the deep worker quality analytics available in Ground Truth.

Key Takeaway: Ground Truth provides built-in inter-annotator agreement (IAA) metrics for assessing human reviewer consistency without custom statistical computation.

Question 9 - Human Review Audit Trail Design

An insurance company uses Amazon Bedrock with AWS Step Functions to automate claims eligibility decisions. State regulations require that any claim exceeding $100,000 be reviewed and approved by a licensed claims adjuster, with a complete, tamper-proof audit trail including reviewer identity, exact action taken, timestamp, and any comments. The review process must not block other claims in the queue while an adjuster reviews a high-value claim.

Which configuration should the ML engineer implement for the human review? (Select ONE)

A) Configure a Step Functions state with waitForTaskToken and a Timeout of 30 days; log all task tokens and review decisions in Amazon CloudTrail; use Amazon DynamoDB for reviewer portal data with point-in-time recovery.
B) Configure a Step Functions state with waitForTaskToken integration and a Heartbeat setting; use Amazon DynamoDB streams to capture review events with reviewer identity in an immutable Amazon S3 bucket with object lock and versioning; use a Parallel state so multiple high-value claims can be reviewed simultaneously.
C) Use an SQS FIFO queue for review tasks with a dead-letter queue; store review decisions in Amazon RDS with Multi-AZ deployment and automatic backups.
D) Configure an Amazon SNS topic with email subscriptions for adjusters; use Amazon CloudTrail for API call logging.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: This configuration addresses all requirements. The waitForTaskToken pattern manages the reviewer interaction, while the Heartbeat setting ensures the task times out if the reviewer disconnects or abandons the review. DynamoDB Streams capture the review event data (reviewer identity, action, timestamp, comments), which is then written to an immutable S3 bucket with versioning and Object Lock (WORM capability) for tamper-proof audit logging. The Parallel state ensures multiple claims can be reviewed independently without blocking each other.

Distractor Analysis

A is Incorrect: A 30-day timeout is too long and the Heartbeat setting is missing (so a reviewer who starts but never completes a review would stall the workflow). CloudTrail logs AWS API calls but does not capture the business-level review details (reviewer identity within the custom application, comments, approval/rejection action). The DynamoDB point-in-time recovery provides recovery capability, but lacks the immutable S3 Object Lock requirement for tamper-proof audit trails.
C is Incorrect: While SQS queues can manage review tasks, a FIFO queue implementation fails to include the Step Functions state management and audit trail requirements. RDS with backups is not an immutable audit log, as data can be modified or deleted by database administrators. The dead-letter queue handles failed reviews but does not capture reviewer identity or actions.
D is Incorrect: SNS email notifications are not a structured review workflow; they lack state management, Heartbeat/timeout handling, and a structured mechanism for capturing reviewer identity and decisions. CloudTrail only logs AWS API calls, not business-level review decisions made via email responses, so it would not capture the required reviewer identity or actions.

Key Takeaway: For tamper-proof human review audit trails, combine Step Functions waitForTaskToken + Heartbeat, DynamoDB Streams for event capture, and S3 Object Lock with versioning for immutable storage.

Question 10 - LLM-as-a-Judge vs Human Evaluation Trade-offs

A data science team at a legal research firm is evaluating two approaches for assessing the quality of case law summaries generated by Amazon Bedrock. The team has a limited budget but high stakes; they must decide between using an LLM-as-a-judge approach exclusively, or a hybrid approach combining LLM-as-a-judge with periodic human evaluation. The summaries must be assessed for legal accuracy, proper citation formatting, and neutral tone. The evaluation must run weekly against 5,000 summaries.

What should the ML engineer recommend as the most cost-effective approach that maintains evaluation quality? (Select ONE)

A) Use LLM-as-a-judge exclusively for all 5,000 summaries, with structured scoring rubrics and Socratic grading; calibrate the judge model against a small golden dataset annotated by senior attorneys before full deployment.
B) Use human reviewers exclusively for all 5,000 summaries through Amazon SageMaker Ground Truth with a private workforce; this provides the highest accuracy for legal content.
C) Use Amazon Bedrock Evaluations in human-review mode for all 5,000 summaries per week; this ensures human expertise is embedded into the evaluation process.
D) Use MLflow to track model performance and implement the evaluation using ROUGE and BLEU scores computed programmatically on an EC2 instance; human review bypasses the need for Bedrock Evaluations.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

A is Correct: Using LLM-as-a-judge exclusively is the most cost-effective scalable approach for 5,000 weekly summaries. By calibrating the judge model against a small golden dataset annotated by senior attorneys, the team ensures the judge's scoring aligns with expert human judgment (calibration). This provides the quality needed while avoiding the cost and latency of manual human review for every summary. The structured rubrics and Socratic grading provide explainability that supports legal accuracy assessment.

Distractor Analysis

B is Incorrect: Human review of 5,000 summaries per week with a specialized private workforce of legal experts would be prohibitively expensive. Note that Ground Truth private workforces require professional availability, detailed instruction, and ongoing management. This does not satisfy the cost-effective constraint and would introduce significant operational overhead.
C is Incorrect: Amazon Bedrock Evaluations in human-review mode for all 5,000 summaries per week would also be cost-prohibitive and time-consuming, given that legal professionals are expensive and have limited availability. Human review is valuable but should be used sparingly where judgment is genuinely required.
D is Incorrect: MLflow is for experiment tracking, not for text generation evaluation. ROUGE and BLEU are n-gram overlap metrics that cannot assess legal accuracy, citation formatting, or neutral tone. Running scripts on an EC2 instance introduces unnecessary operational overhead and does not provide rubric-based or explainable evaluation.

Key Takeaway: Calibrate LLM-as-a-judge evaluation against small human-annotated golden datasets to achieve scalable, cost-effective quality assessment, reserving human review for edge cases and periodic audits.

Performance Metric	Traditional ML	GenAI Text Output	Judgment Approach
Correctness	Accuracy, precision, recall vs. labels	Factual accuracy vs. source docs	LLM-as-judge rubric scoring vs. references
Quality	F1, AUC	Readability, tone, engagement, style	Human ratings on multi-dimensional scales
Grounding	Not applicable	Answer grounded in retrieved context?	Retrieval evaluation + Socratic grading
Drift	Model Monitor baseline drift	Prompt sensitivity and semantic shift	Repeated judge scoring trends over time
Calibration	Probabilistic calibration (ECE)	Judge vs. human expert agreement (Kappa)	Inter-annotator agreement analysis