Skip to content

Skill 2.3.9: Perform AI evaluation (for example, model output assessment, content quality validation, bias detection, LLM-as-a-judge frameworks)

Question 1 - LLM-as-a-Judge Framework Selection

A financial services company has deployed a generative AI application using Amazon Bedrock that generates investment summaries from quarterly earnings reports. The compliance team requires automated evaluation of every generated summary before it reaches customers. The evaluation must assess factual accuracy against source documents, detect potential regulatory compliance violations, and provide an explanation for each score to support audit trails. The team wants to minimize operational overhead and avoid maintaining custom evaluation infrastructure.

Which approach should the ML engineer use? (Select ONE)

  • A) Configure an Amazon Bedrock evaluation job using an LLM-as-a-judge model with custom prompt datasets containing evaluation criteria for factual accuracy and compliance.
  • B) Deploy a separate SageMaker endpoint running an open-source evaluation model, and write a custom orchestration workflow using AWS Step Functions to score each summary.
  • C) Use Amazon CloudWatch Logs with metric filters to detect keywords associated with compliance violations in generated text.
  • D) Implement a rule-based validation system using AWS Lambda functions and regular expressions to validate factual claims against the source documents.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Amazon Bedrock evaluation jobs support LLM-as-a-judge evaluations where a judge model scores responses against custom criteria (factual accuracy, compliance) and provides explanations/justifications for each score. This is a fully managed, zero-infrastructure approach that satisfies the compliance audit trail requirement.

Distractor Analysis

  • B is Incorrect: Deploying a custom SageMaker endpoint and orchestrating evaluation with Step Functions introduces significant operational overhead—the opposite of the "minimize operational overhead" requirement. Bedrock's managed evaluation service already provides this capability.
  • C is Incorrect: CloudWatch keyword-matching detects specific strings but cannot assess factual accuracy or provide meaningful explanations. It would miss semantic errors, paraphrased violations, and nuanced compliance issues, failing the audit trail requirement.
  • D is Incorrect: Rule-based regex systems cannot evaluate the semantic meaning of generated text against source documents. They operate on pattern matching only and cannot provide the deep understanding needed for factual accuracy assessment or compliance detection.

Key Takeaway: Amazon Bedrock evaluation jobs with LLM-as-a-judge provide managed, explainable AI evaluation without requiring custom infrastructure.

Question 2 - Human-in-the-Loop Evaluation Workflow

A healthcare company is building a medical triage assistant using Amazon SageMaker AI and Amazon Bedrock that provides preliminary health assessments to patients. Due to regulatory requirements, every AI-generated response must be reviewed by a licensed medical professional before being shown to patients. The review process must capture ratings on response quality, safety, and accuracy, and these human ratings must be stored as structured data for ongoing model improvement.

Which architecture best satisfies these requirements? (Select ONE)

  • A) Configure an Amazon Bedrock evaluation job with human-based evaluation, where medical professionals rate responses through the Bedrock console interface, and export the ratings for model fine-tuning.
  • B) Build a SageMaker Ground Truth labeling job that presents AI responses to medical professionals for review, captures structured ratings and feedback, and produces an output manifest.
  • C) Create an Amazon Augmented AI (A2I) human review workflow integrated with the Bedrock pipeline, capturing reviewer decisions and ratings as structured metadata.
  • D) Use AWS Step Functions with a manual approval step where reviewers receive email notifications, manually rate responses in a spreadsheet, and upload results to Amazon S3.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

  • C is Correct: Amazon Augmented AI (A2I) is specifically designed for human-in-the-loop review workflows within AI pipelines. It routes AI outputs to human reviewers when conditions are met, captures structured review decisions and metadata, and integrates natively with SageMaker and other AWS AI services. This satisfies the mandatory review requirement while maintaining structured data capture.

Distractor Analysis

  • A is Incorrect: While Bedrock evaluation jobs support human-based evaluation, these are batch evaluation workflows for model assessment, not real-time production review pipelines. They're designed for periodic evaluation, not gating every production response.
  • B is Incorrect: SageMaker Ground Truth is designed for creating labeled datasets for model training, not for production inference review pipelines. While it captures structured ratings, it's not the right tool for real-time response gating.
  • D is Incorrect: Manual email workflows with spreadsheets introduce high operational overhead, lack real-time integration with the AI pipeline, and are error-prone. Structured data capture in spreadsheets is not a robust enterprise solution.

Key Takeaway: Use Amazon A2I for production human review pipelines; use Bedrock human evaluations for periodic model assessment.

Question 3 - Bias Detection in Model Outputs

A talent acquisition platform uses an LLM through Amazon Bedrock to generate job descriptions and screen candidate resumes. The company has discovered that the model occasionally produces gendered language in job descriptions (e.g., "he will manage," "nurturing personality for admin roles") that violates their equal opportunity policies. They need a systematic evaluation approach to detect and quantify bias in model outputs across thousands of generated job descriptions.

Which TWO approaches should the ML engineer implement? (Select TWO)

  • A) Configure an Amazon Bedrock evaluation job with a judge model that uses custom evaluation criteria specifically designed to detect gendered language and other bias indicators.
  • B) Use Amazon Comprehend with custom classification to identify personal attributes in generated text and flag bias patterns.
  • C) Create a SageMaker Clarify job to analyze the generated text for bias metrics based on protected attributes like gender.
  • D) Implement Amazon CloudWatch dashboards tracking token usage patterns to identify bias correlations.
  • E) Write custom Python scripts using AWS Lambda to search for specific gendered words and log matches to Amazon S3.
Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: Amazon Bedrock evaluation jobs with LLM-as-a-judge can assess outputs against custom bias detection criteria. The judge model can understand contextual bias beyond simple keyword matching and provide explanations for flagged content.
  • C is Correct: Amazon SageMaker Clarify provides bias detection metrics specifically designed to evaluate model predictions for bias related to protected attributes. It can be configured to analyze text outputs and quantify bias patterns.

Distractor Analysis

  • B is Incorrect: Amazon Comprehend custom classification is designed for document classification tasks, not bias detection in generated text. While it can identify entities, it doesn't provide bias metrics or evaluation frameworks.
  • D is Incorrect: Token usage patterns in CloudWatch do not correlate with content bias. Tracking tokens measures resource consumption, not the semantic content or fairness of generated text.
  • E is Incorrect: Simple keyword searching via Lambda cannot detect contextual or semantic bias. Gendered language can appear in forms that don't match simple keyword patterns, and this approach lacks any scoring or quantification mechanism.

Key Takeaway: Use SageMaker Clarify for traditional ML bias metrics and Bedrock evaluation jobs with LLM-as-a-judge for contextual GenAI bias assessment.

Question 4 - Content Quality Validation

A media company generates news article summaries using Amazon Bedrock and has received complaints that summaries sometimes contain inappropriate tone for professional news content. The editorial team defines quality requirements: summaries must be neutral, factual, and appropriate for a general audience. The engineering team needs to automatically evaluate every summary against these requirements and flag those that need human editorial review.

Which solution provides the most cost-effective automated quality validation? (Select ONE)

  • A) Deploy multiple dedicated evaluation models, one for each quality dimension, using SageMaker endpoints and orchestrate evaluation with AWS Step Functions.
  • B) Configure an Amazon Bedrock evaluation job with a single judge model that evaluates summaries against all three criteria (neutrality, factuality, audience appropriateness) and returns a quality score.
  • C) Use Amazon OpenSearch to store summaries and create custom queries that flag inappropriate content based on keyword patterns.
  • D) Implement a sentence embedding similarity comparison against a curated set of "good" examples using SageMaker processing jobs.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: A single LLM-as-a-judge model through Bedrock evaluation jobs can evaluate multiple quality dimensions simultaneously using carefully crafted evaluation criteria. This is the most cost-effective approach because it requires no dedicated infrastructure and one judge model handles all three quality requirements.

Distractor Analysis

  • A is Incorrect: Deploying multiple dedicated evaluation models for each quality dimension introduces unnecessary operational overhead and cost. A single judge model can handle multiple criteria when given proper evaluation instructions. This violates the cost-effectiveness requirement.
  • C is Incorrect: OpenSearch keyword queries cannot evaluate semantic qualities like "neutrality" or "audience appropriateness." This approach would miss subtle tone issues and generate false positives for legitimate content using flagged keywords.
  • D is Incorrect: Embedding similarity to a curated set of examples is a heuristic approach that doesn't directly evaluate the specified quality criteria. It may flag unusual but appropriate content, and misses nuanced quality issues. It's a workaround, not a proper quality evaluation solution.

Key Takeaway: One well-crafted LLM-as-a-judge can evaluate multiple quality dimensions simultaneously, making it more cost-effective than purpose-built evaluation systems.

Question 5 - RAG System Evaluation with Bias Detection

A legal research platform uses Amazon Bedrock Knowledge Bases for retrieval and generation of legal document summaries. The system must be evaluated for two separate concerns: (1) whether the retrieval system returns relevant legal precedents without bias toward certain jurisdictions, and (2) whether the generated summaries accurately represent the retrieved documents without introducing bias. The evaluation must provide separate metrics for retrieval quality and generation quality.

Which evaluation strategy best addresses these requirements? (Select ONE)

  • A) Run a single Amazon Bedrock evaluation job that scores the final answers against ground truth and treats the pipeline as a unified system.
  • B) Configure separate Amazon Bedrock evaluation jobs: one with retrieval metrics (context precision, context recall) and another with generation metrics (faithfulness, bias detection criteria).
  • C) Use Amazon SageMaker Clarify exclusively to evaluate both the retrieval and generation components as a single bias detection task.
  • D) Monitor Amazon CloudWatch metrics for Amazon OpenSearch Serverless query latency and Bedrock token usage as proxies for retrieval and generation quality.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: Splitting evaluation into separate jobs for retrieval and generation aligns with the RAG triad evaluation methodology. Retrieval evaluation (context precision/recall) measures whether relevant, unbiased precedents are returned. Generation evaluation (faithfulness, bias criteria) measures whether the summary accurately represents retrieved content without introducing bias.

Distractor Analysis

  • A is Incorrect: Treating retrieval and generation as a black box prevents identifying whether quality issues originate from poor retrieval or poor generation. A fluent answer that omits facts from retrieved documents is a retrieval failure, not a generation failure, and this approach would mask that distinction.
  • C is Incorrect: SageMaker Clarify is designed for bias detection in ML models using training data and predictions, not for evaluating retrieval system relevance or RAG generation faithfulness. It cannot measure retrieval accuracy metrics like context precision.
  • D is Incorrect: Query latency and token usage are operational metrics that measure system performance, not content quality or bias. Low latency does not indicate relevant retrieval, and high token usage does not indicate faithful generation.

Key Takeaway: Always evaluate retrieval and generation components separately in RAG systems to pinpoint quality issues and detect bias at each stage.

Question 6 - LLM-as-a-Judge Framework Selection

An e-commerce company wants to evaluate product descriptions generated by their AI system for three specific quality dimensions: helpfulness, accuracy, and brand voice consistency. The team wants to build an evaluation framework that can score thousands of descriptions daily, provide explanations for each score, and allow human reviewers to override automated scores. They need minimal infrastructure management and the ability to analyze evaluation patterns over time.

Which combination of AWS services best meets these requirements? (Select ONE)

  • A) Deploy an open-source evaluation framework on Amazon EKS with custom scoring models, store results in Amazon RDS, and build dashboards in Amazon QuickSight.
  • B) Use Amazon Bedrock evaluation jobs with LLM-as-a-judge for automated scoring, integrate Amazon A2I for human review workflows, and store all scores in Amazon S3 with Amazon Athena for pattern analysis.
  • C) Implement scoring rules in AWS Lambda functions, store scores in Amazon DynamoDB, and create dashboards in Amazon CloudWatch.
  • D) Use Amazon SageMaker Processing jobs with custom evaluation scripts, SageMaker Ground Truth for human review, and Amazon SageMaker Model Monitor for tracking score patterns.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: This solution uses managed services that directly serve each requirement: Bedrock evaluation jobs provide automated LLM-as-a-judge scoring with explanations; Amazon A2I integrates human review/override workflows; S3 with Athena enables scalable storage and analysis of evaluation patterns over time. This minimizes infrastructure management while providing a complete evaluation framework.

Distractor Analysis

  • A is Incorrect: Running evaluation frameworks on EKS requires managing Kubernetes clusters, custom model deployment, and RDS infrastructure. This violates the minimal infrastructure management requirement. While technically feasible, it's operationally heavy compared to managed Bedrock evaluations.
  • C is Incorrect: Lambda functions with rule-based scoring cannot evaluate qualitative dimensions like "helpfulness" or "brand voice consistency." These require semantic understanding that only an LLM-as-a-judge can provide. CloudWatch dashboards are also not designed for deep pattern analysis of evaluation scores.
  • D is Incorrect: SageMaker Processing jobs with custom scripts reintroduce infrastructure management for evaluation. Ground Truth is designed for dataset labeling, not production human review workflows. Model Monitor is for production model drift, not evaluation pattern analysis.

Key Takeaway: Combine Bedrock evaluation jobs (automated scoring) with Amazon A2I (human review) for a complete, managed evaluation framework that supports both automated and human judgment.

Question 7 - Bias Detection in LLM Responses

A financial services company uses an LLM through Amazon Bedrock to provide investment advice to customers in multiple countries. The system has been flagged for potentially exhibiting socioeconomic bias: responses appear to offer more conservative advice to customers from lower-income postal codes. The compliance team requires automated bias detection that can quantify this pattern and generate a bias report for regulatory review.

Which approach best satisfies these requirements? (Select ONE)

  • A) Create an Amazon Bedrock evaluation job with a judge model that evaluates responses for socioeconomic bias patterns and generates structured bias scores across demographic categories.
  • B) Use Amazon SageMaker Clarify with a custom dataset containing customer demographic attributes and model predictions to generate bias metrics and a detailed bias report.
  • C) Configure Amazon CloudWatch alerts to trigger when response patterns vary significantly across customer regions, using custom metrics for advice conservatism.
  • D) Implement Amazon QuickSight dashboards that visualize customer feedback scores across demographic segments and manually identify bias patterns.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: SageMaker Clarify is specifically designed for bias detection in ML systems. It can analyze model predictions against protected attributes (like socioeconomic indicators derived from postal codes) and generate detailed bias metrics and reports suitable for regulatory review. This is a quantifiable, systematic approach to bias detection.

Distractor Analysis

  • A is Incorrect: While Bedrock evaluation jobs with LLM-as-a-judge can detect qualitative bias, they're less suited for quantifying systematic bias patterns across large datasets with protected attributes. SageMaker Clarify provides more rigorous bias metrics specifically designed for regulatory compliance.
  • C is Incorrect: CloudWatch custom metrics for "advice conservatism" would require building an operational metric system that doesn't exist. CloudWatch is for monitoring system performance, not detecting nuanced socioeconomic bias in generated advice. This approach also lacks the structured reporting needed for regulatory review.
  • D is Incorrect: Manual dashboards with customer feedback are reactive rather than proactive. This approach detects bias only after customer complaints rather than systematically evaluating all responses. It also lacks the structured bias metrics required for regulatory compliance.

Key Takeaway: Use SageMaker Clarify for quantitative bias detection against protected attributes; use Bedrock LQA evaluations for qualitative bias assessment in natural language outputs.

Question 8 - Multi-Dimensional Content Quality Validation

A healthcare chatbot powered by Amazon Bedrock provides medical information to patients. The quality assurance team defines a comprehensive evaluation framework with four dimensions: medical accuracy, safety (does not recommend harmful actions), empathy, and readability. They require daily automated evaluation of all chatbot responses and need a system that can provide separate scores for each dimension with explanations.

Which approach is most appropriate for this multi-dimensional evaluation requirement? (Select ONE)

  • A) Configure four separate Amazon Bedrock evaluation jobs, each with a judge model and custom evaluation criteria for one quality dimension.
  • B) Configure a single Amazon Bedrock evaluation job with a judge model that evaluates all four dimensions simultaneously and returns structured scores with explanations for each dimension.
  • C) Deploy four dedicated models on SageMaker endpoints, one for each quality dimension, and orchestrate evaluation using AWS Step Functions.
  • D) Use Amazon Comprehend for sentiment analysis (empathy), Amazon Transcribe for readability analysis, and custom Lambda functions for safety checking.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: A single LLM-as-a-judge can evaluate multiple quality dimensions when given comprehensive evaluation criteria. Bedrock evaluation jobs support structured output formats that can include separate scores for multiple dimensions with explanations. This minimizes operational overhead while providing the granular, multi-dimensional evaluation required.

Distractor Analysis

  • A is Incorrect: Running four separate evaluation jobs doubles or quadruples the cost and complexity without benefit. A judge model can handle four dimensions in one evaluation pass. Multiple jobs also introduce consistency issues across dimension evaluations.
  • C is Incorrect: Deploying four dedicated models for evaluation is architecturally redundant. A single LLM-as-a-judge is capable of evaluating multiple quality dimensions. This approach maximizes operational overhead and cost without improving evaluation quality.
  • D is Incorrect: Amazon Transcribe is for speech-to-text, not readability analysis. Comprehend sentiment analysis captures only one narrow aspect of empathy. This fragmented approach lacks the holistic understanding that an LLM-as-a-judge provides.

Key Takeaway: Leverage a single LLM-as-a-judge for multi-dimensional evaluation—it can assess multiple criteria simultaneously when given well-crafted evaluation criteria.

Question 9 - Human Evaluation Framework Integration

A legal tech company has implemented an AI contract analysis system using Amazon Bedrock. The system generates contract risk assessments that attorneys must review before delivery to clients. The company needs to collect structured feedback from attorneys on assessment quality, track reviewer agreement rates to measure system reliability, and use this feedback to trigger model improvements when agreement rates fall below thresholds.

Which architecture best meets these requirements? (Select ONE)

  • A) Create a SageMaker Ground Truth labeling job that continuously routes contract assessments to attorneys for review and automatically retrains the model when labeling is complete.
  • B) Implement an Amazon A2I human review workflow that captures structured attorney feedback and ratings, with Amazon CloudWatch metrics tracking reviewer agreement rates that trigger SageMaker Model Monitor actions when thresholds are violated.
  • C) Configure an Amazon Bedrock evaluation job with human-based evaluation where attorneys rate assessments through the Bedrock console, and export results to drive model updates.
  • D) Build a custom web application using AWS Amplify where attorneys review assessments, store feedback in Amazon DynamoDB, and use AWS Step Functions to trigger model retraining when disagreements exceed acceptable levels.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: Amazon A2I provides structured human review workflows for production AI systems, capturing reviewer decisions as standardized metadata. CloudWatch can track agreement rate metrics, and SageMaker Model Monitor can trigger actions (like retraining requests) when agreement rates fall below thresholds. This is a complete, managed architecture for continuous quality monitoring with human feedback.

Distractor Analysis

  • A is Incorrect: SageMaker Ground Truth is designed for offline dataset labeling for model training, not production review workflows. It doesn't provide the real-time routing and structured feedback capture needed for production contract assessment review.
  • C is Incorrect: Bedrock human evaluations are batch assessment workflows for periodic model evaluation, not production review pipelines. Attorneys reviewing through the Bedrock console is not a scalable production review interface.
  • D is Incorrect: While technically functional, building a custom web application with Amplify introduces unnecessary development overhead. Amazon A2I provides the required human review workflow without custom application development.

Key Takeaway: Use Amazon A2I for production human review workflows that require structured feedback capture and integration with monitoring systems.

Question 10 - Automated Content Quality Validation at Scale

An enterprise knowledge management system generates document summaries using Amazon Bedrock for internal use. The summaries must be evaluated for factual consistency with source documents, appropriate classification level (public, internal, confidential), and absence of sensitive information leakage. With millions of summaries generated monthly, the team needs the most cost-effective automated evaluation approach that can scale without dedicated evaluation infrastructure.

Which approach provides the most cost-effective scaling solution? (Select ONE)

  • A) Deploy an open-source evaluation model cluster on Amazon EC2 Spot Instances with autoscaling to handle evaluation demand spikes.
  • B) Configure an Amazon Bedrock evaluation job with a judge model that evaluates summaries against all three criteria (factual consistency, classification accuracy, sensitive information detection) and returns structured quality scores.
  • C) Use Amazon Macie to detect sensitive information in all generated summaries and Amazon Comprehend for document classification, with custom Lambda functions for factual consistency checking.
  • D) Implement sampling-based evaluation where only 1% of summaries are evaluated using manual review processes to minimize evaluation costs.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: Bedrock evaluation jobs provide managed, automated evaluation with LLM-as-a-judge that can assess all three quality dimensions simultaneously. The serverless nature of Bedrock means no infrastructure management and automatic scaling. A single judge model evaluating millions of summaries through batch evaluation jobs is the most cost-effective approach.

Distractor Analysis

  • A is Incorrect: Running open-source evaluation models on EC2 Spot requires extensive infrastructure management, model deployment, and operational overhead. While Spot Instances reduce cost, they add complexity and reliability challenges. This is significantly more operationally expensive than managed Bedrock evaluations.
  • C is Incorrect: Amazon Macie is designed for data discovery and classification in S3 buckets, not for evaluating content in generated summaries. Amazon Comprehend handles classification but cannot assess factual consistency. The fragmented approach with custom Lambda functions for factual checking introduces development overhead and operational complexity.
  • D is Incorrect: Sampling only 1% of summaries leaves 99% unevaluated, creating significant risk of quality issues and sensitive information leakage. This approach fails the requirement for comprehensive evaluation and potentially violates internal compliance requirements.

Key Takeaway: Bedrock evaluation jobs with LLM-as-a-judge provide the most cost-effective, scalable solution for automated content quality validation without requiring evaluation infrastructure management.