Skill 2.3.9: Perform AI evaluation (for example, model output assessment, content quality validation, bias detection, LLM-as-a-judge frameworks)
| Question | Answer & Explanation |
|---|---|
| ### Question 1 - Automated Content Quality Validation via LLM-as-a-Judge A media company is building a generative AI application using Amazon Bedrock to summarize daily news articles. The engineering team needs to evaluate the summarization quality of multiple Foundation Models (FMs) across a large dataset of 5,000 articles. The evaluation must be highly scalable, automated, and must provide a rationale for each assigned score to ensure transparency. Which solution meets these requirements with the LEAST operational overhead? - A) Deploy a custom AWS Lambda function that calculates the ROUGE-1 score for each generated summary against a reference summary and stores the results in Amazon S3. - B) Use Amazon SageMaker Shadow Testing to deploy a candidate FM alongside a production FM and route live traffic to compare the summaries. - C) Create an Amazon Bedrock Evaluation job using the LLM-as-a-judge framework, selecting a capable model to score the summaries and provide reasoning for the scores. - D) Build a custom evaluation pipeline using Amazon SageMaker Processing jobs that runs a Hugging Face BERTScore implementation on the 5,000 articles. |
Correct Answer: C Correct Answer Analysis: - C is Correct: Amazon Bedrock Evaluations natively supports the LLM-as-a-judge framework, allowing an automated model to score the outputs of other models across large datasets. It provides built-in rationale generation, requires no custom infrastructure, and scales natively with minimal operational overhead. Distractor Analysis: - A is Incorrect: While ROUGE-1 is a programmatic metric, building a custom Lambda function introduces unnecessary operational overhead and does not provide a qualitative rationale for the scores, missing the transparency requirement. - B is Incorrect: SageMaker Shadow Testing is used to evaluate models on live inference traffic, not for offline dataset evaluation of summarization quality. - D is Incorrect: Building a custom SageMaker Processing job with BERTScore introduces infrastructure management overhead compared to the native Bedrock Evaluations feature. > Key Takeaway: Use Amazon Bedrock Evaluations with LLM-as-a-judge for scalable, automated, and rationale-backed content quality assessment of GenAI outputs. |
| ### Question 2 - Automated Bias Detection in GenAI Outputs An AI team is deploying a Large Language Model (FM) via Amazon Bedrock to generate candidate job descriptions. The compliance department mandates that the generated text must be evaluated for demographic bias before deployment. The team requires an automated solution that specifically targets bias in generative text without writing complex custom classifiers. Which approach should the team use to evaluate the model's outputs for bias? - A) Use Amazon SageMaker Clarify to calculate pre-training bias metrics on the text generation training dataset. - B) Configure an Amazon Bedrock Evaluation job utilizing the built-in bias metric to evaluate the model's generated responses. - C) Deploy an Amazon Comprehend custom classification model to flag biased terminology in the output. - D) Use AWS Lambda to run regular expression matching against a dictionary of prohibited words. Select ONE that applies. |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock Evaluations includes built-in metrics specifically designed for GenAI, including a bias metric. This allows automated evaluation of an FM's output for demographic bias without requiring custom models or complex data science pipelines. Distractor Analysis: - A is Incorrect: SageMaker Clarify is primarily used for traditional ML bias detection (pre-training and post-training bias on tabular data), not for evaluating the generative text outputs of Foundation Models. - C is Incorrect: While Amazon Comprehend can analyze text, it requires training a custom classifier and does not natively evaluate GenAI bias as a standardized metric out-of-the-box like Bedrock Evaluations. - D is Incorrect: Regex matching is too rigid, cannot understand contextual bias in natural language, and requires manual maintenance of prohibited word lists. > Key Takeaway: For GenAI output bias detection, leverage the native built-in bias metrics in Amazon Bedrock Evaluations rather than traditional ML bias tools. |
| ### Question 3 - Isolating RAG Generation Failures A Retrieval-Augmented Generation (RAG) application built on Amazon Bedrock Knowledge Bases is generating highly fluent answers, but users report that the answers frequently omit critical facts that are verifiably present in the retrieved source documents. The engineering team must perform an AI evaluation to isolate whether the issue is a retrieval failure or a generation failure. Which TWO evaluation metrics should the team use to determine if the LLM is accurately utilizing the retrieved context? (Select TWO) - A) Context Precision - B) Context Recall - C) Context Utilization (Faithfulness) - D) Answer Relevance - E) Semantic Similarity |
Correct Answers: C and D Correct Answer Analysis: - C is Correct: Context Utilization (often measured as Faithfulness) assesses whether the generated answer contains only facts that are supported by the retrieved context, ensuring the model isn't hallucinating or dropping retrieved facts. - D is Correct: Answer Relevance evaluates whether the generated response actually addresses the user's prompt, isolating the generation step's effectiveness. Distractor Analysis: - A is Incorrect: Context Precision measures whether the retrieval step returned relevant chunks, not whether the generation step utilized them. - B is Incorrect: Context Recall evaluates if the retrieval step successfully fetched all necessary ground-truth facts, a retrieval metric, not a generation metric. - E is Incorrect: Semantic Similarity generally compares the generated answer to a reference answer, but does not specifically isolate whether the LLM utilized the retrieved context. > Key Takeaway: To isolate generation failures in RAG, evaluate Context Utilization (Faithfulness) and Answer Relevance, separate from retrieval metrics like Context Precision and Recall. |
| ### Question 4 - Integrated Human Evaluation Frameworks A legal technology startup is fine-tuning an LLM to draft legal briefs based on case files. Because of the high stakes and regulatory compliance requirements, the output must be evaluated by in-house legal experts. The team needs an AWS-native service that can present prompts and generated responses to their internal workforce, collect categorical ratings (e.g., "Accurate", "Inaccurate", "Needs Revision"), and compile the results. Which AWS service provides this integrated human evaluation framework with the LEAST operational overhead? - A) Amazon SageMaker Ground Truth - B) Amazon Bedrock Evaluations using human workers - C) Amazon Mechanical Turk via an AWS Lambda orchestration layer - D) Amazon Augmented AI (A2I) with a custom worker task template |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock Evaluations natively supports human evaluation workflows. It allows teams to bring their own workforce (internal employees) to rate model outputs through a managed UI, directly integrating with the model evaluation lifecycle. Distractor Analysis: - A is Incorrect: SageMaker Ground Truth is primarily designed for labeling data (images, text bounding boxes) for training ML models, not for evaluating the generative outputs of Foundation Models. - C is Incorrect: Mechanical Turk provides a public crowdsource workforce, which is inappropriate for confidential legal case files, and building a custom Lambda orchestration layer adds unnecessary overhead. - D is Incorrect: While A2I can be used for human review, it requires building custom worker task templates and workflow definitions, making it higher overhead than the native human evaluation capabilities in Bedrock Evaluations. > Key Takeaway: Amazon Bedrock Evaluations supports human-in-the-loop workflows, allowing you to use your internal SMEs to rate GenAI outputs without building custom UI infrastructure. |
| ### Question 5 - Continuous Toxicity and Safety Assessment A gaming company is using an LLM via Amazon Bedrock to generate dynamic dialogue for NPCs. Due to the unpredictable nature of player interactions, the company needs to evaluate the model's outputs continuously for toxicity and safety. They require a native AWS GenAI service metric that can be run as part of an automated evaluation pipeline without managing custom inference endpoints. Which solution is the most cost-effective and operationally efficient? - A) Deploy an Amazon Comprehend endpoint and route all NPC dialogue outputs to it for toxicity detection before displaying them to the user. - B) Create an Amazon Bedrock Evaluation job utilizing the built-in toxicity metric against a dataset of potential edge-case prompts. - C) Use SageMaker Model Monitor to track data drift on the input prompts. - D) Write a custom toxicity classifier using Scikit-Learn and host it on AWS Lambda. Select ONE that applies. |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock Evaluations includes a built-in toxicity metric specifically designed to assess GenAI outputs. Running this as an evaluation job against edge-case datasets is highly automated, requires no custom endpoint management, and is natively integrated into the GenAI workflow. Distractor Analysis: - A is Incorrect: While Amazon Comprehend can detect toxicity, deploying and routing real-time inference to a separate endpoint for evaluation introduces significant operational overhead and latency, and is better suited for real-time filtering rather than offline evaluation. - C is Incorrect: SageMaker Model Monitor tracks data drift on traditional ML model inputs, not the toxicity of LLM generative outputs. - D is Incorrect: Building and hosting a custom classifier on Lambda introduces significant operational overhead and maintenance compared to using a native Bedrock feature. > Key Takeaway: Use Amazon Bedrock Evaluations' built-in toxicity metric to assess content safety and quality of LLM outputs natively without managing separate NLP inference endpoints. |
| ### Question 6 - Customizing LLM-as-a-Judge Criteria A retail company is configuring an LLM-as-a-judge evaluation in Amazon Bedrock to assess the helpfulness of their customer service chatbot. The Data Science team wants the judge model to evaluate responses on a strict 1-to-5 scale based on clarity, tone, and accuracy. Furthermore, to improve the reliability of the judge, they need the judge to provide a reasoning step before assigning the final score. How should the team configure the LLM-as-a-judge evaluation to meet these requirements? - A) Use a programmatic evaluation job and configure the BLEU score threshold. - B) Select a human evaluation workflow and provide a 1-to-5 rubric to the human raters. - C) Define a custom prompt template for the LLM-as-a-judge that includes the specific scoring criteria and a chain-of-thought instruction to output reasoning before the score. - D) Deploy the judge model on a SageMaker Real-Time endpoint and pass the chatbot output via a custom Boto3 script. |
Correct Answer: C Correct Answer Analysis: - C is Correct: In Amazon Bedrock Evaluations, LLM-as-a-judge can be highly customized using prompt templates. The team can define specific criteria (clarity, tone, accuracy), specify the 1-5 scale, and instruct the model to output its reasoning (chain-of-thought) before the final score, ensuring reliable and transparent evaluation. Distractor Analysis: - A is Incorrect: Programmatic evaluations compute mathematical metrics like BLEU and do not use an LLM to judge clarity or tone. - B is Incorrect: The scenario specifies using an LLM-as-a-judge (automated model), not human raters. - D is Incorrect: Deploying a custom SageMaker endpoint and writing custom Boto3 scripts introduces unnecessary operational overhead compared to the native Bedrock Evaluation LLM-as-a-judge configuration. > Key Takeaway: LLM-as-a-judge in Bedrock can be customized with prompt templates to enforce specific scoring rubrics and chain-of-thought reasoning for higher evaluation fidelity. |
| ### Question 7 - Programmatic N-gram Overlap Metrics A localization team is evaluating an FM's capability to translate technical manuals from English to Spanish. They have a dataset of source texts and high-quality reference human translations. To establish a baseline, they want to compute a metric that directly compares the exact word-level overlap (n-grams) of the FM's output with the reference translation. Which evaluation metric should they use in an Amazon Bedrock Evaluation job? - A) BERTScore - B) Semantic Similarity - C) BLEU - D) LLM-as-a-judge Select ONE that applies. |
Correct Answer: C Correct Answer Analysis: - C is Correct: BLEU (Bilingual Evaluation Understudy) is a programmatic evaluation metric that measures the exact n-gram overlap between a generated text and a reference text. It is widely used for machine translation evaluation to assess lexical accuracy. Distractor Analysis: - A is Incorrect: BERTScore uses contextual embeddings to measure semantic similarity, not exact word-level n-gram overlap. - B is Incorrect: Semantic Similarity evaluates meaning and context, allowing for different word choices, whereas the requirement specifically asks for exact word-level overlap. - D is Incorrect: LLM-as-a-judge uses a language model to subjectively score the output based on custom criteria, rather than computing a deterministic mathematical n-gram overlap. > Key Takeaway: Use BLEU or ROUGE for programmatic, exact n-gram overlap evaluation, and use BERTScore or LLM-as-a-judge for semantic or subjective quality assessment. |
| ### Question 8 - Simultaneous Evaluation of Multiple Foundation Models An enterprise is building a coding assistant and wants to compare the performance of Anthropic's Claude 3 Haiku, Amazon Titan Text Premier, and Meta Llama 3 8B. They have a standardized dataset of 1,000 coding prompts and want to automatically score the outputs for correctness and completeness using an LLM-as-a-judge. The evaluation must be performed in a single, consolidated workflow to minimize operational overhead. Which approach satisfies these constraints? - A) Deploy all three models to SageMaker endpoints and use A/B testing to route live traffic to each, comparing user feedback. - B) Create a single Amazon Bedrock Evaluation job, selecting all three FMs as targets, and configure an LLM-as-a-judge to evaluate the outputs. - C) Run three separate AWS Step Functions state machines, one for each model, invoking a Lambda function to call Bedrock and store results in DynamoDB. - D) Use Amazon Bedrock Guardrails to evaluate the correctness of the outputs for each model simultaneously. |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock Evaluations allows you to select multiple FMs in a single evaluation job. You can provide one dataset and an LLM-as-a-judge, and the service will automatically run the prompts against all selected models and compare the scores side-by-side, providing the least operational overhead. Distractor Analysis: - A is Incorrect: A/B testing routes live traffic to users and requires user feedback, rather than using an automated dataset and LLM-as-a-judge for offline evaluation. - C is Incorrect: Building custom Step Functions and Lambda pipelines introduces significant operational overhead compared to the native Bedrock Evaluations multi-model comparison feature. - D is Incorrect: Bedrock Guardrails is used for content filtering, PII redaction, and safety, not for evaluating the correctness or completeness of code generation. > Key Takeaway: Amazon Bedrock Evaluations supports multi-model evaluation, allowing you to compare several FMs against a single dataset using programmatic or LLM-as-a-judge metrics in one consolidated job. |
| ### Question 9 - Evaluating Answer Relevance for Customer Service Bots A customer service bot powered by Amazon Bedrock is generating responses to user queries. The engineering team notices that while the generated answers are grammatically correct and fluent, they frequently fail to actually address the user's specific question. The team needs to implement an AI evaluation metric to measure how well the generated answers align with the intent of the user's prompt. Which evaluation metric should the team configure? - A) Context Utilization - B) Answer Relevance - C) Context Recall - D) Toxicity Select ONE that applies. |
Correct Answer: B Correct Answer Analysis: - B is Correct: Answer Relevance is a metric used in GenAI evaluation (often computed via LLM-as-a-judge or semantic similarity) specifically to evaluate whether the generated response actually addresses the core intent of the user's prompt or question. Distractor Analysis: - A is Incorrect: Context Utilization (Faithfulness) evaluates if the generated answer relies only on the provided retrieved context, not whether it answers the prompt. - C is Incorrect: Context Recall evaluates if the retrieval system successfully fetched all the necessary information from the knowledge base, a retrieval metric rather than a generation metric. - D is Incorrect: Toxicity measures the safety and harmfulness of the text, not its relevance to the user's query. > Key Takeaway: Answer Relevance specifically evaluates if the generated output directly addresses the user's prompt, isolating prompt comprehension from factual accuracy. |
| ### Question 10 - Storing Automated Evaluation Results An AI engineering team runs a large-scale Amazon Bedrock Evaluation job using an LLM-as-a-judge across 10,000 prompts. The results need to be stored securely in an AWS-native storage service for downstream analysis, compliance auditing, and dashboarding. The solution must be implemented with the LEAST operational overhead. Where does Amazon Bedrock automatically store these evaluation results, requiring no custom data pipeline? - A) Amazon DynamoDB - B) Amazon CloudWatch Logs - C) Amazon S3 - D) Amazon Redshift |
Correct Answer: C Correct Answer Analysis: - C is Correct: Amazon Bedrock Evaluation jobs automatically write the detailed results, including the prompts, generated responses, judge rationale, and scores, to an Amazon S3 bucket specified during the evaluation job setup. This native integration requires no custom data pipelines. Distractor Analysis: - A is Incorrect: Bedrock does not natively write evaluation results to DynamoDB without a custom integration layer. - B is Incorrect: While operational logs from the job might appear in CloudWatch, the actual evaluation results (scores and responses) are stored in S3. - D is Incorrect: Bedrock does not natively write directly to a Redshift data warehouse; doing so would require a custom ETL pipeline, adding operational overhead. > Key Takeaway: Amazon Bedrock Evaluations natively outputs detailed scoring results and LLM-as-a-judge rationales directly to an Amazon S3 bucket for secure, low-overhead downstream analysis. |