Skip to content

4.1.6

Question 1 - Foundation Model Evaluation with Amazon Bedrock

A financial services company is deploying a generative AI chatbot built on Amazon Bedrock that answers customer queries about investment products from a knowledge base. Before releasing the chatbot to production, compliance requires the team to prove that the model's responses are factually grounded in source documents and maintain high answer quality across a curated set of prompts. The team has limited data science resources and cannot manually review hundreds of responses, but needs to quantify both accuracy and verbosity.

Which TWO actions should the team take to automate the model evaluation process using Amazon Bedrock Model Evaluation? (Select TWO)

  • A) Create an automatic evaluation job that calculates metrics like faithfulness, answer relevancy, and helpfulness using pricing-side algorithms without requiring manual reviews.
  • B) Configure an LLM-as-a-judge evaluation job where a judge model scores the foundational model's responses on criteria such as helpfulness, correctness, and faithfulness, providing explanations for each score.
  • C) Deploy the chatbot with Amazon SageMaker Inference Recommender to automatically benchmark response quality against a baseline model.
  • D) Use AWS X-Ray to trace each invocation and compute aggregate response quality metrics over time.
  • E) Set up a human-based evaluation job where the financial compliance team manually scores closed-ended prompts on metrics like correctness.
Click to reveal Answer & Explanation

Correct Answers: B and A

Correct Options Analysis

  • B is Correct: Amazon Bedrock Model Evaluation's LLM-as-a-judge mode uses a judge model to evaluate responses against predefined quality dimensions (helpfulness, correctness, faithfulness), automating the scoring process while also providing explanations for each score—essential for compliance review.
  • A is Correct: Bedrock's automatic evaluation mode computes heuristic-based metrics such as faithfulness, answer relevancy, and helpfulness, requiring no manual review and allowing the team to quantify quality across a large prompt set with minimal data science effort.

Distractor Analysis

  • C is Incorrect: Amazon SageMaker Inference Recommender optimizes instance types and cost-performance trade-offs for model hosting, not response quality scoring or evaluation of FM outputs.
  • D is Incorrect: AWS X-Ray traces request flows and identifies latency or error bottlenecks, but does not calculate semantic response quality metrics like faithfulness or helpfulness.
  • E is Incorrect: While human-based evaluation is a valid feature, it violates the "limited data science resources" and "cannot manually review" constraints—the question asks for automated evaluation approaches.

Key Takeaway: Amazon Bedrock Model Evaluation offers three modes—automatic metrics (heuristic-based), LLM-as-a-judge (with explanations), and human-based—each serving different cost/resource trade-offs depending on the compliance rigor required.