2.3.8

Question	Answer & Explanation
### Question 1 - Evaluating Translation Models with N-Gram Precision A machine learning team is evaluating an Amazon Bedrock model fine-tuned for translating technical documentation from English to Spanish. The team needs to programmatically evaluate the quality of the translations against human-generated reference translations. The evaluation metric must heavily penalize missing important technical keywords and measure precise n-gram overlap, applying a brevity penalty for outputs that are too short. Which evaluation metric should the team use in an Amazon Bedrock Evaluation job? Which ONE metric best fits this requirement? (Select ONE) - A) ROUGE-N - B) BLEU - C) BERTScore - D) Perplexity	Correct Answer: B Correct Answer Analysis: B is Correct: BLEU (Bilingual Evaluation Understudy) is a precision-oriented metric used primarily for machine translation. It measures the exact n-gram overlap between the generated text and the reference text, applying a brevity penalty to penalize translations that are too short, ensuring all critical keywords are present. Distractor Analysis: - A is Incorrect: ROUGE-N is a recall-oriented metric typically used for summarization, measuring how many n-grams from the reference appear in the candidate text. - C is Incorrect: BERTScore evaluates semantic similarity using contextual embeddings rather than exact n-gram precision and overlap, which does not strictly enforce keyword matching. - D is Incorrect: Perplexity measures how well a probability model predicts a sample, typically used to evaluate language models intrinsically rather than translation quality against a reference. > Exam Tip: Use BLEU for translation (precision-focused with brevity penalty) and ROUGE for summarization (recall-focused).
### Question 2 - Evaluating Summarization Models with Recall Metrics An AI startup is using a large language model (LLM) on Amazon Bedrock to generate automated executive summaries of quarterly financial reports. The engineering team must set up a Bedrock Evaluation job to assess the summarization quality. The chosen metric must evaluate how well the generated summary covers the key points present in the human-written reference summaries, prioritizing the recall of overlapping n-grams. The solution must be implemented with the least operational overhead. Which ONE metric best fits this requirement? (Select ONE) - A) BLEU - B) BERTScore - C) ROUGE - D) Word Error Rate (WER)	Correct Answer: C Correct Answer Analysis: C is Correct: ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is designed for evaluating summarization. It measures recall by calculating how many n-grams from the reference summary appear in the generated summary, ensuring that the model captures all the critical information. Distractor Analysis: - A is Incorrect: BLEU is a precision-based metric primarily used for translation, measuring how many n-grams in the generated text appear in the reference, not the other way around. - B is Incorrect: BERTScore evaluates semantic similarity using embeddings rather than explicit n-gram recall. - D is Incorrect: Word Error Rate (WER) measures the edit distance (insertions, deletions, substitutions) between a candidate and reference, commonly used in speech recognition rather than summarization recall. > Exam Tip: If the scenario emphasizes capturing all key points from the reference, use ROUGE (Recall). If it emphasizes exact matches in the output, use BLEU (Precision).
### Question 3 - Overcoming Synonym Penalties with Semantic Similarity A healthcare company is using Amazon Bedrock Knowledge Bases to retrieve and generate patient education materials. The generated text often uses different but synonymous medical terms compared to the reference documents (e.g., "high blood pressure" instead of "hypertension"). The team must evaluate the semantic similarity of the generated text to the reference text without penalizing the use of synonyms. Traditional n-gram overlap metrics are explicitly deemed insufficient by the compliance team. Which ONE metric should the team implement in their Amazon Bedrock Evaluation? (Select ONE) - A) BLEU-4 - B) BERTScore - C) ROUGE-L - D) Exact Match Accuracy	Correct Answer: B Correct Answer Analysis: B is Correct: BERTScore computes similarity using contextual embeddings (like BERT) rather than exact string matching. It maps tokens to embeddings and calculates cosine similarity, meaning synonyms like "hypertension" and "high blood pressure" will score highly without being penalized for lack of exact n-gram overlap. Distractor Analysis: - A is Incorrect: BLEU-4 strictly evaluates 4-gram precision, requiring exact word matches. It would heavily penalize synonymous replacements. - C is Incorrect: ROUGE-L evaluates the longest common subsequence based on exact word matches, failing to account for semantic equivalence. - D is Incorrect: Exact Match Accuracy requires the generated text to match the reference text word-for-word, making it the worst choice for evaluating synonyms. > Exam Tip: When a scenario explicitly mentions synonyms, paraphrasing, or different wordings with the same meaning, select BERTScore or Semantic Similarity.
### Question 4 - Evaluating Longest Common Subsequence for Code Generation A data science team is evaluating an LLM used for generating code documentation snippets. The generated documentation must closely follow the structural sequence of the reference documentation. The team is configuring an automated evaluation pipeline in Amazon Bedrock. They need a metric that captures the longest common subsequence of words between the candidate and reference text to measure structural overlap. The evaluation must be fully managed to minimize operational overhead. Which ONE metric should they choose? (Select ONE) - A) ROUGE-L - B) BLEU-2 - C) BERTScore - D) Cosine Similarity	Correct Answer: A Correct Answer Analysis: A is Correct: ROUGE-L specifically measures the Longest Common Subsequence (LCS) between the reference and candidate text. It is useful for evaluating the structural alignment and sequence of words, as it doesn't require n-grams to be perfectly contiguous, only in the same order. Distractor Analysis: - B is Incorrect: BLEU-2 evaluates contiguous 2-gram (bigram) precision, which requires exact sequential matches and does not inherently measure the longest common subsequence. - C is Incorrect: BERTScore calculates embedding cosine similarity, evaluating semantic meaning rather than structural sequence overlap. - D is Incorrect: While Cosine Similarity is a vector space metric, it is not the specific NLP metric designed to measure the longest common subsequence. > Exam Tip: ROUGE-L = Longest Common Subsequence. It evaluates structural order without requiring contiguous matches.
### Question 5 - Combined Precision and Recall Evaluation An AI engineering team is configuring an automated evaluation job in Amazon Bedrock to assess a text generation model. The compliance team mandates that the evaluation must capture both the precision of n-grams (to ensure no irrelevant information is hallucinated) and the recall of n-grams (to ensure no critical information from the reference is omitted). The solution must use native AWS capabilities with the least operational overhead. Which TWO metrics should the team select in their evaluation configuration? (Select TWO) - A) BLEU - B) ROUGE - C) BERTScore - D) Perplexity - E) Toxicity	Correct Answers: A and B Correct Answer Analysis: - A is Correct: BLEU is a precision-oriented metric, measuring the proportion of n-grams in the generated text that appear in the reference text. This helps ensure the model isn't generating out-of-context or irrelevant information. - B is Correct: ROUGE is a recall-oriented metric, measuring the proportion of n-grams in the reference text that appear in the generated text. This ensures the model is capturing all necessary information. Distractor Analysis: - C is Incorrect: BERTScore measures semantic similarity using embeddings, not explicit n-gram precision or recall based on exact word overlaps. - D is Incorrect: Perplexity measures the model's predictive confidence and does not compare output against a reference text for precision or recall. - E is Incorrect: Toxicity metrics evaluate the safety and harmfulness of the content, not n-gram precision and recall. > Exam Tip: Use BLEU and ROUGE together when you need to balance Precision (BLEU) and Recall (ROUGE) in text generation tasks.
### Question 6 - Semantic vs Lexical Evaluation for RAG Responses A retail company is evaluating a Retrieval-Augmented Generation (RAG) application built on Amazon Bedrock. The team notices that the generated answers are factually correct and comprehensive but are phrased completely differently from the reference answers in their evaluation dataset. The team needs an evaluation metric that leverages pre-trained contextual embeddings to measure how close the meaning of the generated answer is to the reference, ignoring exact word matches. Which ONE approach satisfies this requirement with the least operational overhead? (Select ONE) - A) Configure an Amazon Bedrock Evaluation using built-in semantic similarity - B) Deploy a custom SageMaker Processing job to calculate ROUGE-1 - C) Use AWS Lambda with a custom script to calculate BLEU - D) Manually compare text using Amazon Mechanical Turk	Correct Answer: A Correct Answer Analysis: A is Correct: Amazon Bedrock Evaluations offers built-in metrics including semantic similarity, which calculates the underlying meaning of the responses using contextual embeddings. This operates fully managed, meeting the least operational overhead constraint. Distractor Analysis: - B is Incorrect: ROUGE-1 evaluates unigram overlap and requires exact word matches, which fails the requirement of ignoring exact word matches. Furthermore, deploying a custom processing job introduces unnecessary operational overhead. - C is Incorrect: BLEU evaluates n-gram precision and requires exact word matches, failing the semantic requirement. Running it in Lambda introduces custom code overhead. - D is Incorrect: Human evaluation via Mechanical Turk introduces significant operational overhead and does not leverage automated contextual embeddings. > Exam Tip: Always prefer Amazon Bedrock's native built-in evaluation metrics (like Semantic Similarity or BERTScore) over deploying custom compute (SageMaker/Lambda) to minimize operational overhead.
### Question 7 - Debugging Low Lexical Scores in Creative Generation A media company is using an LLM to generate creative news headlines from article bodies. They are evaluating the model using the BLEU metric in an Amazon Bedrock Evaluation job. The team observes very low BLEU scores despite the headlines being highly relevant, catchy, and meaningually equivalent to the reference headlines. The team wants to add a metric that better evaluates the underlying meaning rather than exact word overlap. Which ONE metric should they add to the evaluation? (Select ONE) - A) BERTScore - B) Word Error Rate (WER) - C) F1-score (Exact Match) - D) ROUGE-N	Correct Answer: A Correct Answer Analysis: A is Correct: BERTScore leverages contextual embeddings to evaluate semantic equivalence. Since the headlines are creatively phrased but meaningfully equivalent to the reference, BERTScore will accurately reflect the high quality of the generation without penalizing lexical differences. Distractor Analysis: - B is Incorrect: Word Error Rate (WER) strictly counts insertions, deletions, and substitutions, heavily penalizing creative phrasing and exact-word differences, similar to BLEU. - C is Incorrect: Exact Match F1-score requires lexical overlap and exact token matches, failing to capture semantic meaning in creative text. - D is Incorrect: ROUGE-N still relies on exact n-gram overlap (recall), which does not solve the problem of evaluating differing word choices in creative headlines. > Exam Tip: When generated text is "creative" or phrased differently from the reference but retains the same meaning, always choose embedding-based metrics (BERTScore or Semantic Similarity) over lexical overlap metrics.
### Question 8 - Measuring Unigram Recall for Data Extraction A financial services firm is running an LLM on Amazon Bedrock to extract structured key financial figures from unstructured earnings reports. The team must evaluate the model's output to ensure that all key financial figures from the reference text are present in the generated output. They specifically require a metric that measures the proportion of unigrams from the reference text that appear in the generated output. Which ONE NLP metric specifically measures this recall? (Select ONE) - A) ROUGE-1 - B) BLEU-1 - C) BERTScore - D) Perplexity	Correct Answer: A Correct Answer Analysis: A is Correct: ROUGE-1 measures unigram recall. It calculates the number of matching unigrams (single words) between the reference and the candidate text, divided by the total number of unigrams in the reference text. This precisely matches the requirement of ensuring all reference figures (represented as words) are present in the output. Distractor Analysis: - B is Incorrect: BLEU-1 measures unigram precision, meaning it checks how many unigrams in the generated text appear in the reference text. This evaluates conciseness/relevance rather than completeness (recall). - C is Incorrect: BERTScore measures semantic similarity through embeddings, not the explicit proportion of overlapping unigrams. - D is Incorrect: Perplexity evaluates the probability distribution of the language model and does not compare extracted facts against a reference text. > Exam Tip: "ROUGE-1" equals unigram recall. "ROUGE-2" equals bigram recall. Use ROUGE-N when the scenario demands verifying that all reference elements are captured in the output.
### Question 9 - Establishing a Baseline with Brevity Penalty An AWS customer is fine-tuning a model in Amazon SageMaker AI to generate short product descriptions. They need to establish a baseline for model performance using standard NLP evaluation metrics. The baseline must specifically evaluate the precision of unigrams and bigrams in the generated output, but it must explicitly apply a brevity penalty to heavily penalize outputs that are too short and fail to provide a complete description. The team must automate this without managing servers. Which ONE metric must be used? (Select ONE) - A) BLEU - B) ROUGE-L - C) BERTScore - D) F1-Score	Correct Answer: A Correct Answer Analysis: A is Correct: BLEU calculates modified n-gram precision (for unigrams, bigrams, etc.) and multiplies the result by a brevity penalty. The brevity penalty explicitly penalizes generated texts that are shorter than the reference text, preventing the model from achieving high precision by simply outputting a single correct word. Distractor Analysis: - B is Incorrect: ROUGE-L measures the longest common subsequence and is recall-oriented; it does not inherently apply a brevity penalty to penalize short outputs based on precision. - C is Incorrect: BERTScore uses embeddings to calculate semantic similarity and does not apply a brevity penalty based on output length. - D is Incorrect: F1-Score combines precision and recall but does not apply a specific brevity penalty for translation or text generation tasks like BLEU does. > Exam Tip: The brevity penalty is a unique mathematical component of the BLEU score, designed to prevent the model from cheating by generating very short, precise outputs.
### Question 10 - Selecting the Right Tool for NLP Metric Computation An AI team wants to evaluate a generative AI model using BERTScore and ROUGE across a large dataset of 10,000 prompt-response pairs. The team needs to execute this evaluation job programmatically, store the results for tracking, and avoid provisioning any underlying compute instances. The evaluation must run automatically upon the completion of model training. Which ONE AWS service offers the most operationally efficient way to achieve this? (Select ONE) - A) Deploy an Amazon SageMaker Processing job with a custom Hugging Face script - B) Create an Amazon Bedrock Evaluation job using built-in metrics - C) Use an AWS Lambda function triggered by Amazon EventBridge - D) Spin up an Amazon EC2 instance with a custom Jupyter notebook	Correct Answer: B Correct Answer Analysis: B is Correct: Amazon Bedrock Evaluations is a fully managed service that supports programmatic evaluation of models using built-in metrics such as BERTScore, ROUGE, and BLEU. It eliminates the need to provision compute infrastructure, handles the dataset processing, and provides the results natively, meeting all constraints with the lowest operational overhead. Distractor Analysis: - A is Incorrect: While a SageMaker Processing job can compute these metrics using custom scripts, it requires managing the container, script dependencies, and compute provisioning, introducing unnecessary operational overhead compared to Bedrock's built-in evaluations. - C is Incorrect: AWS Lambda has execution time limits (15 minutes) and memory constraints that make it unsuitable for running embedding computations (like BERTScore) across a 10,000-row dataset. - D is Incorrect: Spinning up an EC2 instance requires full operating system management, manual script execution, and lacks native tracking capabilities, representing the highest operational overhead. > Exam Tip: Always favor Amazon Bedrock Evaluations or SageMaker Clarify for built-in algorithmic evaluations over custom code on EC2, Lambda, or Processing jobs to minimize operational overhead.