Skip to content

Skill 1.1.1: Extract data from data sources

Question 1 - Selecting Appropriate NLP Evaluation Metric for Machine Translation

A multinational e-commerce company is building an automated product description translation system using Amazon Translate and a custom fine-tuned model on Amazon SageMaker AI. The system translates product titles and descriptions from English into German, French, and Japanese for their global marketplaces.

The ML team needs to evaluate the quality of machine-generated translations against human reference translations stored in their evaluation dataset. The team wants to measure the precision of n-gram matches between the generated translations and reference translations, with a specific focus on ensuring the generated text does not contain excessive word repetitions that would make translations sound unnatural. The evaluation must be battery-runnable as an automated programmatic metric using their existing Amazon Bedrock evaluations infrastructure.

Which metric should the team select as their primary automated evaluation measure? (Select ONE)

  • A) ROUGE-N with recall-focused scoring
  • B) BLEU (Bilingual Evaluation Understudy)
  • C) Semantic Similarity using cosine distance between sentence embeddings
  • D) BERTScore with embedding-based token matching
  • E) Perplexity measured on a held-out validation corpus
Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: BLEU is specifically designed for machine translation evaluation and measures n-gram precision (how many n-grams in the generated translation appear in the reference translations). BLEU includes a brevity penalty to discourage overly short translations, and its precision-focused design naturally penalizes excessive word repetition in generated text, making it ideal for this use case.

Distractor Analysis

  • A is Incorrect: ROUGE is designed for summarization tasks and comes in multiple variants (ROUGE-N, ROUGE-L, ROUGE-SU). While ROUGE-N can be recall-focused, recall measures how many reference n-grams appear in the generated text, which is not the primary concern for translation quality where precision against over-generation matters more.
  • C is Incorrect: Semantic similarity using sentence embeddings (e.g., via Amazon Titan Text Embeddings or BERT) captures meaning similarity but does not directly measure n-gram precision or address word repetition issues in translation tasks.
  • D is Incorrect: BERTScore uses contextual token embeddings to compute similarity between generated and reference text, measuring semantic token alignment rather than exact n-gram precision. It is better suited for paraphrase evaluation rather than translation precision.
  • E is Incorrect: Perplexity measures how well a language model predicts the next token in a sequence (model confidence), not how closely machine translations match human reference translations. It is a language model quality measure, not a translation evaluation metric.

Key Takeaway: BLEU measures n-gram precision in machine translation with a built-in penalty for overly short translations, making it the standard automated metric for translation quality evaluation where word repetition must be controlled.

Question 2 - Evaluating Abstractive Summarization with Multiple Reference Summaries

A healthcare research organization uses Amazon Bedrock to generate abstractive summaries of clinical trial reports stored in their Amazon S3 data lake. Their evaluation framework uses Amazon SageMaker AI MLflow for experiment tracking.

The organization needs to evaluate the quality of generated summaries against a dataset of human-written reference summaries. The summaries vary significantly in length, and multiple valid summaries exist for each source document. The metric must capture both the recall of key information units from the reference summaries and handle varying summary lengths without a length bias penalty. The evaluation must also work reliably when comparing against multiple reference summaries per source.

Which TWO metrics should form the primary evaluation suite? (Select TWO)

  • A) BLEU-4 with unmodified brevity penalty
  • B) ROUGE-L with longest common subsequence matching
  • C) BERTScore with TF-IDF weighted token matching
  • D) ROUGE-SU4 with skip-bigram and unigram co-occurrence
  • E) Exact match accuracy (EM) against each reference summary
Click to reveal Answer & Explanation

Correct Answers: B and C

Correct Options Analysis

  • B is Correct: ROUGE-L uses longest common subsequence (LCS) matching between generated and reference summaries. LCS naturally handles variable-length text without requiring consecutive word matches, and ROUGE-L is the standard recall-oriented metric for abstractive summarization tasks where information recall from source references is the primary concern.
  • C is Correct: BERTScore computes token-level similarities using contextual embeddings from BERT models and can be weighted by TF-IDF to emphasize important content words over common stopwords. It handles variable-length summaries well through greedy token matching and supports multiple reference summaries by averaging scores across all references.

Distractor Analysis

  • A is Incorrect: BLEU-4 with a brevity penalty is precision-focused and penalizes shorter generated text that may still capture key information. BLEU's brevity penalty can unfairly penalize concise abstractive summaries and is better suited for machine translation where length should match source closely.
  • D is Incorrect: ROUGE-SU4 uses skip-bigram matching, which is designed to capture sentence-level structural similarity but can be computationally expensive and less interpretable than ROUGE-L for document-level summarization evaluation.
  • E is Incorrect: Exact Match (EM) is a brittle metric that requires the generated summary to exactly match a reference summary word-for-word. For abstractive summarization with multiple valid summaries, EM would approach zero even for high-quality generated summaries.

Key Takeaway: For abstractive summarization with variable lengths and multiple references, use ROUGE-L for recall of key information and BERTScore for semantic token-level similarity — avoiding precision-biased metrics like BLEU that penalize conciseness.

Question 3 - Semantic Similarity for Paraphrase Detection Across Languages

A global customer support platform using Amazon Connect and Amazon Lex needs to evaluate whether their chatbot responses are semantically equivalent to a curated set of high-quality response templates. The evaluation happens in real-time for chat interactions across English, Spanish, and German.

The constraint is that lexical overlap metrics are insufficient because the responses and references use different but valid wording, and the evaluation must capture meaning across multiple languages using a multilingual embedding model. The team wants to use a metric that leverages cross-lingual semantic representations from a transformer model to measure token-level meaning alignment without requiring exact word matches.

Which evaluation approach is most appropriate? (Select ONE)

  • A) ROUGE-N with unigram matching against the response template
  • B) Cosine similarity between multilingual sentence embeddings (e.g., using amazon.titan-embed-text-v2)
  • C) BLEU with smoothing for cross-lingual n-gram matching
  • D) BERTScore with multilingual contextual embeddings (e.g., BERT multilingual variant) using cosine similarity between token embeddings
  • E) Hamming distance between feature vectors of the response and reference
Click to reveal Answer & Explanation

Correct Answer: D

Correct Option Analysis

  • D is Correct: BERTScore with multilingual BERT embeddings computes contextual token-level similarities between generated and reference text using cosine similarity. The contextual embeddings from multilingual BERT models (e.g., bert-base-multilingual-cased) are trained on 100+ languages, enabling cross-lingual semantic comparison without requiring lexical overlap. This makes it ideal for evaluating paraphrase equivalence across languages.

Distractor Analysis

  • A is Incorrect: ROUGE-N with unigram matching requires lexical overlap of exact words between the generated response and reference. In a paraphrased response with different valid wording and cross-lingual context, ROUGE-N would score near zero.
  • B is Incorrect: While cosine similarity between sentence embeddings is a valid semantic similarity approach, sentence-level embeddings compress the entire sentence into a single vector, losing token-level detail. BERTScore provides more granular token-to-token similarity metrics (precision, recall, F1), which is more appropriate for detailed evaluation of response quality.
  • C is Incorrect: BLEU operates on exact character or word n-gram matches within the same language. BLEU does not support cross-lingual n-gram matching without a separate translation layer, and adding translation would introduce an unnecessary external dependency.
  • E is Incorrect: Hamming distance measures binary difference between feature vectors (e.g., hash codes or one-hot encodings). It is not designed for semantic text comparison and would fail to capture meaning equivalence between differently worded responses.

Key Takeaway: BERTScore with multilingual transformer embeddings provides token-level semantic similarity across languages, making it the best choice when exact word overlap metrics (BLEU/ROUGE) fail due to paraphrasing or cross-lingual variation.

Question 4 - RAG System Context Relevance Evaluation

A financial services firm has deployed a Retrieval-Augmented Generation (RAG) system using Amazon Bedrock Knowledge Bases with Amazon OpenSearch Serverless as the vector store. Customer queries retrieve semantic chunks of financial documentation, and an LLM generates answers based on the retrieved context.

Compliance requires that the team monitor the relevance of retrieved passages separately from the quality of generated answers. The team needs a metric that evaluates whether the retrieved contexts pulled from the vector store are semantically similar to the reference ground-truth contexts in their evaluation dataset. The metric should use embedding-based similarity and provide a continuous score in the range [0, 1], where 1 indicates perfect semantic alignment between retrieved and ground-truth contexts.

Which evaluation technique should the team use for context relevance assessment? (Select ONE)

  • A) ROUGE-L between retrieved passages and ground-truth passages
  • B) BLEU between generated answers and ground-truth answers
  • C) Cosine similarity between embeddings of retrieved passages and ground-truth passages using Amazon Titan Text Embeddings
  • D) Human evaluation of retrieved passages by compliance officers
  • E) Token count ratio between retrieved and ground-truth passages
Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

  • C is Correct: Cosine similarity between embedding vectors (e.g., generated by Amazon Titan Text Embeddings) measures semantic alignment between retrieved passages and ground-truth passages. It produces a continuous score typically in [-1, 1] (or [0, 1] for normalized positive embeddings), where higher values indicate greater semantic similarity. This is the standard approach for automated RAG retrieval relevance evaluation.

Distractor Analysis

  • A is Incorrect: ROUGE-L measures lexical overlap (longest common subsequence) between text strings. While it can measure word-level overlap between retrieved and ground-truth passages, it fails to capture semantic equivalence when passages use different wording, which is common in retrieved chunk comparisons.
  • B is Incorrect: BLEU is a precision-focused n-gram overlap metric designed for machine translation. It evaluates generated answers, not retrieved passage relevance, and fails to capture semantic similarity when retrieved text paraphrases the ground truth with different words.
  • D is Incorrect: Human evaluation by compliance officers is a human-based evaluation approach, not an automated continuous scoring metric. While it provides high-quality judgment, it does not produce automated continuous scores in [0, 1] that can be monitored programmatically at scale.
  • E is Incorrect: Token count ratio is a simple length comparison metric that measures whether retrieved and ground-truth passages have similar token lengths, not whether they are semantically aligned. This metric completely ignores content relevance.

Key Takeaway: For RAG retrieval monitoring, use embedding-based cosine similarity to measure semantic alignment between retrieved and reference contexts — keeping retrieval evaluation separate from generation evaluation to isolate failure points.

Question 5 - LLM-as-a-Judge with BLEU and ROUGE for Customer Support Quality

A telecommunications company uses Amazon Bedrock to power a customer support chatbot that answers billing and plan-related questions. The company will evaluate the chatbot responses against a curated set of 500 reference question-answer pairs.

The evaluation framework has two stages. First, an automated programmatic evaluation computes ROUGE and BLEU scores against the reference answers. Second, an Amazon Bedrock judge model scores the responses on a 1–5 Likert scale for helpfulness with reasoning. The team wants to ensure the correlation between automated lexical scores and human/judge model quality scores is understood and that both overlap metrics are interpreted correctly.

Given that customer support answers often vary in length and paraphrasing, which TWO statements correctly interpret the results? (Select TWO)

  • A) Low ROUGE-L scores combined with high judge model scores indicate lexical overlap metrics are poor proxies for quality in this paraphrase-heavy domain.
  • B) High BLEU scores with high judge model scores indicate translation-quality responses in a customer support context.
  • C) ROUGE-L measures recall of important information units from reference answers and is more appropriate for evaluating key information coverage than BLEU.
  • D) High ROUGE scores always indicate high-quality customer support responses, regardless of judge model scores.
  • E) BLEU and ROUGE are sufficient for evaluating customer satisfaction without any additional human or judge model assessment.
Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: In domains with high paraphrase variability, lexical metrics (ROUGE, BLEU) suffer from n-gram mismatch equivalence. A response can use different but valid wording while the judge model rates it highly. This discrepancy signals that overlap metrics are imperfect proxies for quality in paraphrase-heavy contexts.
  • C is Correct: ROUGE-L is recall-oriented and measures how many reference answer n-grams (via longest common subsequence) are present in the generated answer. This makes it better suited for evaluating whether key information from the reference is covered in the response, compared to precision-focused BLEU that penalizes longer responses.

Distractor Analysis

  • B is Incorrect: BLEU is designed for machine translation where the generated text should closely mirror the source in length. In customer support, a terse, helpful answer (e.g., "Yes, your bill is $50") may score low on BLEU despite being excellent in quality. High BLEU does not indicate "translation-quality responses" in this context.
  • D is Incorrect: High ROUGE scores measure lexical overlap but do not guarantee factual correctness, helpfulness, or customer satisfaction. A response can mimic the reference wording while containing hallucinations or being inappropriate for the specific customer context.
  • E is Incorrect: Automated lexical metrics are insufficient for evaluating open-ended customer support quality dimensions like empathy, policy compliance, and contextual appropriateness. Human evaluation or judge model assessment is needed to capture these qualitative aspects.

Key Takeaway: BLEU and ROUGE measure n-gram overlap, not semantic quality — in paraphrase-heavy domains, always pair automated metrics with human evaluation or judge model scores to capture aspects lexical metrics miss.

Question 6 - BERTScore Implementation with Amazon Bedrock Evaluations

An enterprise knowledge management team wants to evaluate their LLM-powered document summarization system using Amazon Bedrock evaluations. They need a metric that measures semantic similarity between generated summaries and reference summaries using contextual token embeddings.

The team needs to configure BERTScore as their evaluation metric within the Bedrock evaluations framework. They will use the bertscore Python library with a pre-trained transformer model.

Which THREE configuration choices correctly apply BERTScore for this task? (Select THREE)

  • A) Use a BERT-based contextual embedding model (e.g., microsoft/deberta-xlarge-mnli) to generate token embeddings for both generated and reference summaries.
  • B) Compute token-level cosine similarity between generated and reference token embeddings and apply greedy matching to calculate precision, recall, and F1 scores.
  • C) Use TF-IDF weighting to emphasize content-bearing tokens over common stopwords during the token similarity calculation.
  • D) Apply a brevity penalty to the BERTScore F1 score to penalize short summaries, similar to BLEU.
  • E) Use the BLEU score as the BERTScore when the summaries are longer than 500 tokens.
Click to reveal Answer & Explanation

Correct Answers: A and B and C

Correct Options Analysis

  • A is Correct: BERTScore requires contextual token embeddings from a pre-trained transformer model (e.g., BERT, RoBERTa, DeBERTa, or multilingual variants). These embeddings capture token-level context, enabling semantic similarity comparison beyond exact n-gram matching.
  • B is Correct: BERTScore computes cosine similarity between all token embedding pairs across generated and reference text, then uses greedy matching (each token matched to the most similar counterpart) to compute precision, recall, and F1 scores. This is the core BERTScore algorithm.
  • C is Correct: TF-IDF weighting is a standard optional enhancement in BERTScore implementations. It assigns higher importance to content-bearing tokens (e.g., domain-specific terms like "quarterly revenue") over common stopwords (e.g., "the", "of"), improving correlation with human judgment.

Distractor Analysis

  • D is Incorrect: BERTScore does not include a brevity penalty. BLEU specifically uses a brevity penalty to discourage overly short translations, but BERTScore's recall component already handles length variations naturally without requiring an explicit penalty.
  • E is Incorrect: BLEU and BERTScore are completely different metrics and cannot be used interchangeably. BLEU measures exact n-gram overlap, while BERTScore measures semantic token similarity via embeddings. There is no rule that substitutes BLEU for BERTScore based on token length.

Key Takeaway: BERTScore requires contextual transformer embeddings, cosine similarity, and greedy token matching as its core; TF-IDF weighting is an optional enhancement to focus on content tokens — no brevity penalty is involved.

Question 7 - Choosing Between ROUGE Variants for Meeting Summarization

A project management SaaS company uses Amazon Transcribe to convert virtual meeting recordings into text and subsequently summarizes the transcript using an LLM through Amazon Bedrock. For evaluation, the company maintains a dataset of meeting transcripts paired with human-generated executive summaries.

The evaluation needs to measure how well the generated summaries capture ordered key meeting outcomes and action items from the human reference summaries. The summaries often have non-contiguous key phrases separated by filler text, and the team prefers a metric that can capture sentence-level structure beyond simple unigram overlap.

Which ROUGE metric is most appropriately designed for this scenario? (Select ONE)

  • A) ROUGE-N with N=1 (unigram ROUGE)
  • B) ROUGE-L with longest common subsequence matching
  • C) ROUGE-SU4 with skip-bigram plus unigram co-occurrence
  • D) ROUGE-WE with weighted LCS
  • E) ROUGE precision with exact n-gram matching
Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

  • C is Correct: ROUGE-SU4 combines skip-bigram matching (capturing pairs of words in order with arbitrary gaps between them) with unigram co-occurrence. This is ideal for capturing key phrases and action items that may be separated by filler text. Skip-bigrams preserve word order while allowing intervening words, matching the scenario of non-contiguous key information in meeting summaries.

Distractor Analysis

  • A is Incorrect: ROUGE-N (unigram) only measures single-word overlap, ignoring word order and phrase structure. It would miss whether key action items appear in the correct sequence, making it a poor fit for meeting summaries where ordered key phrases matter.
  • B is Incorrect: ROUGE-L with LCS captures word order through longest common subsequence but does not explicitly reward non-contiguous paired words (skip-bigrams) that might be separated by unrelated filler content.
  • D is Incorrect: ROUGE-WE (Weighted LCS) assigns different weights to LCS matches based on consecutive match length, which is designed to penalize gaps within matches. This would be counterproductive for this scenario where key phrases may be non-contiguous.
  • E is Incorrect: Simple precision-based exact n-gram matching (not a standard ROUGE variant) would penalize summaries that use valid sentence restructuring while preserving meaning and ordered key points.

Key Takeaway: ROUGE-SU variants combine skip-bigram and unigram overlap, making them ideal for evaluating phrase-level structure in summaries where key items may appear non-contiguously with filler text between them.

Question 8 - Semantic Similarity for Answer Validation in Contact Center Analytics

An insurance company processes claim-related customer calls using Amazon Connect Contact Lens and generates automated response suggestions for agents using a fine-tuned LLM on Amazon SageMaker AI. The system suggests agent responses based on customer queries and historical resolution notes.

The evaluation dataset contains 5,000 agent response suggestion candidates paired with 10 reference response templates each. The team needs to validate that each generated suggestion is semantically equivalent to at least one of the reference templates, not merely lexically similar. The evaluation will run as a batch process after model updates using AWS Step Functions orchestration.

Which combination of evaluation technique and justification is correct? (Select ONE)

  • A) Use ROUGE-L between the suggested response and all reference templates, select the highest score, and classify as correct if ROUGE-L ≥ 0.50
  • B) Use BLEU-4 between the suggested response and each reference template, select the highest score, and classify as correct if BLEU ≥ 0.30
  • C) Use cosine similarity between sentence embeddings of the suggested response and each reference template using a sentence transformer model; classify as correct if similarity ≥ threshold (e.g., 0.80)
  • D) Use exact string matching between the suggested response and reference templates; classify as correct only if an exact match exists
  • E) Use BERTScore F1 between the suggested response and all reference templates with greedy token matching; classify as correct if the maximum F1 score across all references exceeds threshold
Click to reveal Answer & Explanation

Correct Answer: E

Correct Option Analysis

  • E is Correct: BERTScore F1 with greedy token matching provides token-level semantic similarity between the generated suggestion and each reference template. Using the maximum F1 score across all 10 reference templates correctly identifies semantic equivalence when at least one reference is meaning-equivalent. BERTScore captures paraphrase-level meaning via contextual token embeddings, making it superior to lexical metrics for semantic equivalence validation.

Distractor Analysis

  • A is Incorrect: ROUGE-L measures lexical overlap via longest common subsequence. While it can capture some ordered word overlap, it fails to recognize semantic equivalence when the suggestion uses synonyms, paraphrasing, or restructuring that alters the word sequence.
  • B is Incorrect: BLEU-4 is precision-focused n-gram overlap designed for machine translation. It penalizes length variations and requires exact word matches, making it unsuitable for validating semantic equivalence in a paraphrase-heavy domain like customer service response suggestions.
  • C is Incorrect: While cosine similarity between sentence embeddings is a valid semantic similarity approach, sentence-level embeddings compress token-level details into a single vector. BERTScore provides more granular token-to-token matching with separate precision/recall/F1 components, which offers more nuanced evaluation for multi-reference comparison tasks.
  • D is Incorrect: Exact string matching is a brittle validation metric that would score zero for valid paraphrased responses. Customer service response suggestions can be written in myriad valid forms, rendering exact-match validation unusable.

Key Takeaway: For semantic equivalence validation with multiple references, use BERTScore F1 with greedy token matching and take the maximum score across all references — this captures paraphrase-level meaning without requiring exact word matches.

Question 9 - Monitoring ROUGE and BERTScore for Drift in Automated Report Generation

A financial reporting automation firm uses a fine-tuned LLM deployed via Amazon SageMaker AI to generate automated quarterly earnings report summaries. The firm has been monitoring ROUGE-L recall and BERTScore F1 as quality metrics for 12 months.

Recently, the ROUGE-L recall metric has dropped from 0.45 to 0.35, while BERTScore F1 has remained stable at 0.72. The system has not been retrained, but the incoming source data has shifted to include more merger and acquisition (M&A) filings than in previous quarters. The ML team is investigating the drift.

What is the most likely cause of this specific metric divergence, and what action should the team take? (Select ONE)

  • A) The LLM has experienced catastrophic forgetting during the data shift, and the team should retrain immediately on the new data distribution.
  • B) The ROUGE-L metric is sensitive to lexical patterns in the source data; the shift to M&A filings has introduced domain-specific terminology not present in the original training data, causing lower n-gram overlap despite maintained semantic quality (BERTScore stability).
  • C) BERTScore is broken and should be recalibrated to match the ROUGE-L trend.
  • D) The drop in ROUGE-L indicates the model is producing longer summaries, and the team should add a brevity penalty to the evaluation pipeline.
  • E) ROUGE-L and BERTScore are measuring different things, so the divergence is irrelevant and the team should ignore ROUGE-L going forward.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: This is a classic case of data drift where the input distribution shifts (more M&A filings introduce new domain-specific terminology). The LLM, trained primarily on standard earnings reports, struggles to generate text that lexically matches reference summaries containing M&A jargon. However, the generated summaries still maintain semantic quality as measured by BERTScore. Lexical overlap metrics like ROUGE-L drop because the reference summaries contain domain-specific n-grams not present in the generated text, while BERTScore captures semantic equivalence through token embeddings.

Distractor Analysis

  • A is Incorrect: Catastrophic forgetting occurs during retraining when new data overwrites previously learned patterns. Since no retraining occurred, this is not the cause. The divergence is better explained by data drift affecting lexical overlap metrics.
  • C is Incorrect: BERTScore is a well-established metric using contextual embeddings and is unlikely to be "broken." The stability of BERTScore at 0.72 specifically indicates semantic quality is maintained despite lexical drift.
  • D is Incorrect: A drop in ROUGE-L could theoretically indicate longer generated summaries (lower recall if the denominator is larger), but the question states the source data shifted, not the summary length. Adding a brevity penalty is not appropriate for ROUGE-L recall (which doesn't use a brevity penalty).
  • E is Incorrect: Both metrics provide complementary information. ROUGE-L captures lexical coverage, and BERTScore captures semantic quality. Divergence between them is diagnostically useful for identifying data drift.

Key Takeaway: When lexical metrics (ROUGE) diverge from semantic metrics (BERTScore) during data drift, it often signals domain-specific terminology changes — retrain on new domain data to restore lexical alignment while maintaining semantic quality.

Question 10 - Evaluating AI-Generated Image Captioning with Semantic Similarity

A travel and tourism company has developed an image captioning system using AWS DeepComposer for music generation and Amazon Rekognition for image labeling, but the system's captioning component uses a vision-language model deployed via Amazon SageMaker AI. The system generates descriptive captions for hotel room images and vacation photos. The evaluation dataset includes 3,000 images with 5 human-generated reference captions per image.

The company wants an evaluation framework that combines two metrics to assess caption quality:

  • Metric 1: Measures semantic similarity between generated caption and reference captions using pre-trained sentence encoders.
  • Metric 2: Assesses whether the generated caption captures the key visual objects present in the image.

Given the requirement for a fully automated evaluation in the company's Amazon Bedrock evaluations pipeline, which evaluation approach best satisfies these requirements? (Select ONE)

  • A) Use ROUGE-L for Metric 1 (semantic similarity) and object-level accuracy via Amazon Rekognition for Metric 2 (object coverage)
  • B) Use BLEU-4 for Metric 1 and human evaluation for Metric 2 by content moderators
  • C) Use cosine similarity between sentence embeddings (e.g., using a CLIP-based text encoder or Amazon Titan Text Embeddings) for Metric 1, and object recall metrics from Amazon Rekognition label detection compared against ground-truth object annotations for Metric 2
  • D) Use BERTScore for Metric 1 and token count ratio between generated and reference captions for Metric 2
  • E) Use ROUGE-SU4 for Metric 1 and BLEU-4 for Metric 2 to measure object coverage
Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

  • C is Correct: Metric 1 (semantic similarity): Cosine similarity between sentence embeddings (CLIP-based or Amazon Titan Text Embeddings) captures meaning-level equivalence between generated and reference captions without requiring exact lexical match. Metric 2 (object coverage): Amazon Rekognition can detect object labels in images programmatically, and comparing these against ground-truth object annotations from the evaluation dataset provides automated object coverage measurement. Both metrics are fully automated and compatible with Bedrock evaluations.

Distractor Analysis

  • A is Incorrect: ROUGE-L is a lexical overlap metric, not a semantic similarity metric. While it measures word sequence overlap, it fails to capture meaning equivalence when captions use paraphrased or synonymous language. Object coverage via Rekognition is valid, but the ROUGE-L component makes this option incorrect.
  • B is Incorrect: BLEU-4 is an n-gram overlap metric designed for machine translation, not semantic similarity. Human evaluation by content moderators is not fully automated, violating the automation constraint.
  • D is Incorrect: Token count ratio is a simple length comparison metric that measures whether generated and reference captions have similar token counts, not object coverage. It does not assess which objects appear in the generated caption.
  • E is Incorrect: BLEU-4 measures n-gram overlap between generated and reference captions, not object coverage. It has no mechanism to assess whether the generated caption mentions the objects present in the image.

Key Takeaway: For image captioning evaluation, combine embedding-based semantic similarity (sentence encoder cosine similarity) with object-level recall (Rekognition label comparison) to measure both meaning alignment and visual content coverage — both fully automatable in Bedrock evaluations.