Skip to content

Foundation Model Automated Evaluation Metrics (AWS AIF-C01)

Evaluating Foundation Models (FMs) using automated metrics is a core objective tested under Task 3.4 (Applications of Foundation Models) in the AWS Certified AI Practitioner (AIF-C01) exam. You must know what each metric measures, its primary target task, and its specific exam triggers.

1. Metric Breakdown

ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

  • Core Concept: Measures Recall (how much of the ground-truth reference text was successfully captured in the generated output).
  • Target Task: Text Summarization.
  • Key Sub-types:
  • ROUGE-N: Measures exact matches of $N$-gram word sequences (e.g., ROUGE-1 for unigrams, ROUGE-2 for bigrams).
  • ROUGE-L: Measures the Longest Common Subsequence (LCS) between generated and reference text, preserving sentence-level word order.
  • Exam Trigger: "Evaluating an Amazon Bedrock foundation model performing document summarization..."

BLEU (Bilingual Evaluation Understudy)

  • Core Concept: Measures Precision (how many $N$-grams in the generated text match the reference) and applies a brevity penalty to prevent short, incomplete outputs from scoring artificially high.
  • Target Task: Machine Translation.
  • Exam Trigger: "Evaluating machine translation quality from Amazon Translate or an LLM against reference translations..."

BERTScore

  • Core Concept: Measures Semantic Similarity using contextual embeddings generated by a pre-trained transformer model (BERT) rather than relying on exact word/string matches. Computes cosine similarity between token embeddings.
  • Target Task: Any task where generated text uses synonyms, rephrasing, or different wording that carries the exact same meaning as the reference text.
  • Exam Trigger: "Measuring semantic equivalence when generated outputs do not match the exact n-gram wording of reference answers..."

Perplexity

  • Core Concept: Measures how well a probability model predicts the next token in a sequence. It represents the model's level of uncertainty/surprise when generating text.
  • Key Direction Rule: Lower is better (a lower perplexity score indicates higher confidence and better language modeling).
  • Exam Trigger: "Evaluating intrinsic language modeling quality and next-token prediction confidence on an un-tuned foundation model..."

2. Exam Decision Rules & Selection Matrix

Evaluation Metric Primary Focus Math / Mechanism Target Task Key Exam Trigger
ROUGE Recall $N$-gram overlap & Longest Common Subsequence Text Summarization "Summarization quality", "Recall of key facts"
BLEU Precision $N$-gram precision + Brevity Penalty Machine Translation "Translation quality", "Penalty for short text"
BERTScore Semantic Similarity Cosine similarity of contextual embeddings Paraphrased / Conceptual QA "Capturing synonyms / rephrased meaning"
Perplexity Model Predictability Exponentiated cross-entropy loss Intrinsic Language Modeling "Next-token prediction confidence (lower is better)"

3. High-Yield Exam Traps

  1. ROUGE vs. BLEU:
  2. Summarization $\rightarrow$ ROUGE (Recall matters most; you want the summary to capture all key facts).
  3. Translation $\rightarrow$ BLEU (Precision matters most; you want translated words to be accurate and non-redundant).
  4. Exact N-gram Matching vs. Meaning:
  5. If the question specifies that the generated answer conveys the same meaning as the reference but uses completely different words or synonyms, choose BERTScore.
  6. Perplexity Direction:
  7. If a scenario asks which model configuration performed best based on perplexity, always choose the option with the lowest perplexity value.