AWS AI Practitioner Exam Notes

Task Statement 3.4: Describe Methods to Evaluate Foundation Model Performance

1. Why Foundation Model Evaluation Is Different

Traditional machine learning models often produce deterministic outputs that can be directly compared with labeled answers.

Examples:

Classification: Compare predicted class with actual class.
Regression: Calculate RMSE or MAE.
Binary prediction: Calculate accuracy, precision, recall, or F1 score.

Foundation models and generative AI applications are more difficult to evaluate because:

The output can be non-deterministic.
Multiple answers may be valid.
A response can be fluent but factually incorrect.
Exact string matching is often inappropriate.
Evaluation depends heavily on the task.
A model can perform well on a benchmark but poorly on a business-specific use case.
Safety, bias, toxicity, latency, cost, and user satisfaction may be just as important as answer quality.

Therefore, foundation model evaluation usually combines:

Automated metrics
Benchmark datasets
Human evaluation
LLM-as-a-judge evaluation
Business and application-level metrics

2. Main Approaches to Evaluating Foundation Models

2.1 Human-in-the-Loop Evaluation

Human evaluation involves having people assess model responses according to a defined rubric.

Humans may evaluate:

Relevance
Helpfulness
Factual accuracy
Completeness
Fluency
Relevance to the prompt
Safety
Toxicity
Bias
Whether the response follows instructions
Whether a response is better than another model’s response

Common human evaluation methods

Rating-based evaluation

Human reviewers assign a score, such as:

1–5 quality score
Pass/fail
Correct/incorrect
Safe/unsafe
Relevant/not relevant

Pairwise comparison

Reviewers compare two responses and select:

Response A is better
Response B is better
Both are equivalent
Neither is acceptable

Pairwise comparison is often easier for humans than assigning an absolute score.

Ranking

Reviewers rank multiple model responses from best to worst.

Advantages

Can evaluate qualities that automated metrics cannot easily measure.
Useful for subjective tasks such as writing quality, helpfulness, tone, and user experience.
Can assess business-specific requirements.
Can identify subtle hallucinations, bias, and safety issues.

Limitations

More expensive and slower than automated evaluation.
Human reviewers may disagree.
Results can be affected by reviewer expertise, cultural background, or bias.
A clear evaluation rubric is required.
Reviewers may be influenced by response length or writing style.

Exam tip

Human evaluation is generally the best choice when the desired quality is subjective or difficult to represent with a reference answer, such as:

Helpfulness
Tone
Creativity
Safety
Business usefulness
User experience

2.2 Benchmark Datasets

Benchmark datasets are standardized datasets and tasks used to compare model performance.

They are useful for:

Comparing multiple foundation models.
Measuring general language understanding.
Measuring reasoning and domain knowledge.
Tracking model improvements over time.
Comparing results with other published models.

Important benchmarks

GLUE

General Language Understanding Evaluation

GLUE is a collection of natural language understanding tasks, such as:

Sentiment analysis
Textual entailment
Similarity
Question answering
Classification

GLUE is designed to test whether models can generalize across multiple language tasks.

SuperGLUE

SuperGLUE is a more difficult successor to GLUE.

It includes more challenging tasks, such as:

Reading comprehension
Multi-sentence reasoning
Inference
Coreference and language understanding

MMLU

Massive Multitask Language Understanding

MMLU evaluates broad knowledge and problem-solving ability across many subjects.

Examples include:

History
Mathematics
Law
Computer science
Medicine
Economics
Physics

MMLU is intended to test more than basic language fluency. It tests whether a model has broad world knowledge and reasoning ability.

BIG-bench

Beyond the Imitation Game Benchmark

BIG-bench contains a broad collection of challenging tasks intended to test capabilities beyond basic language imitation.

Tasks can cover:

Mathematics
Biology
Physics
Reasoning
Linguistics
Bias
Software development
Childhood development
Social reasoning

HELM

Holistic Evaluation of Language Models

HELM is designed to provide a more comprehensive and transparent evaluation of language models.

It considers multiple scenarios and metrics, including:

Question answering
Summarization
Sentiment analysis
Bias
Toxicity
Robustness
Efficiency
Accuracy

HELM is useful when evaluating models across several dimensions rather than using only one score.

Benchmark limitations

Benchmark results should not be treated as proof that a model is the best choice for every application.

Potential limitations include:

The benchmark may not represent the organization’s actual data.
The benchmark may not measure latency or cost.
The model may have seen similar benchmark data during training.
High benchmark performance does not guarantee business success.
A model may perform well generally but fail on specialized company terminology.
Benchmark scores may not measure safety, factuality, or user satisfaction adequately.

Exam tip

Use benchmarks to compare general model capabilities. Use business-specific test sets and application metrics to determine whether a model is suitable for a real use case.

2.3 Amazon Bedrock Model Evaluation

Amazon Bedrock provides model evaluation capabilities for comparing foundation models and assessing generated responses.

Model evaluation can help assess:

Model quality
Accuracy
Robustness
Toxicity
Relevance
Semantic similarity
Other task-specific qualities

Amazon Bedrock supports evaluation using:

Automatic model evaluation
Human-based model evaluation
Built-in datasets
Custom prompt datasets
Human reference answers, where applicable

Automatic evaluation

Automatic evaluation is useful when a large number of model outputs must be evaluated consistently.

A typical process is:

Select the foundation model or models.
Provide prompts or an evaluation dataset.
Generate model responses.
Compare responses using selected metrics.
Review the evaluation results.

Human-based evaluation

Human workers can compare responses from:

Different Amazon Bedrock models
Models hosted outside AWS
Different prompts
Different versions of an application

This is useful when the evaluation requires human judgment.

Key exam point

Amazon Bedrock model evaluation is a managed AWS capability for evaluating and comparing foundation models. It is not limited to manually reviewing one response at a time.

2.4 Amazon SageMaker Clarify and Model Evaluation Jobs

Amazon SageMaker Clarify can be used to evaluate machine learning models, including large language models and text-based foundation models available through SageMaker JumpStart.

A model evaluation job can help:

Evaluate model quality.
Compare models.
Calculate selected metrics.
Evaluate text-generation behavior.
Use human workers for response comparisons.
Include models outside AWS in comparisons, depending on the evaluation setup.

Exam distinction

Amazon Bedrock model evaluation: Used for evaluating foundation models accessed through Amazon Bedrock.
SageMaker Clarify model evaluation: Used for model evaluation in the SageMaker environment, including text-based models and SageMaker JumpStart models.

Do not assume that all model evaluation features are provided by the same AWS service.

3. Important Foundation Model Evaluation Metrics

3.1 ROUGE

ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation.

ROUGE is commonly used for:

Text summarization
Comparing generated text with reference summaries
Some text-generation and translation evaluation scenarios

ROUGE measures the overlap between generated text and a reference text.

Common forms include:

ROUGE-1: Unigram overlap
ROUGE-2: Bigram overlap
ROUGE-L: Longest common subsequence

Interpretation

A higher ROUGE score generally means that the generated response contains more words or sequences that overlap with the reference answer.

Strengths

Useful for summarization.
Easy to calculate.
Useful when reference answers are available.

Limitations

May penalize valid answers that use different wording.
Word overlap does not guarantee factual correctness.
Does not fully measure readability or usefulness.
A response can have high ROUGE but still contain hallucinations.

Exam tip

ROUGE is most strongly associated with summarization and recall-oriented overlap with a reference answer.

3.2 BLEU

BLEU stands for Bilingual Evaluation Understudy.

BLEU is commonly used to evaluate:

Machine translation
Generated text compared with one or more reference translations

BLEU primarily measures n-gram precision: how much of the generated text matches the reference text.

Interpretation

A higher BLEU score generally indicates greater similarity between generated and reference translations.

Strengths

Widely used for machine translation.
Efficient and inexpensive to calculate.
Can compare generated translations against reference translations.

Limitations

May not recognize valid synonyms or alternative sentence structures.
Does not directly measure factuality.
Can favor literal translation.
May not reflect human judgment perfectly.
Requires reference translations.

Exam tip

BLEU is most strongly associated with machine translation.

3.3 BERTScore

BERTScore evaluates generated text by comparing contextual embeddings rather than only exact word overlap.

It uses a language model such as BERT to measure semantic similarity between:

A generated response
A reference response

BERTScore is useful when two responses have similar meanings but use different words.

Strengths

Captures semantic similarity better than exact word matching.
More tolerant of synonyms and paraphrasing.
Often more appropriate for natural language generation than exact-match metrics.

Limitations

Semantic similarity does not necessarily mean factual correctness.
It may reward a response that is semantically similar but includes an important error.
Results depend on the underlying embedding model.
It does not replace human review for subjective or safety-sensitive tasks.

Exam tip

BERTScore is based on semantic similarity using contextual embeddings, unlike BLEU and ROUGE, which primarily rely on word or n-gram overlap.

3.4 LLM-as-a-Judge

LLM-as-a-judge uses one language model to evaluate the output of another model.

The judging model may score a response on:

Correctness
Relevance
Helpfulness
Coherence
Groundedness
Instruction following
Style
Safety

The judge may:

Assign a numeric score.
Select the better of two responses.
Determine whether a response satisfies a rubric.
Provide a written explanation.

Advantages

More scalable than human evaluation.
Can evaluate open-ended answers.
Can use a detailed evaluation rubric.
Can assess qualities that simple lexical metrics cannot measure.

Limitations and risks

The judge can be biased.
The judge may favor longer or more confident answers.
The judge may have difficulty detecting subtle factual errors.
The judge may share similar weaknesses with the model being evaluated.
Scores may vary depending on judge model, prompt, and temperature.
A judge should be calibrated against human reviewers.

Best practice

Use LLM-as-a-judge together with:

Human review
Objective checks
Reference answers
Ground-truth validation
Safety and toxicity filters

Exam tip

LLM-as-a-judge is an evaluation method, not a guarantee of objective truth. It is scalable but can introduce evaluator bias.

4. Evaluating Factuality, Faithfulness, and Hallucinations

These terms are related but distinct.

Factuality

Whether the generated answer is factually correct.

Example:

“The capital of France is Paris.”

This is factually correct.

Faithfulness or groundedness

Whether the answer is supported by the supplied source or context.

For example, in a RAG application, a response may be factually true in the real world but not supported by the retrieved documents. It may therefore fail a strict groundedness requirement.

Hallucination

A response that contains unsupported, fabricated, or incorrect information.

A model can produce:

Fluent language
Good grammar
High semantic similarity
High user appeal

and still hallucinate.

Important exam trap

High ROUGE, BLEU, or BERTScore does not automatically prove that a response is factually correct or free of hallucinations.

5. Evaluating RAG Applications

RAG applications have multiple stages and should not be evaluated only by looking at the final answer.

A typical RAG flow is:

User submits a query.
The system creates an embedding for the query.
A search system retrieves relevant documents or chunks.
The retrieved context is inserted into a prompt.
The foundation model generates a response.
The application may provide citations or take an action.

Each stage can introduce errors.

5.1 Retrieval evaluation

Evaluate whether the correct information was retrieved.

Relevant metrics include:

Recall@K: Whether relevant documents appear in the top K results.
Precision@K: How many of the top K results are relevant.
Mean Reciprocal Rank: How high the first relevant result appears.
Normalized Discounted Cumulative Gain: Quality of ranking when multiple results have different relevance.
Context relevance: Whether retrieved passages relate to the question.
Context completeness: Whether the retrieved passages contain enough information.

Example

If the correct document is ranked 20th but the application only passes the top five documents to the model, the generation step may fail even if the correct document exists in the knowledge base.

5.2 Generation evaluation

Evaluate whether the final answer:

Uses the retrieved context.
Answers the user’s question.
Is factually correct.
Avoids unsupported claims.
Includes appropriate citations.
Follows the required format.
Does not reveal sensitive information.

Useful metrics include:

Answer correctness
Answer relevance
Faithfulness
Groundedness
Citation accuracy
Hallucination rate
Refusal accuracy
Human or LLM-judge ratings

5.3 End-to-end RAG evaluation

Evaluate the entire user experience:

Task completion rate
User satisfaction
Time to answer
Number of turns required
Escalation rate
Cost per interaction
Retrieval latency
Generation latency
Percentage of answers requiring human correction

Exam tip

For RAG, evaluate both:

Retrieval quality
Final generated answer quality

A poor final response may be caused by incorrect retrieval, insufficient context, prompt design, or model generation.

6. Evaluating Agents and Workflows

Foundation model applications may use agents that:

Select tools.
Call APIs.
Query databases.
Execute multi-step workflows.
Plan tasks.
Validate results.
Perform actions in external systems.

Evaluating an agent requires more than evaluating the text of its final answer.

Agent evaluation areas

Planning quality

Did the agent create a suitable plan?

Tool selection

Did it choose the correct tool?

Tool-call accuracy

Did it use the correct parameters and input format?

Workflow completion

Did it complete all required steps?

Task success

Did the agent achieve the user’s intended outcome?

Error recovery

Can the agent recover from:

Tool failures
Invalid inputs
Missing information
Timeouts
API errors

Safety and authorization

Did the agent:

Avoid unauthorized actions?
Request confirmation for high-impact actions?
Protect private information?
Follow access-control requirements?

Efficiency

Did it:

Use an unnecessary number of tool calls?
Repeat failed actions?
Consume excessive tokens?
Take too long?
Exceed the cost budget?

Metrics for agents

Task completion rate
Successful tool-call rate
Invalid tool-call rate
Number of steps per task
Number of tool calls
Error recovery rate
Human escalation rate
Average latency
Cost per completed task
Unauthorized action rate
User satisfaction

Exam trap

An agent that generates a correct-looking final response may still have failed if it did not actually complete the requested action.

7. Evaluating Workflows and FM-Based Applications

The evaluation target may be:

The foundation model alone
A prompt
A RAG pipeline
An agent
A complete application
A human-AI workflow

Model-level evaluation

Measures the capability of the model independently of the application.

Examples:

MMLU score
BLEU
ROUGE
BERTScore
Toxicity
Bias
Accuracy on a test set

Application-level evaluation

Measures whether the complete system solves the intended business problem.

Examples:

Customer support resolution rate
Correct document retrieval rate
Percentage of invoices processed without manual intervention
Time saved per employee
Number of successful bookings
Percentage of cases escalated to humans

Key principle

A high-performing foundation model does not guarantee a high-performing application.

Application performance also depends on:

Prompt design
Retrieval configuration
Data quality
Chunking strategy
Embedding model
Tool integration
User interface
Workflow design
Guardrails
Latency
Cost
Security controls

8. Determining Whether the Model Meets Business Objectives

Before evaluating a foundation model, define the business objective.

Examples:

Improve employee productivity.
Increase customer engagement.
Reduce customer support handling time.
Improve search quality.
Automate document processing.
Increase sales conversion.
Reduce operating cost.
Improve response speed.
Reduce the number of human escalations.

Step 1: Define the problem

A vague objective is difficult to evaluate.

Weak objective:

“Use generative AI to improve customer service.”

Better objective:

“Reduce the average time required to resolve customer support cases while maintaining or improving customer satisfaction.”

Step 2: Define success metrics

Metrics should be measurable and tied directly to the business objective.

For productivity:

Time saved per task
Tasks completed per employee
Output per employee
Reduction in manual effort
Percentage of work automated
Time to draft or review a document

For user engagement:

Daily or monthly active users
Session duration
Return-user rate
Number of completed interactions
Click-through rate
Conversation completion rate
User retention

For customer service:

First-contact resolution rate
Average handling time
Escalation rate
Customer satisfaction score
Net Promoter Score
Percentage of answers requiring correction

For document or workflow automation:

Task completion rate
Straight-through processing rate
Extraction accuracy
Exception rate
Human review rate
Processing time

For cost control:

Cost per interaction
Cost per completed task
Token cost
Infrastructure cost
Cost of human review
Cost savings compared with the existing process

Step 3: Establish a baseline

Compare the AI-enabled process with:

The existing manual process.
The previous model.
A smaller or less expensive model.
A rules-based system.
A human-only workflow.

Without a baseline, it is difficult to determine whether the AI system actually improved performance.

Step 4: Monitor after deployment

Evaluation should continue after launch.

Monitor:

Quality drift
Data drift
Changes in user behavior
Hallucination rates
Safety incidents
Latency
Cost
User feedback
Task success
Model and application failures

Exam tip

Evaluation is not a one-time activity. The application should be measured, monitored, and reviewed over time.

9. Performance and Inference Trade-Offs

Model performance is not only about accuracy or quality.

Deployment also requires consideration of:

Inference latency
Compute requirements
Storage requirements
Throughput
Cost
Availability
Scalability
Real-time versus batch processing

Smaller models

Reducing model size may:

Reduce memory requirements.
Reduce storage needs.
Reduce loading time.
Improve inference speed.
Reduce cost.

However, a smaller model may:

Produce lower-quality answers.
Have weaker reasoning ability.
Have less domain knowledge.
Be less capable of following complex instructions.

Prompt optimization

A more concise prompt can reduce:

Input token count.
Latency.
Cost.
Context-window usage.

However, removing necessary instructions or context can reduce quality.

Retrieved context optimization

Reducing the number or size of retrieved snippets can:

Reduce latency.
Reduce token usage.
Lower cost.
Reduce distraction from irrelevant information.

But retrieving too little context can:

Reduce answer accuracy.
Increase hallucinations.
Cause the model to miss important information.

Generation parameter optimization

Parameters such as:

Maximum output tokens
Temperature
Top-p
Stop sequences

can affect:

Output length
Creativity
Consistency
Latency
Cost

Exam trap

Optimizing for speed or cost can reduce model quality. The correct solution is usually a trade-off based on business requirements, not simply choosing the fastest or largest model.

10. Additional Data and RAG Considerations

Foundation models may have:

Outdated training knowledge.
No access to private organizational data.
Difficulty with current events.
Difficulty with precise calculations.
No direct access to enterprise applications.

RAG allows the model to access external data at inference time.

Benefits of RAG

Uses current information without retraining the model.
Can access private enterprise data.
Can improve factual grounding.
Can reduce hallucinations.
Can provide source citations.
Avoids the recurring cost of retraining whenever documents change.

RAG trade-offs

Retrieval quality affects answer quality.
Poor chunking can cause missing context.
Irrelevant documents can confuse the model.
The vector store and retrieval layer add latency and cost.
Access permissions must be enforced.
Retrieved data may contain sensitive information.
RAG does not guarantee factuality.

Important exam trap

RAG is not the same as fine-tuning.

RAG supplies external information during inference.
Fine-tuning changes model behavior by training on additional examples.

RAG is often preferable when information changes frequently or must remain outside the model’s parameters.

11. Application Architecture Factors That Affect Evaluation

An FM application commonly includes several layers.

Infrastructure layer

Provides:

Compute
Storage
Networking
Model hosting
Scalability
Monitoring

Consider:

Latency requirements
Capacity
Availability
Security
Encryption
Isolation
Cost

Model layer

Includes:

Foundation model selection
Model size
Inference configuration
Fine-tuning or customization
Prompt templates

Data and retrieval layer

May include:

Document repositories
Embedding models
Vector databases
Search services
Data pipelines
Access controls

Orchestration layer

Coordinates:

User inputs
Prompts
Model calls
Retrieval
Tool calls
Memory
Workflow steps
Output processing

Application interface layer

Examples include:

Website
Chat interface
Mobile application
REST API
Internal business application

Each layer can affect the final business metric. For example, a good model may appear ineffective if the user interface is confusing or the retrieval system returns poor results.

12. Common Metric Selection Guide

Evaluation need	Useful approach or metric
Summarization overlap	ROUGE
Machine translation	BLEU
Semantic similarity	BERTScore
Broad language understanding	GLUE or SuperGLUE
Broad knowledge and reasoning	MMLU
Broad and challenging capabilities	BIG-bench
Holistic model comparison	HELM
Subjective quality	Human evaluation
Scalable rubric-based evaluation	LLM-as-a-judge
RAG document retrieval	Recall@K, Precision@K, MRR, NDCG
RAG factual grounding	Faithfulness, groundedness, citation accuracy
Agent effectiveness	Task completion rate, tool-call success
Business value	Productivity, satisfaction, cost per interaction
Production operation	Latency, throughput, availability, token usage

13. Exam Tips and Traps

Tip 1: Match the metric to the task

ROUGE → Summarization
BLEU → Machine translation
BERTScore → Semantic similarity
MMLU → Knowledge and problem solving
GLUE → General language understanding
HELM → Holistic and transparent evaluation

Tip 2: Do not confuse similarity with correctness

A high ROUGE, BLEU, or BERTScore result does not necessarily mean:

The answer is factually correct.
The answer is grounded.
The answer is safe.
The answer meets business requirements.

Tip 3: Do not evaluate only the foundation model

For RAG, agents, and workflows, evaluate the complete system.

The final result depends on:

Retrieval
Prompting
Tool calls
Orchestration
User interface
Data quality
Model quality

Tip 4: Business metrics are different from model metrics

A model score is not the same as business value.

Examples:

BERTScore measures semantic similarity.
Task completion rate measures whether the user’s task was completed.
Cost per interaction measures economic efficiency.
User satisfaction measures perceived usefulness.

Tip 5: Human review remains important

Human evaluation is especially useful for:

Open-ended responses
Helpfulness
Tone
Safety
Bias
Business usefulness
Responses without a single correct answer

Tip 6: LLM-as-a-judge is scalable but imperfect

It can reduce evaluation effort, but it may have:

Bias
Inconsistent scoring
Difficulty detecting subtle errors
Preference for verbose answers

Use it with calibration and human review.

Tip 7: RAG improves grounding but does not eliminate hallucinations

RAG can provide current and relevant context, but failures can still occur when:

The wrong documents are retrieved.
The context is incomplete.
The model ignores the context.
The source data is incorrect.
The prompt does not require grounded answers.

Tip 8: Optimize with trade-offs in mind

Smaller models, shorter prompts, fewer retrieved passages, and shorter outputs may reduce cost and latency but also reduce quality.

Tip 9: Establish a baseline

To determine whether an AI system is successful, compare it with:

The current process.
A human workflow.
Another model.
A previous application version.

Tip 10: Monitor continuously

Model and application evaluation should continue after deployment because:

Data changes.
User behavior changes.
Documents become outdated.
Model behavior may drift.
Costs and latency may change.
New safety issues may appear.

14. Quick Review Summary

Foundation model outputs are non-deterministic, so evaluation is more complex than standard ML evaluation.
Use automated metrics, benchmarks, human review, and LLM-as-a-judge together.
ROUGE is commonly used for summarization.
BLEU is commonly used for machine translation.
BERTScore measures semantic similarity using contextual embeddings.
GLUE and SuperGLUE evaluate language understanding.
MMLU evaluates broad knowledge and problem solving.
BIG-bench evaluates challenging and diverse capabilities.
HELM provides holistic and transparent evaluation across multiple dimensions.
Amazon Bedrock supports foundation model evaluation using automatic and human-based approaches.
SageMaker Clarify supports model evaluation jobs, including text-based foundation models.
RAG must be evaluated at both the retrieval and generation stages.
Agents must be evaluated on planning, tool use, task completion, safety, and efficiency.
Business success requires metrics such as productivity, user satisfaction, task completion rate, latency, and cost per interaction.
A high model benchmark score does not guarantee business success.
Evaluation should include a baseline and continue throughout the application lifecycle.