AIF-C01 Task 3.4: Methods to Evaluate Foundation Model Performance

Overview

Foundation model evaluation determines whether a model or an application:

Produces accurate and relevant results.
Follows instructions.
Performs consistently across different inputs.
Meets safety and quality requirements.
Provides value to users and the business.
Operates within acceptable latency and cost limits.

Evaluation should consider both:

Model-level performance
How well the foundation model performs a task.
Application-level performance
How well the complete solution performs, including prompts, retrieval, tools, workflows, user interfaces, and business processes.

A model with a high benchmark score may still be unsuitable for a specific business application.

1. Approaches for Evaluating FM Performance

1.1 Human-in-the-loop evaluation

Human-in-the-loop evaluation uses people to review and assess model responses.

Reviewers may evaluate:

Accuracy.
Relevance.
Completeness.
Clarity.
Helpfulness.
Tone.
Safety.
Bias.
Whether the response follows instructions.
Whether the response is grounded in provided information.

Reviewers may use:

Numerical ratings.
Pass or fail decisions.
Pairwise comparisons.
Rankings of multiple responses.
Written comments.
Labels for specific error types.

Example

A reviewer compares two model responses to the same customer question and selects the one that is more accurate and helpful.

Advantages

Can evaluate qualities that are difficult to measure automatically.
Can assess usefulness, tone, and safety.
Can incorporate domain expertise.
Useful for ambiguous or open-ended tasks.

Limitations

Can be expensive and time-consuming.
Reviewers may disagree.
Results may be subjective.
Review quality depends on reviewer training.
Reviewers may be influenced by formatting or writing style.
Privacy controls are required if reviewers see sensitive data.

Best practices

Create clear evaluation criteria.
Train reviewers.
Use multiple reviewers for important tasks.
Measure reviewer agreement.
Anonymize or minimize sensitive information.
Include domain experts for specialized use cases.
Record the reason for failure, not only a numerical score.

1.2 Benchmark datasets

A benchmark dataset is a standardized collection of inputs and expected outputs used to compare model performance.

Benchmark datasets may test:

General language understanding.
Reasoning.
Translation.
Summarization.
Question answering.
Coding.
Classification.
Safety behavior.

Advantages

Enables repeatable comparisons.
Can compare different models or model versions.
Provides standardized test cases.
Can reduce the effort needed to create an initial evaluation.

Limitations

May not reflect the organization’s real use case.
May not include proprietary terminology or data.
May become outdated.
A model may perform well on a benchmark but poorly in production.
Benchmark data may be present in a model’s training data, creating a risk of misleading results.

Exam tip

Use business-specific evaluation data in addition to public benchmarks. A benchmark is useful for comparison, but it is not proof that the model meets business requirements.

1.3 Amazon Bedrock Model Evaluation

Amazon Bedrock Model Evaluation helps compare foundation models or model configurations using evaluation datasets and selected metrics.

It can support:

Comparing different FMs.
Comparing model responses for a defined task.
Evaluating models using automatic evaluation.
Using human workers for evaluation where appropriate.
Assessing model performance before selecting a model for an application.

The exact evaluation features, supported models, datasets, and metrics can vary.

Typical evaluation process

Define the business task.
Select candidate models.
Prepare an evaluation dataset.
Select automatic or human evaluation.
Run the evaluation.
Compare the results.
Review qualitative examples and failure cases.
Select the model that best meets the overall requirements.

Important distinction

Amazon Bedrock Model Evaluation helps assess model performance. It does not automatically determine whether a model is suitable for every business, legal, security, or compliance requirement.

1.4 Offline evaluation

Offline evaluation tests a model or application before or outside live production traffic.

It uses:

Historical examples.
Curated test datasets.
Synthetic test cases.
Expert-created test cases.
Adversarial inputs.
Golden answers or expected behavior.

Benefits

Safe and repeatable.
Useful during development.
Does not affect live users.
Supports comparisons between prompts, models, or application versions.

Limitations

May not represent actual user behavior.
Cannot fully measure real-world engagement or business value.
May miss new types of inputs.

1.5 Online evaluation

Online evaluation measures performance using live or controlled production traffic.

Methods include:

A/B testing.
Canary deployments.
User feedback.
Production monitoring.
Human review of sampled interactions.
Business outcome analysis.

Benefits

Reflects real user behavior.
Measures actual user satisfaction and task completion.
Captures operational performance, latency, and cost.

Limitations

Poor results may affect real users.
Sensitive data requires careful handling.
Results can be influenced by changes in user population or workload.
A/B tests require suitable traffic and controls.

2. Metrics for FM Performance

No single metric fully describes model quality. Choose metrics based on the task.

2.1 ROUGE

Recall-Oriented Understudy for Gisting Evaluation, or ROUGE, compares generated text with one or more reference texts.

ROUGE is commonly used for:

Summarization.
Text generation tasks with reference answers.

It examines overlapping units such as:

Words.
Sequences of words.
Longest common subsequences.

General interpretation

A higher ROUGE score generally means greater overlap with the reference answer.

Limitations

Word overlap does not guarantee factual accuracy.
A correct answer using different words may receive a lower score.
A copied but incorrect answer may receive a higher score.
It may not measure style, usefulness, or safety.

Exam clue

If the question describes comparing a generated summary with a reference summary, consider ROUGE.

2.2 BLEU

Bilingual Evaluation Understudy, or BLEU, compares generated text with reference translations using token or phrase overlap.

BLEU is commonly used for:

Machine translation.
Text generation tasks where reference translations are available.

General interpretation

A higher BLEU score generally indicates greater similarity between the generated translation and the reference translation.

Limitations

It is based heavily on word or phrase overlap.
Multiple valid translations may receive lower scores.
It does not fully measure fluency, meaning, factuality, or usefulness.
It is less suitable for open-ended conversations.

Exam clue

If the question describes evaluating machine translation against reference translations, consider BLEU.

2.3 BERTScore

BERTScore evaluates semantic similarity between generated text and reference text using contextual embeddings from a language model.

It is useful when:

Different wording can express the same meaning.
Exact word overlap is not sufficient.
Semantic similarity is important.

Advantages

Captures meaning better than simple word-overlap metrics.
Can recognize semantically similar phrases with different wording.

Limitations

A semantically similar response may still contain factual errors.
It depends on the quality and behavior of the underlying language model.
It may not measure usefulness, safety, or business value.
It can be more computationally expensive than simple overlap metrics.

Exam clue

If the question emphasizes semantic similarity rather than exact wording, consider BERTScore.

2.4 LLM-as-a-judge

LLM-as-a-judge uses one language model to evaluate the output of another model.

The judge can score or compare outputs for:

Relevance.
Correctness.
Helpfulness.
Style.
Instruction following.
Safety.
Groundedness.

A judge prompt may ask the evaluating model to:

Assign a score.
Compare two answers.
Identify specific errors.
Explain why an answer failed.

Advantages

Scales evaluation more easily than human review.
Can evaluate open-ended responses.
Can use detailed evaluation criteria.
Faster and less expensive than reviewing every response manually.

Limitations

The judge can be wrong.
The judge may favor certain writing styles.
The judge may share biases with the evaluated model.
The judge may be sensitive to prompt wording.
It can be vulnerable to manipulated or adversarial outputs.
It should be calibrated against human reviewers.

Best practice

Use LLM-as-a-judge with:

Clear evaluation criteria.
Representative test data.
Periodic human validation.
Multiple evaluation methods.
Review of disagreements and unusual scores.

Exam trap

LLM-as-a-judge does not provide objective truth automatically. It is an evaluation aid and should not be treated as infallible.

2.5 Exact match

Exact match checks whether the generated answer exactly matches the expected answer.

It is useful for:

Fixed labels.
Short answers.
Structured fields.
Multiple-choice results.
Exact entity extraction.

Limitation

It can mark a correct answer as incorrect if the wording differs slightly.

2.6 Accuracy

Accuracy measures the percentage of predictions that are correct.

It is useful for:

Classification.
Multiple-choice tasks.
Discrete labels.

It is less informative for open-ended generation unless a reliable definition of correctness exists.

2.7 Precision and recall

These metrics are useful for classification and retrieval.

Precision measures how many selected results are relevant.
Recall measures how many relevant results were successfully found.

For a RAG system:

Retrieval precision asks whether the retrieved documents are relevant.
Retrieval recall asks whether the system found the important relevant documents.

Example

If a knowledge-base search returns ten documents and eight are relevant, retrieval precision is strong. If the system missed the key document, retrieval recall may still be poor.

2.8 Groundedness or faithfulness

Groundedness, also called faithfulness in some evaluation contexts, measures whether the response is supported by the supplied source material.

It is especially important for:

RAG.
Enterprise knowledge assistants.
Document question answering.
Compliance or policy assistants.

A response may be fluent and relevant but still not be grounded in the provided documents.

2.9 Answer relevance

Answer relevance measures whether the response addresses the user’s question.

A response can be:

Factually correct but irrelevant.
Relevant but factually incorrect.
Both relevant and correct.
Neither relevant nor correct.

Good evaluation should measure these dimensions separately where possible.

2.10 Toxicity and safety metrics

Safety evaluation may assess:

Harmful content.
Harassment.
Hate speech.
Sexual content.
Violent content.
Self-harm content.
Privacy violations.
Dangerous instructions.
Policy violations.

Safety metrics should be selected based on the application’s risk profile.

3. Choosing Metrics by Task

Task	Useful metrics or evaluation methods
Text summarization	ROUGE, BERTScore, human review, factuality, completeness
Machine translation	BLEU, BERTScore, human fluency and adequacy review
Classification	Accuracy, precision, recall, F1 score, confusion matrix
Question answering	Exact match, answer relevance, groundedness, human review
RAG	Retrieval precision, retrieval recall, context relevance, groundedness, citation accuracy
Conversational assistant	Human ratings, answer relevance, resolution rate, user satisfaction
Code generation	Test pass rate, static analysis, human review, security checks
Agent application	Task completion, tool-selection accuracy, action success, unsafe-action rate
Image generation	Human preference, prompt adherence, safety review, image quality
Safety behavior	Policy pass rate, refusal quality, harmful-output rate, adversarial testing

4. Determining Whether an FM Meets Business Objectives

Model quality is only one part of success. A model meets business objectives when the complete application produces useful business outcomes within acceptable constraints.

4.1 Define the task

First define exactly what the application must accomplish.

Examples:

Answer customer questions using approved documentation.
Reduce the time required to summarize support cases.
Classify incoming requests.
Help employees find internal policies.
Automate appointment scheduling.
Reduce the time required to generate draft reports.

A vague task makes evaluation difficult.

4.2 Define success criteria

Success criteria should be measurable.

Examples:

At least a defined percentage of support questions answered correctly.
A specified reduction in handling time.
A target task completion rate.
A maximum acceptable hallucination rate.
A maximum cost per interaction.
A maximum response latency.
A target user satisfaction score.
No critical privacy or safety violations.

4.3 Evaluate quality and business impact

Evaluate both:

Output quality: Is the response correct, relevant, safe, and well formatted?
Business impact: Does the system save time, reduce cost, increase productivity, or improve user experience?

A model can produce high-quality text without producing business value if it is too slow, too expensive, or difficult to integrate into the workflow.

4.4 Productivity metrics

Productivity metrics measure whether employees or business processes perform more efficiently.

Examples include:

Time saved per task.
Reduction in average handling time.
Number of cases processed per employee.
Throughput.
Reduction in repetitive work.
Reduction in manual data entry.
Time to produce a report.
Percentage of tasks completed with AI assistance.
Employee adoption rate.

Example

For an AI summarization tool, measure:

Time required to prepare a summary before and after deployment.
Percentage of summaries accepted without major edits.
Number of cases processed per day.
Reviewer-rated summary quality.

4.5 User engagement metrics

User engagement measures how users interact with the application.

Examples include:

Number of active users.
Repeat usage.
Session frequency.
Conversation length.
Feature adoption.
Retention.
User feedback volume.
Drop-off rate.
Percentage of users who return to the application.

Engagement alone does not prove that the application is effective. Users may interact frequently because they are confused or because the system does not solve their problem.

4.6 Task completion rate

Task completion rate measures the percentage of user tasks completed successfully.

Examples:

Percentage of support questions resolved without escalation.
Percentage of appointments scheduled successfully.
Percentage of forms completed correctly.
Percentage of orders processed without manual intervention.
Percentage of incidents triaged correctly.

For agentic applications, task completion should include whether the correct action was completed—not merely whether the agent generated a response.

4.7 User satisfaction

User satisfaction can be measured through:

Ratings.
Surveys.
Thumbs-up or thumbs-down feedback.
Customer effort score.
Net Promoter Score, where appropriate.
Qualitative comments.
Human support escalation rates.

User satisfaction should be considered together with objective quality metrics. Users may prefer a confident but inaccurate response, so satisfaction alone is insufficient.

4.8 Cost per interaction

Cost per interaction is the average cost of processing a user request.

It may include:

Input tokens.
Output tokens.
Embedding generation.
Vector database queries.
Agent tool calls.
Model customization or hosting.
Application infrastructure.
Logging and monitoring.
Human review.

A high-quality solution may still be unsuitable if its cost exceeds the business value generated.

4.9 Latency and availability

Operational metrics are important for user-facing applications.

Measure:

Average response time.
Percentile latency, such as the slowest typical responses.
Time spent retrieving context.
Time spent in tool calls.
Error rates.
Throttling.
Availability.
Timeout frequency.

For agents, measure the latency and failure rate of each step, not only the total response time.

5. Evaluating Applications Built with Foundation Models

5.1 RAG application evaluation

A RAG application should be evaluated in at least two stages:

Retrieval evaluation
Generation evaluation

Retrieval evaluation

Assess whether the application retrieves the right content.

Useful measures include:

Retrieval precision.
Retrieval recall.
Recall at a specified number of results.
Context relevance.
Search latency.
Citation or document-selection accuracy.
Metadata and access-control correctness.

Generation evaluation

Assess whether the model uses the retrieved content correctly.

Useful measures include:

Groundedness.
Faithfulness.
Answer relevance.
Completeness.
Factual accuracy.
Citation accuracy.
Unsupported-claim rate.
Human satisfaction.

Common RAG failure patterns

The correct document is not retrieved.
The wrong document is retrieved.
The retrieved document is outdated.
The model ignores the retrieved context.
Retrieved content contains conflicting information.
A user receives documents they are not authorized to access.
The response includes unsupported claims.

Exam tip

A RAG system can fail even when the foundation model is capable. Evaluate retrieval and generation separately.

5.2 Agent application evaluation

Agent evaluation should measure both reasoning and actions.

Important metrics include:

Task completion rate.
Tool-selection accuracy.
Correctness of tool parameters.
Action success rate.
Number of steps.
Unnecessary tool calls.
Failure recovery.
Time to completion.
Cost per completed task.
Human escalation rate.
Unauthorized-action rate.
Unsafe-action rate.
Quality of the final response.

Example

For an order-management agent, evaluate whether it:

Retrieved the correct order.
Verified the user’s permissions.
Selected the correct refund tool.
Supplied the correct order identifier.
Requested approval when required.
Completed the refund correctly.
Communicated the result accurately.

A fluent final answer does not prove that the agent performed the correct action.

5.3 Workflow evaluation

A workflow is a sequence of application steps that may include:

Prompt construction.
Retrieval.
Model invocation.
Content filtering.
Data transformation.
Tool calls.
Human approval.
Storage.
Notification.

Evaluate the end-to-end workflow for:

Overall task success.
Step-level failure rates.
Data accuracy.
Error handling.
Latency.
Cost.
Reliability.
Security.
Auditability.
Human handoff quality.

Example

An invoice-processing workflow should be evaluated for:

Correct document extraction.
Correct validation.
Correct database update.
Duplicate detection.
Exception handling.
Human escalation.
Final processing time.
Cost per invoice.

5.4 Prompt and model comparison

When comparing prompts or models, hold other factors constant where possible:

Same evaluation dataset.
Same task definition.
Same output requirements.
Same retrieval data.
Same inference parameters.
Same scoring criteria.

Record:

Model version.
Prompt version.
Inference parameters.
Retrieved context.
Tool configuration.
Evaluation results.

This allows the organization to reproduce results and identify what caused a performance change.

6. Offline and Online Evaluation Strategy

A strong evaluation program commonly uses multiple stages.

Stage 1: Development evaluation

Use a small test set to identify prompt and configuration problems.

Stage 2: Controlled evaluation

Use a representative benchmark or business-specific dataset to compare models and prompts.

Stage 3: Human review

Have qualified reviewers assess quality, safety, relevance, and edge cases.

Stage 4: Limited deployment

Use a pilot, canary, or limited user group.

Stage 5: Online monitoring

Monitor quality, business outcomes, cost, latency, safety, and user feedback.

Stage 6: Continuous improvement

Update:

Test datasets.
Prompts.
Retrieval logic.
Models.
Evaluation criteria.
Guardrails.
Training data.

7. Evaluation Metrics and Their Limitations

Metric	Measures	Best suited for	Important limitation
ROUGE	Word or phrase overlap with reference text	Summarization	Does not guarantee factual correctness
BLEU	Overlap with reference translations	Machine translation	May penalize valid alternative wording
BERTScore	Semantic similarity using contextual embeddings	Summarization and generation	Similar meaning does not guarantee factual accuracy
Accuracy	Percentage of correct predictions	Classification	Less useful for open-ended text
Precision	Percentage of selected results that are relevant	Classification and retrieval	Does not measure missed relevant results
Recall	Percentage of relevant results found	Classification and retrieval	Does not measure how many irrelevant results were returned
Exact match	Exact equality with expected output	Fixed labels and structured answers	Penalizes valid wording variations
LLM-as-a-judge	Model-based assessment of outputs	Open-ended generation	Can be biased or inconsistent
Human evaluation	Human judgment of quality and safety	Complex or subjective tasks	Expensive and potentially subjective
Groundedness	Support from supplied context	RAG	Does not independently prove that the source is correct
Task completion rate	Successful completion of user tasks	Applications and agents	Requires a clear definition of success
User satisfaction	User perception of value and quality	User-facing applications	Users may prefer inaccurate but confident responses
Cost per interaction	Average cost per request or task	Production applications	Lower cost does not necessarily mean better quality

8. Exam Tips and Common Traps

Tip 1: Match metrics to task type

Summarization → ROUGE, BERTScore, human review.
Translation → BLEU, BERTScore, human review.
Classification → Accuracy, precision, recall, or F1.
RAG → Retrieval recall, groundedness, answer relevance.
Agents → Task completion, tool-call accuracy, unsafe-action rate.
Production business value → Productivity, satisfaction, cost, and completion rate.

Tip 2: No single metric is sufficient

A high ROUGE, BLEU, or BERTScore result does not guarantee that an answer is:

Correct.
Safe.
Useful.
Grounded.
Suitable for the business.

Use multiple metrics and human review where appropriate.

Tip 3: LLM-as-a-judge is not a human replacement

An LLM can scale evaluation, but it may introduce its own biases and errors. Validate it against human judgments.

Tip 4: Separate model evaluation from application evaluation

A model can perform well in isolation while the application fails because of:

Poor prompting.
Incorrect retrieval.
Faulty tool calls.
Bad data preparation.
Weak access control.
Workflow errors.
Excessive cost or latency.

Tip 5: Evaluate RAG retrieval separately

If the correct answer is not retrieved, the generation model may not have the information needed to answer correctly.

Tip 6: Agents must be evaluated on actions

For agents, evaluate whether the correct tool and parameters were used and whether the intended action was successfully completed.

Tip 7: User satisfaction is not the same as accuracy

Users may rate an answer highly because it sounds confident or helpful. Combine satisfaction with correctness, groundedness, and safety metrics.

Tip 8: Business objectives should be measurable

“Improve productivity” is not sufficient by itself. Define measurable indicators such as:

Time saved.
Tasks completed.
Cost per interaction.
Error reduction.
Resolution rate.
User satisfaction.

Tip 9: Include production constraints

Evaluate:

Latency.
Availability.
Throttling.
Cost.
Scaling behavior.
Human review requirements.

Tip 10: Use representative and adversarial test cases

Include typical, difficult, ambiguous, unsafe, and malicious inputs. An application that performs well only on ideal inputs is not production-ready.

Practice Questions and Answers

## Question 1 A company wants to compare several foundation models using a standardized set of question-answering examples before selecting one for an application. Which evaluation approach should it use? A. Benchmark dataset evaluation B. Increasing the model temperature C. Prompt injection testing only D. Data deletion **Answer: A. Benchmark dataset evaluation** **Explanation:** A benchmark dataset provides repeatable test cases for comparing models. The company should also use business-specific data and human review before making a final decision. --- ## Question 2 A company wants human reviewers to compare model responses for helpfulness, correctness, and safety. Which approach is being used? A. Human-in-the-loop evaluation B. Vector indexing C. Continuous pre-training D. Prompt caching **Answer: A. Human-in-the-loop evaluation** **Explanation:** Human-in-the-loop evaluation uses human reviewers to assess model outputs according to defined criteria. --- ## Question 3 Which metric is most commonly associated with evaluating machine translation against reference translations? A. BLEU B. ROUGE C. Recall at retrieval rank D. Cost per interaction **Answer: A. BLEU** **Explanation:** BLEU compares generated translations with reference translations using token or phrase overlap. --- ## Question 4 Which metric is commonly used to evaluate generated summaries by comparing their word or phrase overlap with reference summaries? A. BLEU B. ROUGE C. Task completion rate D. Latency **Answer: B. ROUGE** **Explanation:** ROUGE is commonly used for summarization and measures overlap between generated and reference text. --- ## Question 5 A model produces a summary using different words from the reference summary but expresses a similar meaning. Which metric is better suited to measure semantic similarity? A. BERTScore B. Exact match only C. Cost per interaction D. Tool-selection accuracy **Answer: A. BERTScore** **Explanation:** BERTScore uses contextual embeddings to evaluate semantic similarity and can recognize similar meaning despite different wording. --- ## Question 6 A company uses an LLM to score another model’s responses for relevance and helpfulness. What is an important limitation of this approach? A. It cannot evaluate text output. B. It always produces objective results. C. The evaluating LLM may have biases or disagree with human reviewers. D. It requires no evaluation criteria. **Answer: C. The evaluating LLM may have biases or disagree with human reviewers.** **Explanation:** LLM-as-a-judge can scale evaluation, but its results should be calibrated and periodically compared with human judgments. --- ## Question 7 A RAG application often generates incorrect answers even though the foundation model performs well on general benchmarks. Which evaluation should be performed first to identify whether the correct documents are being retrieved? A. Retrieval precision and recall B. BLEU only C. User retention only D. Model size comparison only **Answer: A. Retrieval precision and recall** **Explanation:** RAG can fail because the relevant documents are not retrieved. Retrieval precision and recall help evaluate the retrieval stage separately from response generation. --- ## Question 8 An AI agent successfully generates fluent responses but frequently selects the wrong API or supplies incorrect parameters. Which metrics are most relevant? A. Tool-selection accuracy and action success rate B. ROUGE and BLEU only C. Image resolution and color accuracy D. User interface load time only **Answer: A. Tool-selection accuracy and action success rate** **Explanation:** Agents must be evaluated on their decisions and actions, not only on the quality of the final text. --- ## Question 9 A company wants to determine whether an AI assistant reduces the time employees spend resolving support tickets. Which metric is most relevant? A. ROUGE score B. BLEU score C. Time saved or average handling time D. Embedding dimension **Answer: C. Time saved or average handling time** **Explanation:** Productivity and business-impact metrics should measure whether the application achieves the intended business objective. --- ## Question 10 Which metric is most directly related to the average expense of processing one user request? A. User satisfaction B. Cost per interaction C. Retrieval recall D. BERTScore **Answer: B. Cost per interaction** **Explanation:** Cost per interaction measures the average cost of processing a request, including model usage and potentially retrieval, tool calls, infrastructure, and human review. --- ## Question 11 A company wants to determine whether an AI application actually completes the customer’s requested task rather than merely producing a plausible response. Which metric should it prioritize? A. Task completion rate B. BLEU C. Prompt length D. Model parameter count **Answer: A. Task completion rate** **Explanation:** Task completion rate measures whether the user’s intended task was successfully completed. This is particularly important for agents and workflows. --- ## Question 12 Which evaluation strategy provides the most complete view of an AI application’s performance? A. Use only a public benchmark score. B. Use only user satisfaction surveys. C. Combine automated metrics, human evaluation, business metrics, and production monitoring. D. Evaluate only the model’s response length. **Answer: C. Combine automated metrics, human evaluation, business metrics, and production monitoring.** **Explanation:** A complete evaluation considers model quality, application behavior, business value, safety, cost, latency, and real-world performance.

Final Summary

Remember these key relationships:

Human-in-the-loop evaluation is useful for quality, safety, tone, and subjective judgments.
Benchmark datasets enable repeatable model comparisons but may not reflect the business use case.
Amazon Bedrock Model Evaluation helps compare supported models and configurations using evaluation datasets and automatic or human evaluation options.
ROUGE is commonly associated with summarization.
BLEU is commonly associated with machine translation.
BERTScore measures semantic similarity.
LLM-as-a-judge scales evaluation but must be validated against human judgment.
RAG applications require separate evaluation of retrieval and generation.
Agents must be evaluated on tool selection, action success, safety, and task completion.
Business alignment requires metrics such as productivity, user satisfaction, task completion rate, latency, and cost per interaction.
The best evaluation strategy combines automated metrics, human review, representative test data, adversarial testing, and production monitoring.