Skill 2.3.9: Perform AI Evaluation

Task 2.3: ML Model and Foundation Model (FM) Development — Analyze and evaluate the performance of ML and AI systems

Core objective: Turn a trained artifact or generated response into a defensible, reproducible quality signal — and keep evaluation ("how good is the output") separate from explanation ("why did this output happen"). Three different questions require three different services: what happened in the experiment, how a candidate behaves on live traffic, and how good the output itself is. Reaching for the wrong one produces a number nobody can act on.

1. Evaluation Modes — Decision Matrix

Mode	Scorer	Strengths	Constraints & Traps
Programmatic (automatic)	Deterministic code	Cheap, fast, repeatable, no model access needed	Only works where a reference exists; blind to paraphrase and tone
LLM-as-a-judge	Judge FM from a supported list	Scales human-like judgment; returns score + rationale; custom metrics via prompt + rating scale	Judge model access must be enabled in-region; scores non-deterministic → pin judge model/version; judge inherits its own bias
Human evaluation	Employees or SME work team	Ground truth for subjective / regulated judgments	Expensive, slow; console-created human jobs require a browser-facing review UI + work team; ratings must be treated as data, not anecdote
Knowledge base (RAG) eval	Programmatic + LLM judge	Isolates retrieval from generation	Two flavors: retrieval-only vs retrieval + response generation

Gotcha: An automatic evaluation job has no display requirement. A human-based job created in the console does — prompts and responses must be rendered for the raters.

2. Metric Selection by Output Type

Output	Metric	What it actually measures	Trap
Classification	Accuracy, Precision/Recall, F1, AUC-ROC	Decision quality on held-out data	Accuracy misleads on imbalanced classes
Regression	RMSE, MAE, R²	Magnitude of error	R² hides outliers
Translation	BLEU	N-gram precision vs reference + brevity penalty	Penalizes valid alternate phrasings
Summarization	ROUGE (ROUGE-½/L)	N-gram / LCS recall vs reference	Recall-oriented; silent on factuality
Any generation	BERTScore, semantic similarity (embedding cosine)	Meaning match, not wording	Depends on embedding model quality
QA / VQA	Exact match, F1, verbatim / quasi-exact match	Literal or token-level agreement	Exact match = 0 for correct paraphrases
Safety / quality	Toxicity, harmfulness, stereotyping (bias), refusal correctness	Policy compliance of the output	Automated score ≠ policy sign-off
RAG	Faithfulness, Answer relevance, Context relevance, Context coverage	Grounding + retrieval quality	See §3

Hard rule: BLEU = precision-oriented; ROUGE = recall-oriented. A single overlap metric never settles an open-ended generation question — read overlap + semantic + judge/human together.

3. RAG Evaluation — Split Retrieval from Generation

A fluent answer that omits a source fact is usually a retrieval failure wearing a generation costume. Measure the stages separately or the cause stays a guess.

Stage	Metric	Failure it exposes	Fix
Retrieval	Context relevance, Context coverage	Chunks missing or irrelevant	Chunking strategy, embedding model, hybrid/keyword search, top-k, metadata filters
Generation	Faithfulness (grounded in context), Answer relevance	Model ignores or invents beyond retrieved context	Prompt instructions, grounding guardrail, model choice

Runtime companion: Bedrock Guardrails contextual grounding check returns grounding + relevance scores to block ungrounded answers in production; automated reasoning checks validate claims against rules.
Monitoring: CloudWatch metrics/logs/alarms turn a baseline into an alertable drift signal.

4. Bias Detection & Content Quality Validation

Traditional ML — SageMaker Clarify: pre-training bias metrics (class imbalance, disparate impact, KL/JS divergence, etc.) and post-training bias metrics, plus SHAP feature attribution. Clarify also produces the baseline Model Monitor consumes.
GenAI — bias as an eval metric: run stereotyping / harmfulness / toxicity via LLM-as-a-judge over a representative prompt dataset; use custom metrics with an explicit rating scale when a domain-specific fairness definition is required.
Guardrails as runtime control plane: content filters (hate/violence/sexual/insults/misconduct), denied topics, word filters, PII / sensitive-information filters, prompt-attack detection. Evaluate offline, enforce at inference.
Anti-pattern: treating a low toxicity score as proof of fairness. Bias is a property of outcomes across groups, not of individual responses.

5. Explaining Outputs — Match Technique to Model

Model type	Explanation mechanism	Limitation
Tabular / classic ML	Clarify SHAP (global + local attributions); feature attribution drift in Model Monitor	Needs a well-defined feature schema
RAG / retrieval-backed GenAI	Retrieved-passage provenance / citations; MLflow trace of each step's inputs and outputs	Requires retrieval — a non-RAG app has no source to point to
Customized / fine-tuned FM	Trace + eval scores only	Cannot point at a source — a genuine distractor when "explain this answer" is asked

Distinction to hold: a score says how well the system did; an explanation says why this answer came out this way. Auditors and regulators ask for the second.

6. Reproducibility, Baselines & Variant Comparison

Track every run — parameters, metrics, artifacts (SageMaker AI with MLflow, or SageMaker Experiments). Unreproducible scores come from unrecorded seeds, package drift, or mutated data.
Tracing captures input/output/metadata at each pipeline step so a bad answer is traceable to code, data, or parameters.
Baselines: derive from a known-good window; monitor departure in prediction quality, input distribution (data drift), and operational metrics (latency, error rate). A launch-time accuracy number tells you that it moved, never why.
Prompts are experiments: Bedrock Prompt Management stores prompts with variables + versions and compares variants — a wording change becomes a tracked comparison, not an unexplained behavior shift.
SageMaker shadow tests: candidate variant sits beside production on the same endpoint, receives a copy of live requests; only production answers the caller; shadow responses are logged or discarded. Validates model, container, and instance type against real payload shapes and concurrency. Some endpoint configurations are excluded.

Method	Live traffic?	Caller sees candidate?	Use when
Shadow test	Yes (copied)	No	Validate a candidate risk-free against real traffic
A/B / canary	Yes (split)	Yes, partially	Measure business impact with real users
Offline eval job	No	No	Gate a release on a static dataset

Trap: "Send half the traffic to the candidate" is A/B testing, not shadow testing — callers receive candidate responses, violating the no-impact requirement.

7. Security, IAM & Compliance

IAM service role per evaluation job: s3:GetObject on the dataset prefix, s3:PutObject on the output prefix; Bedrock evaluation also requires model access enabled in the region.
Encryption: SSE-KMS with a customer-managed key on dataset and output buckets; KMS CMK for CloudWatch Logs when logs carry prompt/response content.
Network isolation: S3 gateway endpoint, Bedrock / SageMaker interface endpoints (PrivateLink), and CloudWatch Logs endpoint so prompt data never traverses the public internet.
Sensitive data in human review: prefer a private workforce — prompts and responses shown to raters are an exposure surface. Redact PII before human rating where possible.
Auditability: CloudTrail records who launched and who approved an evaluation; retain eval artifacts as evidence for regulated sign-off.

8. Anti-Patterns → Correct Pattern

Anti-pattern	Why it fails	Correct pattern
One accuracy figure for GenAI	Many correct wordings; exact match collapses	Overlap + semantic + judge/human, multi-metric
Evaluating only the final RAG answer	Cannot isolate retrieval vs generation	Separate context metrics from faithfulness/relevance
Replacing prod with the candidate to "test"	Direct customer exposure	Shadow test
Reporting only the better of two runs	Hides variance and non-determinism	Track and diff every run
Judging quality from a stale launch baseline	Drift stays invisible	Known-good baseline window + continuous drift monitoring
Human ratings kept as anecdote	Not reproducible, not auditable	Aggregate ratings as data in the eval record
Single judge model, no version pin	Non-reproducible scores	Pin judge model/version; pair with programmatic metrics

Final gotcha set: LLM-as-a-judge is a supplement, never the sole gate. Human evaluation is the ground truth for subjective and regulated outputs. Programmatic metrics are the only ones cheap enough to run on every change — layer all three rather than choosing one.