Skill 2.3.4: Explain Model Outputs
Task 2.3: ML Model and Foundation Model (FM) Development — Analyze and evaluate the performance of ML and AI systems
Core objective: Move from "what score did it get?" (evaluation) to "why did this output come out this way?" (explanation). Evaluation is the performance artifact; explanation is the audit, regulatory, and stakeholder artifact.
Exam gotcha: Explanation ≠ evaluation. A metric says how well; an explanation says why this instance. If the question says justify / audit / trace / regulatory / dispute, it wants explanation tooling — not another metric.
Two Explanation Families — Pick by Model Type
| Model type | Explanation mechanism | AWS service |
|---|---|---|
| Tabular / classical ML | Feature attribution (SHAP), PDP, ICE, counterfactuals | SageMaker Clarify |
| Classical NLP | Token-level attribution | Clarify (NLP explainability) |
| Computer vision | Saliency / heatmap attribution | Clarify (CV explainability) |
| Tree ensembles / built-in algos | Native feature importance (Gini/gain) | Algo-native; Clarify SHAP for consistency |
| RAG / GenAI | Source citations — which retrieved passage the answer was built from | Bedrock Knowledge Bases (Return citations), Guardrails grounding |
| Agentic / tool-using | Trace — reasoning steps, tool invocations, KB lookups | Bedrock Agents trace |
| Any GenAI | Judge explanation, CoT rationale, confidence/logprob | Bedrock Evaluations (LLM-as-a-judge) |
Global vs Local — The First Fork
| Global | Local | |
|---|---|---|
| Answers | "What drives the model overall?" | "Why this prediction?" |
| Methods | Aggregated |SHAP|, PDP, permutation importance, native importance | Per-instance SHAP, ICE, counterfactual, anchors |
| Use | Model documentation, feature selection, model card | Customer dispute, denial letter, auditor question |
| Trap | Aggregate |SHAP| hides sign/direction | Local SHAP is not causal |
Hard rule: SHAP values are relative to a baseline (expected value over a background dataset). No baseline dataset → attributions are undefined. Clarify requires a baseline dataset in the analysis config.
SHAP vs Alternatives
| Method | Scope | Model-agnostic? | Cost | Key caveat |
|---|---|---|---|---|
| KernelSHAP (Clarify default) | Local → Global | Yes | Slow (sampling) | More samples = more accuracy, more runtime |
| TreeSHAP | Local → Global | No (trees only) | Fast, exact | Restricted to tree ensembles |
| Native importance | Global only | No | Free | Impurity-based = biased toward high-cardinality features |
| PDP | Global | Yes | Cheap | Assumes feature independence — misleading with correlated features |
| ICE | Local (per-instance curves) | Yes | Cheap | Preferred over PDP when features interact |
| Counterfactual | Local | Yes | Moderate | "Smallest change that flips the outcome" — actionable |
| Anchors | Local | Yes | Moderate | Rule-like if-then explanation, high precision |
Exam gotcha: Correlated features → never trust PDP alone; use SHAP dependence or ICE. SHAP's additivity (attributions sum to prediction − baseline) and consistency axioms are why it is preferred over raw importance for regulatory explanation.
SageMaker Clarify — Operational Facts
- Produces: an explainability report (JSON to S3) plus optional pre-training and post-training bias reports.
- Runs as: a processing job (batch) or attached to an endpoint (online explainability).
- Requires: model container or endpoint + baseline dataset + analysis configuration. Missing baseline = failure or garbage.
- Modalities: tabular, NLP (token-level), CV (saliency maps).
- Key integration: Clarify + Model Monitor → model explainability monitor tracks drift in feature attributions, not just drift in predictions.
- Visualization: SageMaker Studio renders the report; raw JSON is for pipelines and audit trails.
- Anti-pattern: unbounded feature sets and high-cardinality categoricals explode runtime — sample the baseline or reduce features.
GenAI Explanation — Attribution, Not Attribution Values
| Need | Mechanism | Service |
|---|---|---|
| "Which document did this come from?" | Citations with chunk-level source URI | Bedrock Knowledge Bases — Return citations enabled |
| "Is the answer grounded in retrieved context?" | Contextual grounding check → grounding score, threshold, action detect/block | Bedrock Guardrails |
| "Is the answer relevant to the question?" | Relevance score (separate from grounding) | Bedrock Guardrails |
| "Are the claims valid?" | Automated reasoning checks (formal verification) | Bedrock Guardrails |
| "What steps did the agent take?" | Trace: orchestration, rationale, tool invocation | Bedrock Agents |
| "Why did the judge score it that way?" | Judge returns score + explanation | Bedrock Evaluations |
| "Was a guardrail triggered?" | Guardrail trace on the invocation | Bedrock Guardrails / Agents |
Critical trade-off: A RAG system can point at its source; a fine-tuned/customized model cannot — knowledge lives in the weights. If the requirement is "cite the source," the architecture must be RAG, not fine-tuning. High-frequency distractor pair.
Hard rule: Grounding and relevance are separate scores. High relevance + low grounding = fluent answer unsupported by context (hallucination). Low relevance = retrieval returned the wrong passages. Splitting them isolates retrieval failure from generation failure.
Explaining a Failing Run (Adjacent, Frequently Co-Tested)
| Symptom | Diagnosis | Evidence artifact |
|---|---|---|
| Loss oscillates, never settles | Learning rate too high | Loss curve in MLflow / Experiments |
| Loss barely moves | LR too low, or model lacks capacity | Loss curve |
| Train improves, validation worsens | Overfitting — not a broken run | Train vs validation curves |
| Score unreproducible across runs | Unseeded randomness, env drift, data mutation | Run lineage: params + metrics + artifacts |
| Attributions shift while accuracy holds | Explanation drift | Clarify explainability monitor |
Exam gotcha: Oscillating loss → learning rate, first. Never instance count, data format, or class count.
Faithfulness vs Plausibility — The Explanation-Quality Trap
- Plausible = convincing to a human. Faithful = actually reflects the model's computation.
- Attention weights are not faithful — they can be reshuffled without changing the output.
- Chain-of-thought is not guaranteed faithful — coherent rationales can be produced post hoc.
- SHAP is faithful by construction (additivity + consistency) but still not causal.
- Validate on stability (similar inputs → similar attributions) and fidelity (does removing top features actually change the prediction?).
Governance, Security & Compliance Layer
- Model Cards / Model Registry: intended use, limitations, evaluation results — the documentation-level explanation artifact.
- Lineage: MLflow / SageMaker Experiments tie an output back to code, data, params, and environment.
- Encryption: explanations leak training-data patterns and PII → KMS at rest, TLS in transit, lock down the S3 bucket holding Clarify reports.
- Network: VPC endpoints (PrivateLink) keep explainability jobs and Bedrock invocations off the public internet.
- IAM: Clarify's execution role needs S3 read on baseline data and S3 write on the report bucket; Bedrock needs model-invocation, Knowledge Base, and Guardrail permissions.
- Regulatory: EU AI Act and financial-services "right to explanation" → local, per-instance, actionable explanations (SHAP + counterfactual), not global importance alone.
Decision Matrix — "Which explanation do I reach for?"
| Question phrasing | Answer |
|---|---|
| "Why did the model deny this specific applicant?" | Local SHAP (Clarify) + counterfactual |
| "What features matter most overall?" | Global SHAP / native importance / PDP |
| "Smallest change to flip the outcome?" | Counterfactual |
| "Which document supports this answer?" | Bedrock KB citations |
| "Is the answer supported by retrieved context?" | Guardrails contextual grounding (grounding score) |
| "Show the agent's reasoning and tool calls" | Bedrock Agents trace |
| "Detect when the reasons change, not just the score" | Clarify + Model Monitor explainability monitor |
| "Reproduce and diff two runs" | MLflow / SageMaker Experiments |
| "Score generated text against reference" | BLEU / ROUGE (overlap); BERTScore / semantic similarity (meaning) |
| "Automate quality judgment with a rationale" | LLM-as-a-judge (Bedrock Evaluations) |
| "Nuanced preference or high-consequence output" | Human-in-the-loop (A2I / Ground Truth) |
Anti-Patterns (Common Distractors)
- Treating SHAP as causal — attribution on the model, not the world.
- PDP with correlated features — yields physically impossible curves.
- Attention weights as explanation — plausible, not faithful.
- CoT rationale as proof of reasoning — post hoc rationalization is possible.
- Fine-tuning when source attribution is required — no citations available.
- One metric as the whole GenAI story — overlap and meaning metrics must be read together.
- Global importance presented as a per-instance explanation — fails audit.
- Clarify without a baseline dataset — attributions are undefined.
- Measuring only the end-to-end RAG answer — cannot separate retrieval failure from generation failure.