Skill 4.1.6: Configure AI-specific performance monitoring for foundation models (FMs), such as Amazon Bedrock evaluations
Task 4.1: Operating, Monitoring, and Securing ML and AI Solutions — Monitor ML and AI model inference and performance
The Three Monitoring Jobs — Pick the Right One First
| Job | Question it answers | Primary service |
|---|---|---|
| Operational health | Is it up, fast, error-free, within quota? | CloudWatch metrics + alarms (namespace AWS/Bedrock) |
| Output quality | Is the answer still correct, faithful, safe? | Amazon Bedrock model evaluation (batch + inference-time) |
| Request tracing | Which component in this one request was slow/wrong? | X-Ray + CloudWatch generative AI observability |
Exam gotcha: CloudWatch metrics prove a model responded; they never prove it responded well. Any option that claims latency/error metrics detect quality degradation or drift is the distractor.
Amazon Bedrock Model Evaluation — Three Job Types
| Automatic (built-in metrics) | LLM-as-a-judge | Human-based | |
|---|---|---|---|
| Scorer | Deterministic algorithms | A designated judge FM | Human reviewers (work team) |
| Metrics | Accuracy, Toxicity, Robustness (text); RAG/KB metrics | Built-in or custom metrics defined by a prompt + rating scale, each with a written explanation | Custom metrics only |
| Best for | Fast, repeatable regression gates in CI/CD | Nuanced/subjective criteria at scale, with rationale | Ground truth on highly subjective or high-stakes output |
| Speed / cost | Fastest, cheapest | Medium | Slowest, most expensive, needs workforce |
| Input dataset | Prompt + reference response + optional categories | Prompt + reference response (ground truth) | Prompt + up to two candidate responses |
RAG / Knowledge Base metrics — evaluation is scoped to the stage:
- Retrieve-only: Context relevance, Context coverage → grades the retriever, not the generator.
- Retrieve-and-generate: Correctness, Faithfulness, Helpfulness, Citation precision → grades the full pipeline.
Guardrails metrics — evaluate guardrail behavior itself (harmful-content detection, intervention rate), separate from model quality.
Anti-pattern trap: Using the same model as both subject and judge invites self-preference bias. Choose a different model family (typically stronger) as the judge.
Scale rule: Bedrock evaluation jobs are sample-sized (order of ~1,000 prompts), not full-traffic replays — and human-based jobs cap out small. Evaluate a representative sample; use inference-time evaluation for continuous coverage.
Batch vs. Inference-Time (Online) Evaluation
| Batch evaluation job | Inference-time evaluation | |
|---|---|---|
| Traffic | Offline dataset in S3 | Live production invocations, scored on a configurable sampling rate |
| When it runs | Pre-release, scheduled regression | Continuously, in the background |
| Where results land | S3 output + console | CloudWatch generative AI observability |
| Use when | Gating a model/prompt version before deploy | Catching quality decay on real prompts after deploy |
Both job families: require an IAM service role trusted by bedrock.amazonaws.com with S3 read (dataset) + S3 write (results), plus invoke permissions on the judge/evaluator model.
CloudWatch Generative AI Observability — the Operational Layer
- Pre-built dashboards: Bedrock model invocations and AgentCore agents — token usage, latency, errors, throttles, plus end-to-end prompt tracing.
- Bedrock runtime metrics (default-on, per
ModelIddimension):Invocations,InvocationLatency,InvocationClientErrors,InvocationServerErrors,InvocationThrottles,InputTokenCount,OutputTokenCount,OutputImageCount. - Streaming nuance: token/latency semantics differ for streaming responses — rely on time-to-first-token style measures rather than total token counts for perceived responsiveness.
- Prompt tracing requires X-Ray tracing enabled (and transaction search for the GenAI observability trace view).
- AgentCore Observability ships OpenTelemetry-compatible telemetry by default for agents on AgentCore Runtime — model inference calls, tool invocations, memory operations — with no per-agent instrumentation.
Agent-specific signals to monitor: coordination failure (agent loops instead of terminating), truncated streaming (response stops mid-answer), tool failure / wrong tool selection, and goal success rate.
Alarm on both directions: a spike in InvocationThrottles/InvocationServerErrors and an Invocations count that falls to zero. Thresholds written only for high values never fire on a dead pipeline.
Batch Jobs vs. Live Traffic — Variant Comparison (A/B)
- Multiple production variants behind one endpoint, traffic split by each variant's weight ÷ sum of weights — enables small-exposure canaries on live traffic.
- Measure both variants on the same metrics over the same window — otherwise it is two observations, not a comparison.
- Shadow variant: receives a copy of live traffic and records responses instead of returning them — validates the serving stack with zero user exposure.
Drift Detection vs. Quality Degradation — Different Tools
| Signal | Definition | Detection |
|---|---|---|
| Data drift (covariate shift) | P(X) changes — prompt/embedding distribution moves | Baseline capture → current capture → distribution-distance statistic on prompt embeddings → threshold alarm |
| Concept drift | P(Y|X) changes — prompts look normal, the accepted answer changed | User feedback + downstream business metrics; evaluation jobs with fresh ground truth |
| Quality degradation | Output quality falls (hallucination, tone, safety) | Bedrock evaluations (batch + inference-time) |
Drift pipeline, four parts (recite these): capture a baseline from an agreed stable window → capture the same measure on current traffic → compare distributions with a statistic suited to the data → alert past a pre-agreed threshold. Build it as a pipeline, not an ad-hoc query.
Security, IAM, and Data Handling
- Evaluation jobs: IAM service role (Bedrock trust policy) scoped to specific S3 prefixes; least privilege on the judge model.
- Encryption: SSE-KMS on the evaluation dataset and results bucket; KMS on the CloudWatch Logs log group holding invocation logs; KMS on X-Ray/observability data.
- VPC endpoints:
bedrock-runtime,bedrock(control plane), CloudWatch Logs, X-Ray, and an S3 gateway endpoint to keep evaluation/logging traffic off the public internet. - Model invocation logging is opt-in and off by default — and it stores raw prompt/response bodies (text, image, embedding data) to CloudWatch Logs or S3. You cannot backfill it, so enable it before you need the evidence.
- Sensitive-data risk: logged prompts are the highest-PII surface in the stack — combine with Guardrails, KMS, tight log-group access, and a retention/lifecycle policy. Use CloudTrail for audit of evaluation job creation and config changes, not for performance.
Exam gotcha: CloudTrail = who called what API (audit). CloudWatch = metrics, logs, alarms. X-Ray = per-request trace. Bedrock evaluations = answer quality. Cost Explorer/Config = neither operational nor quality telemetry.
Anti-Patterns to Recognize Instantly
- Dashboard-only monitoring — dashboards without alarms; no alerting on throttles, error rates, or zero-invocation flatlines.
- Monitoring job completion instead of job output — an ETL/feature job exits successfully while writing an empty partition or nulled column; the symptom surfaces as downstream invocation errors.
- Judging with the subject model — self-preference bias inflates scores.
- Evaluating only on the original test distribution — misses concept drift entirely.
- Ignoring token metrics — token counts are the leading indicator of both cost blowout and prompt-injection/prompt-bloat regressions.
- Relying on CloudWatch metrics to catch hallucinations or unsafe output — that is evaluation's job, not the metrics plane's.