Skip to content

Skill 4.1.1: Monitor model performance in production by using Amazon CloudWatch generative AI observability, Amazon Bedrock Model Evaluation, and drift detection pipelines

Task 4.1: Operating, Monitoring, and Securing ML and AI Solutions - Monitor ML and AI model inference and performance

Objective: Detect degradation before the business notices. A model that scored well at training degrades quietly as data shifts; GenAI adds its own failure modes (tool-call failures, truncated streaming, runaway agent coordination). Each needs its own signal.

The Three Monitoring Jobs — Find the Missing Job, Not a New Dashboard

Job Question it answers Primary service
Collect What is the workload emitting? CloudWatch metrics/logs, Bedrock model invocation logging
Judge Is the output still good enough? Bedrock Model Evaluation (automatic / LLM-judge / human), SageMaker Model Monitor
Trace Which component produced this request? X-Ray, CloudWatch GenAI observability prompt traces, AgentCore Observability

Amazon CloudWatch Generative AI Observability

  • Scope: pre-built views over Bedrock model invocations and Bedrock AgentCore agents — no custom dashboarding required.
  • FM metrics: token usage (input/output tokens), latency (time-to-first-token and end-to-end), error rates (client/server), throttling, invocation counts.
  • Agent metrics: end-to-end prompt tracing across reasoning steps, model invocations, and tool calls; memory operations.
  • Namespace: Bedrock runtime metrics publish to AWS/Bedrock (Invocations, InvocationLatency, InvocationClientErrors, InvocationServerErrors, InvocationThrottles), dimensioned per model/operation.
  • CloudWatch Anomaly Detection learns a band per metric — use it for seasonality instead of a static threshold. Composite alarms cut noise (e.g., latency and error rate breach together).

Gotcha — Logging is off by default. Bedrock model invocation logging must be explicitly enabled and pointed at CloudWatch Logs or S3. Without full request/response payloads you cannot compute prompt-embedding drift or feed a judge model.

Gotcha — The silent failure. An invocation count dropping to zero is a broken pipeline. A threshold written only for high values never fires on a flat line — pair every upper threshold with a low/absent-traffic alarm (treat missing data as breaching).

Amazon Bedrock Model Evaluation — Which Job Type?

Job type Metric style Use when Trade-off
Automatic Built-in computed metrics (accuracy, robustness, toxicity, bias; ROUGE/BERTScore for summarization) Fast, repeatable regression checks at release time Cheapest, fastest
LLM-as-a-judge Custom metrics + natural-language explanation per score Subjective quality (helpfulness, tone, correctness) at scale Judge model must be available in region; judge choice biases results
Human Subjective ratings from a reviewer workforce Ground truth for what users actually accept; calibrating the judge Slowest, most expensive — sample, don't sweep
  • Evaluation targets: FMs, Knowledge Bases (RAG: faithfulness, relevance, groundedness), and Agents (task success, tool-call accuracy).
  • Flow: dataset in S3 (JSONL) → evaluation job → metrics to S3 + CloudWatch. Run as a scheduled job post-deployment, not only pre-release.
  • Baseline role: an automatic job at launch sets the reference; repeat on live samples to catch quality regression.

Gotcha — A judge model is not ground truth. Treat LLM-as-a-judge scores as a relative, versioned signal. Changing the judge invalidates comparison with prior runs; pin judge model and version.

Drift Detection Pipelines — Four Parts to Recite

  1. Baseline — capture the measure from a period agreed to be stable (training data or a validated production window).
  2. Current — capture the same measure from live traffic on a schedule.
  3. Compare — apply a statistic suited to the data type.
  4. Alert — fire when distance passes a pre-agreed threshold.
Drift type What shifts How it surfaces
Data drift (covariate) Input distribution P(X) Baseline vs. current feature / prompt-embedding distributions; monitor job + CloudWatch alarm
Concept drift Relationship P(Y|X) — same prompts, new expected answer Not found by input comparison; surfaces via user feedback, downstream business KPIs, human/judge evaluation
Label / prediction drift P(Y) or prediction distribution Output distribution tracking; ground-truth comparison when labels arrive
Feature attribution drift Which features drive predictions SageMaker Clarify / Model Monitor explainability monitoring
  • Tabular drift: SageMaker Model Monitor — DataQuality (data drift), ModelQuality (requires ground-truth labels, which are delayed), ModelBias, ModelExplainability. Baselines come from a baseline job; results publish to CloudWatch + S3 reports.
  • GenAI drift: the measure is usually the distribution of input prompt embeddings; use a multi-dimensional statistic (MMD / KL divergence / cosine-distance distributions) — per-feature tests degrade in high dimensions.
  • Delayed-label workaround: monitor proxy metrics until ground truth arrives — latency, token count, refusal rate, tool-error rate, guardrail interventions, user thumbs-up/down.

Gotcha — Drift ≠ degradation. A drift alert says inputs changed, not quality dropped. Confirm with an evaluation or business metric before retraining — and drift detection cannot find concept drift.

Anomalies & Errors Across the Pipeline

  • Monitor upstream of the symptom. Most production surprises are handoff failures: a job exits 0 but writes an empty partition, a silent schema change nulls a column, a retry double-writes a record.
  • Alarm on output, not just completion: record counts, null rates, schema/version checks, partition freshness, run-over-run row-count deltas.
  • High-value alarm set: error rate, throttle rate, invocation count (spikes and drops), duration growth run-over-run, queue/backlog depth.
  • Agent failure modes: coordination failure (loops instead of terminating), truncated streaming, tool failures (empty/exception tool output), step-count and token-budget overruns.
  • AgentCore Observability is the default telemetry for agents on AgentCore Runtime — OpenTelemetry-compatible, auto-capturing inference calls, tool invocations, and memory operations with no per-agent instrumentation.

Comparing Versions with Live Traffic (A/B)

  • A production variant = a model + its serving resources. Multiple variants sit behind one endpoint; traffic split = variant weight ÷ sum of weights.
  • A/B: give the candidate a small weight and measure both variants on the same metrics over the same window — otherwise it is two observations, not a comparison. Shift weight as evidence accumulates.
  • Shadow variant: receives a copy of live traffic, records responses, returns nothing to users — validates the serving stack at zero user risk, but yields no user-feedback signal.

Gotcha — Bedrock evaluations are offline. They score a dataset or captured payloads, not live user traffic. Use A/B / shadow for online comparison and evaluations for offline scoring — different questions.

Security, IAM & Compliance

  • Least privilege: monitoring/evaluation jobs need S3 (baseline + output), CloudWatch (PutMetricData, logs), and KMS permissions; Bedrock invocation logging needs logs:CreateLogStream / logs:PutLogEvents on the target log group.
  • Encryption: SSE-KMS on baseline/report buckets; KMS CMK on CloudWatch Logs groups holding prompts and completions; TLS in transit for all Bedrock/SageMaker calls.
  • Network isolation: VPC interface endpoints for CloudWatch, CloudWatch Logs, Bedrock, SageMaker; S3 gateway endpoint — keeps prompt data off the public internet.
  • Multi-account: CloudWatch cross-account observability to centralize metrics/logs/traces in a monitoring account.
  • Retention: CloudWatch metrics — 1-min data 15 days, 5-min 63 days, 1-hour 455 days; CloudWatch Logs never expire by default — set retention explicitly for prompt data.
  • Governance boundaries: CloudTrail = API audit trail, AWS Config = configuration/compliance. Neither reports latency, tokens, or traces.

Anti-Patterns & Distractor Traps

  • Dashboard without an alarm — observability nobody is paged on.
  • Monitoring only the inference call — the cause usually sits upstream in data processing.
  • Completion status as data quality — a successful job can emit unusable output.
  • Static thresholds on seasonal metrics — use CloudWatch anomaly detection bands.
  • Confusing drift types — input distribution change = data drift; changed user expectations = concept drift.
  • Assuming ModelQuality monitoring works without labels — it does not; use proxy metrics until labels land.
  • Service mix-ups: CloudWatch (metrics/alarms/logs) vs CloudTrail (audit) vs AWS Config (compliance) vs X-Ray (traces) vs Cost Explorer (billing). Cost allocation tags never affect runtime behavior.