Skip to content

Skill 4.1.2: Monitor workflows to detect anomalies or errors in data processing or model inference

Task 4.1: Operating, Monitoring, and Securing ML and AI Solutions - Monitor ML and AI model inference and performance

Core Objective: Detect failures in data pipelines and model inference before users are impacted. Monitor both job completion and output validity. Combine metrics, logs, traces, and drift detection to catch silent failures: empty partitions, schema changes, duplicates, latency spikes, throttling, zero invocations, tool failures, truncated streaming, and coordination loops.

Monitoring Stack by Layer | Layer | What to Monitor | AWS Services | |-------|----------------|--------------| | Data Processing | Job success/failure, duration, DPU usage, record counts, schema, nulls, duplicates, freshness, distribution | AWS Glue (Data Quality), CloudWatch Logs/Metrics/Alarms, Step Functions, EventBridge, Lambda, EMR | | Model Inference | Invocations, latency, error rates (4XX/5XX), throttling, zero invocations, model quality, data drift | SageMaker AI (Model Monitor, Endpoints), CloudWatch, X-Ray | | Generative AI / Agents | Token usage, prompt/completion latency, errors, throttling, tool failures, truncated streaming, coordination failures | CloudWatch generative AI observability, Bedrock Model Evaluation, Bedrock AgentCore Observability | | End-to-End Tracing | Request path across components, bottleneck/failure attribution | AWS X-Ray, CloudWatch ServiceLens |

Key Signals & Alarms

  • Job completion ≠ success: A job can exit 0 but write empty/corrupt output. Monitor output metrics (row counts, null %, schema).
  • Invocation count drop to zero: Critical alarm. Indicates broken pipeline, auth failure, or upstream data issue.
  • Throttling: CloudWatch metric Throttling or 429 errors. Indicates concurrency/rate limits.
  • Latency growth: ModelLatency, InvocationTime trending up. Use anomaly detection.
  • Error rates: 4XX (client), 5XX (server), ModelError. Alarm on both spikes and flat lines.
  • Data drift: Input distribution shifts vs baseline. Use SageMaker Model Monitor or custom pipelines.
  • Concept drift: Input distribution stable but correct answer changes. Detected via business metrics, user feedback, model quality monitoring.
  • Agent failures: Tool invocation errors, coordination loops, truncated streaming. Use AgentCore Observability + X-Ray traces.

Exam Gotcha: CloudWatch = operational metrics, logs, alarms, dashboards. CloudTrail = API audit. AWS Config = resource compliance. Do not use CloudTrail/Config for latency or token usage.

Exam Gotcha: SageMaker Model Monitor detects data quality, model quality, bias drift, and feature attribution drift. It requires a baseline and a schedule. It does not replace CloudWatch for basic endpoint metrics.

Exam Gotcha: For generative AI, CloudWatch generative AI observability provides pre-built views (token usage, latency, errors, throttling) and prompt tracing. Bedrock Model Evaluation scores model/knowledge base responses (automatic, judge model, human). AgentCore Observability is default OpenTelemetry-compatible telemetry for agents.

Anomaly Detection Workflow

  1. Baseline: Capture stable period data (training distribution, known-good outputs, normal latency/error bands).
  2. Capture current: Collect same metrics from live traffic or job runs.
  3. Compare: Use statistics suited to data (e.g., embeddings for prompts, distribution distance). For metrics, use CloudWatch Anomaly Detection.
  4. Alert: CloudWatch Alarm -> SNS -> Lambda/EventBridge for automated response.
  5. Act: Rollback, retrain, scale, or fix upstream data.

Trade-Offs & Decision Matrix | Need | Use | Why | Avoid | |------|-----|-----|-------| | Basic endpoint metrics & alarms | CloudWatch | Native metrics, alarms, dashboards | CloudTrail (audit only) | | Distributed tracing | AWS X-Ray | End-to-end request path, latency attribution | CloudWatch metrics alone | | Data drift / model quality drift | SageMaker Model Monitor | Automated baseline comparison, scheduled | Manual log inspection | | GenAI runtime observability | CloudWatch GenAI observability | Token, latency, errors, prompt traces | Cost Explorer | | Agent telemetry | Bedrock AgentCore Observability | Tool calls, memory ops, coordination | Custom instrumentation only | | Model quality scoring | Bedrock Model Evaluation | Automatic, judge, human evaluation | CloudWatch metrics | | A/B testing model versions | SageMaker multi-variant endpoints | Weighted traffic, same metrics | Shadow variant if risk too high | | Shadow testing | SageMaker shadow variant | Copies traffic, no user impact | A/B if risk acceptable |

Security & Compliance

  • IAM: Monitoring services need roles to read metrics/logs, write alarms, invoke actions. Least privilege.
  • Encryption: Enable KMS encryption for CloudWatch Logs, X-Ray data, SageMaker data, Bedrock. In transit via TLS.
  • VPC Endpoints: Use VPC endpoints for CloudWatch, X-Ray, SageMaker, Bedrock to keep traffic private.
  • Audit: CloudTrail for API calls; AWS Config for resource config compliance. Not for operational telemetry.
  • Data Privacy: Avoid logging PII in prompts/responses. Use masking or redaction. Model Monitor data capture can be encrypted.

Anti-Patterns

  • Monitoring only job completion: Ignores bad output (empty, null, schema mismatch).
  • Alarming only on high values: Misses zero invocations, flat lines, missing data.
  • No baseline: Drift detection without baseline is guesswork.
  • Ignoring throttling: Leads to silent failures under load.
  • Using CloudTrail for latency/token metrics: Wrong tool.
  • Manual drift inspection: Not scalable; build pipeline.
  • No upstream monitoring: Inference errors often originate in data processing.
  • Forgetting agent-specific failures: Tool failures, truncated streaming, coordination loops need dedicated signals.

Hard Rule: An alarm on a flat line is as important as an alarm on a spike. Invocation count = 0 is a broken pipeline.

Hard Rule: Data drift ≠ concept drift. Data drift = P(X) changes. Concept drift = P(Y|X) changes. Concept drift surfaces in business metrics and user feedback.

Hard Rule: SageMaker Model Monitor requires a baseline and a schedule. It is not real-time by default; configure monitoring schedule.

Quick Reference: Symptom -> Signal -> Service | Symptom | Likely Cause | Monitor | Service | |---------|--------------|---------|---------| | Model invocation errors spike | Bad upstream data | Output validation, schema checks | Glue Data Quality, CloudWatch | | Invocation count drops to 0 | Broken pipeline, auth, throttling | Invocation count, Throttling | CloudWatch Alarms | | Latency increases over time | Model degradation, scaling | ModelLatency, Anomaly Detection | CloudWatch, X-Ray | | Users reject answers, inputs same | Concept drift | Business metrics, feedback | Model Monitor, custom | | Prompts distribution shifts | Data drift | Input distribution vs baseline | SageMaker Model Monitor | | Agent loops or stops mid-response | Coordination failure, truncation | Tool calls, streaming status | AgentCore Observability, X-Ray | | Tool call fails | External dependency | Tool invocation errors | AgentCore Observability |