Skill 4.1.3: Detect changes in data distribution that can affect model performance.
Task 4.1: Operating, Monitoring, and Securing ML and AI Solutions - Monitor ML and AI model inference and performance.
Exam North Star: Drift detection is a pipeline: baseline → capture → compare → alert. Missing any step = invisible drift. Inputs can look stable while expected answers change = concept drift.
Drift Types & Detection Signals
| Drift type | What changes | Detect with | Ground truth? |
|---|---|---|---|
| Data drift / covariate shift | P(X) – input feature distributions, prompt embeddings | SageMaker Model Monitor Data Quality; K-S, Chi-square, Jensen-Shannon, KL, PSI, L-infinity, MMD | No |
| Concept drift | P(Y|X) – input→label relationship | Model Quality Monitor; business KPIs; user feedback | Yes |
| Label drift / prior shift | P(Y) – label distribution | Model Quality Monitor; label distribution tracking | Yes |
| Feature attribution drift | Feature importance changes | SageMaker Clarify / Model Monitor Feature Attribution | No |
| Bias drift | Fairness metrics change | SageMaker Clarify / Model Monitor Bias Drift | Yes |
Gotcha: Data Quality Monitor catches input drift only. If prompts match baseline but accepted answers change, that is concept drift – use Model Quality Monitor, feedback, or business KPIs.
Drift Detection Pipeline (4 Moving Parts)
- Baseline – capture statistics/constraints from a stable period (Model Monitor baseline job).
- Current capture – endpoint data capture or batch output; for GenAI, prompt embeddings.
- Compare – choose statistic suited to data: K-S/PSI for tabular, Chi-square for categorical, Jensen-Shannon/KL/MMD for high-dimensional embeddings.
- Alert – threshold agreed in advance; CloudWatch alarm + SNS; optionally trigger retraining.
Threshold rule: Set upper and lower bounds. A flat line (zero invocations) is an incident; high-only alarms never fire.
SageMaker Model Monitor Monitors
| Monitor | Monitors | Ground truth? | Use when |
|---|---|---|---|
| Data Quality | Input schema, nulls, ranges, feature drift | No | Detect P(X) shift |
| Model Quality | Accuracy, precision, recall, RMSE | Yes | Detect P(Y|X) shift |
| Bias Drift | Fairness metrics | Yes | Regulatory/fairness |
| Feature Attribution Drift | Feature importance | No | Explainability drift |
Key configs: endpoint data capture → S3; baseline job → statistics + constraints files; monitoring schedule (hourly/daily/weekly); CloudWatch metrics/alarms. IAM role needs S3, CloudWatch, SageMaker permissions. KMS at rest; TLS in transit; VPC endpoints for private access.
Trap: Model Quality Monitor cannot run without ground truth labels. If labels are delayed, use Data Quality + proxy metrics until labels arrive.
GenAI / FM-Specific Monitoring
- CloudWatch generative AI observability – pre-built views for model invocations and Bedrock AgentCore agents: token usage, latency, errors, throttling, end-to-end prompt tracing.
- Amazon Bedrock Model Evaluation – automatic metrics, LLM-as-a-judge, human review; scores model and knowledge base responses.
- GenAI drift – baseline = input prompt embedding distribution; compare current embeddings with high-dimensional statistic. Concept drift surfaces in user feedback/business KPIs, not embeddings.
- Agent monitoring – Bedrock AgentCore Observability (OpenTelemetry-compatible) captures model inference, tool invocations, memory operations. Watch coordination failure, truncated streaming, tool failures.
Gotcha: Token usage and latency are operational metrics, not quality metrics. A model can be fast and cheap while answers degrade.
A/B Testing & Production Variants
- Single endpoint, multiple production variants – traffic split by variant weight / sum of weights.
- A/B test – candidate gets small weight; both variants measured on same metrics over same window.
- Shadow variant – receives copy of traffic; responses recorded, not returned. Tests serving stack with zero user risk.
- When to use: A/B when small real-user exposure is acceptable; shadow when output cannot be shown or risk is too high.
Trap: Comparing variants over different windows is not an A/B test. Same window, same metrics.
Anomaly & Error Detection Across Pipeline
- Monitor upstream data processing – job success ≠ valid output. Validate record counts, null constraints, schema integrity, partition existence, duplicate writes.
- Alarm on: error rates, throttling, invocation count drops to zero, duration growth run-over-run, schema changes, null spikes.
- Agent-specific: tool failures, truncated streaming, coordination loops.
Gotcha: Most production surprises are handoff failures: one component reports success while handing the next something unusable.
Decision Matrix: Which Tool?
| Need | Use |
|---|---|
| Input feature drift, schema, nulls | SageMaker Model Monitor Data Quality |
| Output quality with labels | SageMaker Model Monitor Model Quality |
| Fairness/bias drift | SageMaker Clarify / Model Monitor Bias Drift |
| Feature importance drift | SageMaker Clarify / Model Monitor Feature Attribution |
| FM token/latency/errors + prompt traces | CloudWatch generative AI observability |
| FM response quality scoring | Amazon Bedrock Model Evaluation |
| Agent telemetry | Bedrock AgentCore Observability |
| Request tracing across components | AWS X-Ray |
| Audit/compliance API activity | AWS CloudTrail / AWS Config |
| Cost only | AWS Cost Explorer (not operational monitoring) |
Anti-Patterns
- No baseline – no stable reference, no drift detection.
- Monitoring only job completion – ignores bad output.
- High-only thresholds – misses zero-invocation flatlines.
- Using Data Quality for concept drift – inputs unchanged, answers changed.
- Ignoring label delay – Model Quality Monitor stalls without ground truth.
- No encryption/IAM – Model Monitor needs S3, CloudWatch, SageMaker permissions; encrypt baseline/capture data with KMS.
- Confusing CloudTrail/Config with operational monitoring – they are audit/compliance, not drift/latency/token metrics.
Security & Compliance Quick Hits
- IAM roles – least privilege for Model Monitor, Bedrock, CloudWatch, S3.
- Encryption – KMS at rest for S3 baseline/capture; TLS in transit.
- VPC endpoints – private access to S3, SageMaker, CloudWatch, Bedrock where required.
- Audit – CloudTrail for API activity; Config for resource compliance.
- Data privacy – redact PII in captured data; use S3 bucket policies and KMS.