Skill 4.1.4: Monitor model performance in production by using A/B testing
Task 4.1: Operating, Monitoring, and Securing ML and AI Solutions – Monitor ML and AI model inference and performance
Core concept: Run two or more production variants behind a single SageMaker endpoint, split live traffic by relative weight, and score both variants on the same metrics over the same time window. Objective: prove a candidate model is better on real traffic before it carries all of it — and roll back instantly if it isn't.
SageMaker A/B mechanics (the exam's default pattern)
| Element | Key point |
|---|---|
| ProductionVariant | A model + its serving resources. Several can share one endpoint. Each needs VariantName, ModelName, InitialVariantWeight, InitialInstanceCount. |
| Traffic distribution | Share = variant weight ÷ sum of all weights. Weights are relative positive integers, not percentages. |
| UpdateEndpoint | Changing only weights = in-place update, no new instances, no downtime. Changing instance count/type = rolling or blue/green. |
| ShadowProductionVariants | Receives a copy of traffic; responses are logged, never returned. Zero user exposure. |
| DataCaptureConfig | Writes request/response payloads to S3 (input, output, or both) at a sampling %. Foundation for offline scoring and Model Monitor. |
| Deployment guardrails | Blue/green + canary/linear traffic shift with auto-rollback CloudWatch alarms (e.g., 5XX or latency breach). |
| Weights | Split |
|---|---|
| 1 / 1 | 50 / 50 |
| 1 / 4 | 20 / 80 |
| 1 / 9 | 10 / 90 |
| 1 / 99 | 1 / 99 (canary-style exposure) |
A/B vs. the alternatives
| Pattern | Candidate sees live traffic | Users exposed | Use when |
|---|---|---|---|
| A/B test | Yes, fixed small share | Yes | Comparing quality / business metrics; small exposure is tolerable |
| Shadow test | Yes (copy) | No | Validating serving stack, latency, cost, hard-failure risk with zero user impact |
| Canary / linear | Progressively increasing | Yes, briefly | Safe deployment, not measurement; pairs with auto-rollback |
| Blue/green | All-at-once cutover | Yes | Instant switch + instant revert |
A/B = measurement. Canary = deployment. If the question asks "which version is better?", it's A/B variants on one endpoint. If it asks "how do I release safely?", it's canary/blue-green with rollback alarms.
Test design rules (high-yield)
- Concurrent, not sequential. Both variants must serve at the same time so seasonality, traffic mix, and upstream releases don't confound the result.
- Identical infrastructure. Same instance type and count per variant — otherwise latency/error deltas reflect hardware, not the model.
- Pre-commit to a horizon and a primary metric. Avoid peeking and stopping on a favourable fluctuation; correct for multiple comparisons if you check repeatedly.
- Cover a full business cycle (at least one weekly cycle) before declaring a winner.
- Guardrail metrics alongside the primary metric: latency (p50/p99), 4XX/5XX rate, throttling, cost per 1K inferences.
- Split is per-invocation, not per-user. A single session can hit both variants — relevant for any per-user or stateful metric.
- Delayed ground truth: score quality via sampled human review, an LLM-as-a-judge job, or a Model Monitor model-quality schedule once labels land.
What to watch per variant
| Signal | Source / dimension |
|---|---|
| Model latency, overhead latency, invocations, 4XX/5XX, per-instance load, CPU/GPU/memory | CloudWatch, dimensions EndpointName + VariantName |
| Token usage, throttling, prompt traces, tool calls, agent coordination | CloudWatch generative AI observability / AgentCore Observability (OpenTelemetry-compatible) |
| Response quality (automatic metrics, judge model, human ratings) | Amazon Bedrock model evaluation jobs (offline/async, not a traffic splitter) |
| Input distribution shift | Model Monitor data-quality schedule vs. a captured baseline |
| Output/quality shift once labels exist | Model Monitor model-quality schedule (needs ground-truth merge) |
| Flat-line invocation count | Alarm on a low threshold — zero traffic is a broken pipeline, and high-only alarms never fire |
Drift split that trips people up: data drift = input distribution P(X) moved (found by comparing captured distributions to baseline). Concept drift = inputs look the same but the expected answer changed, P(Y|X) moved (found in downstream business metrics and user feedback). Model Monitor finds the first, not the second.
Security & compliance
- IAM: distinct least-privilege execution role per model; separate the control-plane
UpdateEndpoint(traffic shift) from the data-planeInvokeEndpoint— weight changes are auditable events. - KMS: CMK for the S3 capture bucket and for endpoint instance EBS volumes (encryption at rest). Captured payloads are sensitive production data — treat the bucket as restricted.
- Network: private VPC config (subnets + SGs) for endpoints; interface VPC endpoints for SageMaker Runtime and CloudWatch, gateway endpoint for S3 — no public egress.
- In transit: TLS on
InvokeEndpoint; never route captured traffic over unencrypted channels. - Audit vs. observe: CloudTrail records who changed the traffic split; CloudWatch records how each variant performed. These are not interchangeable.
Anti-patterns & distractor traps
- Two separate endpoints compared against each other — infrastructure and scaling differences confound the result. Use variants on one endpoint.
- Time-sliced "A/B" (week 1 old, week 2 new) — seasonality, not signal.
- No data capture configured — you cannot enable capture retroactively for traffic already served; and without it there is no offline analysis.
- Stopping the test the moment one variant leads — inflates false positives.
- Using a shadow variant expecting responses — shadow responses are recorded only.
- Treating Model Monitor as an A/B tool — it detects drift against a baseline; it does not split traffic.
- Leaving the losing variant attached — remove it (or delete the endpoint) after the decision; idle variants bill continuously.
- Service substitution distractors: CloudTrail = API audit, AWS Config = resource compliance, Cost Explorer = billing. None report latency, token usage, or variant quality.
Hard rules to memorize: traffic share = weight ÷ sum of weights (need not total 100); shadow variants never return responses; A/B variants share one endpoint; data capture must be configured before traffic arrives; auto-rollback requires CloudWatch alarms wired to a deployment guardrail.