Skill 2.3.3: Compare the performance of shadow variants to production variants
Task 2.3: ML Model and Foundation Model (FM) Development - Analyze and evaluate the performance of ML and AI systems
Core concept: SageMaker AI shadow testing deploys a candidate (shadow) variant beside the production variant on the same real-time endpoint. Production serves 100% of caller responses; shadow receives an asynchronous copy of live requests. Shadow responses are logged, never returned. Objective: validate model, container, or instance changes against real production traffic with zero user risk.
How it works
- Production variant: Handles live traffic; returns responses to callers.
- Shadow variant: Receives mirrored requests (sampling % configurable); processes them; responses captured for offline analysis.
- Isolation: Shadow failures do not affect production. Production latency and errors remain independent.
- Configuration: Endpoint config with shadow variants. Real-time endpoints only.
- Data capture: Enable SageMaker Data Capture to S3 (KMS, VPC) to log requests/responses from both variants.
- Metrics: CloudWatch emits per-variant metrics. Use dashboards and alarms.
Key Metrics to Compare
| Category | Production Baseline | Shadow Variant | Notes |
|---|---|---|---|
| Operational | Latency P50/P90/P99, throughput, error rate, CPU/GPU/mem | Same | Shadow must not degrade production. Compare latency distributions. |
| Model Quality | Accuracy, F1, precision/recall, AUC, RMSE | Same | Requires ground truth. Use captured data + offline evaluation. |
| Drift/Bias | Input drift, prediction drift, bias metrics | Same | Use Model Monitor on captured data if supported. |
| GenAI | BLEU, ROUGE, BERTScore, semantic similarity, toxicity, hallucination, faithfulness | Same | Use Bedrock Evaluations, LLM-as-a-judge, human review. |
| Cost | Instance cost, inference cost | Same | Shadow doubles compute cost. Account for sampling %. |
Shadow vs A/B vs Canary vs Blue/Green
| Strategy | Traffic | Caller sees candidate? | Measures user impact? | Rollback | Use when |
|---|---|---|---|---|---|
| Shadow | Copy of live | No | No | N/A | Validate on real traffic without risk |
| A/B | Split (e.g., 50/50) | Yes (some users) | Yes | Yes | Measure business metrics (CTR, conversion) |
| Canary | Gradual shift | Yes (small %) | Yes | Yes | Gradual release with monitoring |
| Blue/Green | Cutover | Yes (after cutover) | Yes | Yes | Instant cutover, easy rollback |
Implementation Checklist
- Baseline first: Record production variant metrics before shadow test.
- Deploy shadow variant: Same endpoint config; can differ in model, container, instance type.
- Set sampling: Use 100% or lower; ensure representative traffic.
- Enable Data Capture: S3 bucket, KMS key, VPC endpoint. Capture requests/responses from both variants.
- Monitor per-variant: CloudWatch metrics, logs, alarms. Alert on shadow errors even though production unaffected.
- Run long enough: Cover traffic seasonality, peak loads, diverse inputs.
- Offline analysis: Compare captured data. Use SageMaker Processing, MLflow, Bedrock Evaluations, human review.
- Decide: Promote shadow to production, iterate, or discard. Remove shadow variant to stop costs.
Security & Compliance
- IAM: Least-privilege execution role for SageMaker endpoint, S3 data capture, CloudWatch Logs, KMS.
- Encryption at rest: KMS for S3 captured data, EBS volumes, CloudWatch Logs.
- Encryption in transit: TLS for endpoint invocations; VPC endpoints (PrivateLink) for SageMaker, S3, CloudWatch, KMS.
- Data privacy: Captured data may contain PII. Use sampling, redaction, bucket policies, access logging. Ensure compliance (GDPR, HIPAA) before capture.
Anti-Patterns & Exam Gotchas
Shadow responses are never returned to callers. Only production responses go back. Shadow testing is not A/B testing. A/B splits traffic and returns candidate responses to some users. Shadow testing is only for real-time endpoints. Not supported for Serverless Inference, Asynchronous Inference, or Batch Transform. Shadow testing does not compute model quality automatically. You must capture data and evaluate offline. Shadow variant costs money. You pay for both production and shadow instances.
- Anti-pattern: Using shadow test to measure user satisfaction or business impact.
- Anti-pattern: Forgetting Data Capture, then having no data to compare.
- Anti-pattern: Not setting per-variant CloudWatch alarms; shadow failures go unnoticed.
- Anti-pattern: Running too short; missing traffic seasonality.
- Anti-pattern: Comparing raw counts without accounting for sampling percentage.
- Anti-pattern: Assuming shadow variant uses same preprocessing; ensure feature pipelines match.
GenAI/FM Shadow Considerations
- Capture prompts/responses from shadow and production. Evaluate with Bedrock Evaluations: programmatic (ROUGE, BLEU, BERTScore), LLM-as-a-judge, human.
- RAG: Measure retrieval accuracy (context precision/recall) separately from generation (faithfulness, answer relevance). Shadow both retrieval and generation components if possible.
- Human-in-the-loop: Use captured shadow responses for offline human review; no user impact.
- Prompt Management: Version and compare prompt variants; log in MLflow for reproducibility.
Decision Matrix: When to Use Shadow
| Requirement | Best Approach |
|---|---|
| Validate model on real traffic without user risk | Shadow |
| Measure business impact (CTR, conversion) | A/B |
| Gradual release with rollback | Canary |
| Instant cutover | Blue/Green |
| No live traffic, cost-sensitive | Offline evaluation |
| Compare GenAI output quality | Shadow + Bedrock Evaluations + human review |