Skill 4.2.1: Select inference instance families to optimize performance and cost
Task 4.2: Operating, Monitoring, and Securing ML and AI Solutions – Optimize and manage ML and AI infrastructure costs and performance
Core Objective: Match compute (CPU/GPU/accelerator, memory, network) to model + traffic pattern to minimize cost per inference while meeting latency, throughput, and reliability SLAs.
Primary AWS Business Objective: Eliminate over-provisioning, align capacity with demand, and select the cheapest instance family that satisfies model performance requirements.
Instance Family Decision Matrix
| Family Category | AWS Examples | Best For | Avoid When |
|---|---|---|---|
| General Purpose | ml.m5, ml.m6i, ml.m7g (Graviton) | CPU-bound models, small/medium models, low-to-moderate traffic | Large deep learning models needing GPU |
| Compute Optimized | ml.c5, ml.c6i, ml.c7g | Compute-heavy CPU inference, batch scoring | Memory-bound models |
| Memory Optimized | ml.r5, ml.r6i, ml.r7g, ml.x2 | Large in-memory models, high memory bandwidth, NLP/embedding | Compute-bound or GPU-accelerated |
| Accelerated GPU | ml.p3, ml.p4, ml.g4dn, ml.g5 | Deep learning, real-time, large models, high throughput | Small models, cost-sensitive, intermittent traffic |
| Inferentia | ml.inf1, ml.inf2 | DL inference at low cost, high throughput, TF/PyTorch via Neuron | Models with unsupported custom ops |
| Trainium | ml.trn1 | Training (not typical for inference) | Inference unless explicitly validated |
Gotcha: Amazon Bedrock on-demand inference has no instance families; cost is token-based. Instance selection applies to SageMaker hosted endpoints.
SageMaker Inference Options vs Instance Selection
| Option | Instance Selection | Cost Driver | When to Use | Anti-Pattern |
|---|---|---|---|---|
| Real-time Endpoint | Yes | Provisioned instance hours (even idle) | Steady, interactive, low-latency | Spiky/intermittent traffic |
| Serverless Inference | No | Request duration + concurrency | Intermittent, unpredictable | Low-latency at high scale |
| Asynchronous | Yes | Provisioned instance hours | Large payloads, long processing, queued | Real-time interactive |
| Batch Transform | Yes | Instance hours for job duration | Offline, large datasets | Low-latency serving |
| Multi-Model Endpoint (MME) | Yes (shared) | Shared instance hours | Many small models, intermittent | High-traffic models (cold starts) |
Gotcha: Real-time endpoints bill for provisioned instances even when idle. Serverless scales to zero but has cold starts.
Key Selection Criteria
- Model Size & Memory: Must fit in CPU RAM or GPU VRAM. Check model artifact size + runtime overhead.
- Compute Type: DL → GPU/Inferentia; tree/linear models → CPU (Graviton often cheapest).
- Latency SLA: GPU/Inferentia for <100ms; Batch for >minutes.
- Throughput: Larger batch sizes benefit from GPU/Inferentia; CPU for small batches.
- Traffic Pattern: Steady → real-time + auto scaling + Savings Plans; Spiky → serverless/async; Intermittent → serverless or MME.
- Cost per Inference: Benchmark with SageMaker Inference Recommender.
- Compilation: SageMaker Neo + Neuron for Inferentia; TensorRT for GPU; OpenVINO for CPU.
SageMaker Inference Recommender
- Purpose: Automates load testing and instance type recommendation.
- Modes: Default, Advanced, Custom.
- Outputs: Latency, throughput, cost per hour, cost per inference.
- Use: Avoid guesswork when selecting instance family and size.
Auto Scaling & Right-Sizing
- Metrics: CPUUtilization, GPUUtilization, MemoryUtilization, InvocationsPerInstance, ModelLatency.
- Policy: Target tracking scaling (e.g., InvocationsPerInstance).
- Right-Size: Analyze CloudWatch utilization over a representative window before changing.
- Scale In/Out: Add/release instances based on demand.
- Scale Up/Down: Change instance family/size for persistent mismatch.
Gotcha: An endpoint busy 1 hour/day and idle rest has a scaling problem, not a sizing problem.
Purchasing Options
| Option | Covers | Best For |
|---|---|---|
| On-Demand | All | Variable/experimental |
| SageMaker AI Savings Plans | SageMaker ML instances | Steady year-round baseline (up to 64% discount) |
| Compute Savings Plans | EC2, Lambda, Fargate | NOT SageMaker |
| EC2 Instance Savings Plans | EC2 only | NOT SageMaker |
| Spot | EC2 (training/batch) | NOT real-time inference |
| Provisioned Throughput | Bedrock FMs | Predictable high-volume FM inference |
Gotcha: SageMaker AI Savings Plans cover SageMaker endpoints. Compute/EC2 Savings Plans do NOT cover SageMaker.
Security & Compliance (Instance-Agnostic)
- IAM Roles: Least privilege for SageMaker execution.
- KMS: Encryption at rest for model artifacts, EBS volumes, endpoint data.
- TLS: Encryption in transit for endpoint invocations.
- VPC Endpoints (PrivateLink): Private inference without internet.
- Network Isolation: No internet access for training/inference containers.
Decision Flow
- Model type? DL → GPU/Inferentia; ML → CPU.
- Latency? Real-time → GPU/Inferentia/CPU; Batch → any.
- Traffic? Steady → real-time + auto scaling + Savings Plans; Intermittent → serverless; Many models → MME.
- Size? Fit memory.
- Benchmark with Inference Recommender.
Cost Optimization Levers
- Right-size instance family from utilization data.
- Auto scale to demand.
- Inferentia for DL inference (lower cost per inference).
- Graviton for CPU inference.
- Serverless for intermittent traffic.
- MME for many small models.
- Savings Plans for steady baseline.
- Compile with Neo/Neuron/TensorRT.
- Monitor with CloudWatch and Cost Explorer.
Anti-Patterns
- Over-provisioning GPU for small/CPU-bound models.
- Ignoring auto scaling for diurnal traffic.
- Real-time endpoints for spiky/intermittent traffic.
- Batch Transform for low-latency serving.
- MME for high-traffic models (cold starts).
- Forgetting cold start in serverless/MME.
- Applying EC2 Savings Plans to SageMaker.
- Not tagging resources for cost attribution.
Exam Gotcha: Inferentia (ml.inf1/inf2) is for inference; Trainium (ml.trn1) is for training. Don't confuse.