Skill 3.1.3: Select model inference strategies (for example, real-time and batch processing)
Task 3.1: Deployment and Orchestration of ML and AI Workflows – Manage deployment infrastructure for ML and AI model types
Core decision: Who is waiting? How fast? How does work arrive? The answers dictate the inference strategy.
SageMaker AI Inference Options – Decision Matrix
| Option | Caller waits? | Latency | Payload / Timeout | Traffic pattern | Cost model | Key constraints |
|---|---|---|---|---|---|---|
| Real-time endpoint | Yes (synchronous) | ms | 6 MB / 60 s | Steady or spiky | Per instance-hour + auto scaling | Supports GPU, VPC, MME/MCE, inference components; scale-to-zero possible but cold start on scale-out |
| Serverless inference | Yes (synchronous) | ms–s (cold start) | 4 MB / 60 s | Intermittent, true idle gaps | Per request + duration | No GPU, no VPC, no MME/MCE; Provisioned Concurrency can keep warm |
| Asynchronous inference | No (queued) | near real-time | 1 GB / up to 1 h | Dynamic, long-running | Per instance-hour; scales to zero | Output to S3, SNS notifications; managed queue |
| Batch transform | No (offline) | job duration | N/A (dataset) | Pre-assembled bulk, scheduled | Per instance-hour during job | Reads from S3, writes to S3; no endpoint; no real-time requests |
Gotcha: Real-time and Serverless both enforce a 60-second timeout. If processing exceeds this, use Asynchronous or Batch.
Bedrock FM Inference Strategies
| Mode | Caller waits? | Use case | Capacity | Cost |
|---|---|---|---|---|
| On-demand | Yes | Interactive, variable traffic | Shared, may throttle | Per token |
| Provisioned Throughput | Yes | High, predictable throughput | Reserved model units | Hourly commitment |
| Batch Inference | No | Large datasets, offline | Asynchronous job | Up to 50% cheaper |
| Custom Model Import | Yes (on-demand) | Bring your own open-weight model | Serverless, on-demand | Per token |
Gotcha: Models customized inside Bedrock require Provisioned Throughput. Imported models via Custom Model Import are invoked on-demand.
Decision Flow – Select the Right Strategy
- Is a caller blocked waiting?
- Yes → Real-time endpoint (SageMaker) or Bedrock On-demand/Provisioned Throughput
- No → continue
- Is the dataset pre-assembled and processed in bulk?
- Yes → Batch Transform (SageMaker) or Bedrock Batch Inference
- No → continue
- Are requests individual, long-running (>60s), or large payload (>6 MB)?
- Yes → Asynchronous Inference
- No → continue
- Is traffic intermittent with true idle gaps?
- Yes → Serverless Inference (accept cold starts or pay for Provisioned Concurrency)
- No → Real-time endpoint with Auto Scaling
Endpoint Packing: Multi-Model vs Multi-Container
| Feature | Multi-Model Endpoint (MME) | Multi-Container Endpoint (MCE) |
|---|---|---|
| Assumption | Many models, same framework | Different frameworks/containers |
| Container | One shared serving container | Up to 15 containers |
| Invocation | Model name in request | Direct by container name or pipeline |
| Load behavior | Loads/unloads models from memory on demand; cold start on first invocation | Containers run in sequence (pipeline) or independently |
| Use case | Thousands of small models, uneven traffic | Ensemble, preprocessing + inference, different frameworks |
Gotcha: A model with much higher throughput or tighter latency than its neighbors should get its own endpoint—sharing a fleet shares its limits.
Key Configurations
- Real-time: Auto Scaling on
InvocationsPerInstance; inference components for scale-to-zero; VPC; GPU; MME/MCE. - Serverless: Provisioned Concurrency to avoid cold starts; memory size; no VPC/GPU/MME.
- Asynchronous: S3 output, SNS notifications, max concurrent invocations, scale-to-zero.
- Batch Transform: Instance count/type, S3 input/output, split/join, max payload.
- Bedrock: On-demand quotas, Provisioned Throughput model units, Batch Inference job, Custom Model Import from S3.
Security & Compliance
- IAM roles: SageMaker execution role for S3, ECR, KMS. Bedrock needs IAM for model invocation and S3 for batch/custom import.
- Encryption: KMS at rest for S3, ECR, SageMaker endpoints, Bedrock custom models. TLS in transit for all API calls.
- Network: Real-time and async SageMaker endpoints support VPC. Serverless does not. Use VPC endpoints (PrivateLink) for SageMaker and Bedrock to avoid public internet.
- Batch: S3 input/output encrypted with KMS; job runs in your VPC if configured.
Anti-Patterns & Exam Traps
- Real-time for batch: Paying for idle endpoints; use Batch Transform.
- Batch for real-time: No endpoint to answer live requests; use Real-time.
- Serverless for steady low-latency: Cold starts violate latency; use Real-time with Auto Scaling.
- Serverless for GPU/VPC/MME: Not supported; use Real-time.
- Asynchronous when caller needs immediate answer: Caller gets queued; use Real-time.
- Provisioned Throughput for spiky unpredictable traffic: Over-provisioning cost; use On-demand.
- On-demand for high sustained throughput: Throttling and higher cost; use Provisioned Throughput.
- Assuming scale-to-zero on real-time is a single setting: Requires inference components + auto scaling policy on the no-capacity metric.
- Forgetting payload limits: Real-time 6 MB, Serverless 4 MB, Async 1 GB. Exceeding causes errors.
Quick Recall
Who is waiting? → Real-time (yes) vs Async/Batch (no).
How does work arrive? → Pre-assembled bulk → Batch; individual queued → Async; intermittent → Serverless; steady/spiky → Real-time.