Skill 3.1.1: Select appropriate compute environments and deployment targets
Task 3.1: Deployment and Orchestration of ML and AI Workflows – Manage deployment infrastructure for ML and AI model types
Core objective: Place each model on the smallest infrastructure that satisfies latency, payload, traffic, control, and compliance requirements—while minimizing operational overhead and cost.
1. Core Decision Signals
| Signal |
Ask |
Implication |
| Who waits? |
Is a user/process blocked? |
Real-time/Serverless = synchronous. Async/Batch = no live caller. |
| Latency |
ms? seconds? minutes? |
Real-time/Serverless for ms–s. Async/Batch for long-running. |
| Payload |
<4/6 MB? >6 MB? up to 1 GB? |
Real-time ≤6 MB; Serverless ≤4 MB; Async ≤1 GB; Batch uses S3 objects. |
| Traffic |
Steady, spiky, idle gaps, scheduled bulk? |
Real-time + Auto Scaling; Serverless for idle; Batch for schedules; Async for queues. |
| Ops control |
Managed vs self-managed? |
SageMaker/Bedrock vs ECS/EKS/EC2/Lambda. |
| Compliance |
VPC, KMS, private connectivity? |
VPC endpoints, KMS CMK, IAM roles. Serverless does not support VPC. |
2. SageMaker AI Inference Options
| Option |
Workload pattern |
Limits |
Scaling / cost |
Constraints & gotchas |
| Real-time |
Interactive, steady or spiky, ms latency |
≤6 MB; 60 s timeout |
Auto Scaling; pay for instances |
Supports GPU, VPC, MME, MCE. Scale-to-zero only with inference components + scale-out policy; requests fail until capacity returns. |
| Serverless |
Intermittent, true idle gaps, cost-sensitive |
≤4 MB; 60 s timeout |
Scales to zero; per-invocation billing |
No GPU, no VPC, no MME. Cold starts after idle unless provisioned concurrency is configured. |
| Asynchronous |
Queued requests, large payloads, long processing |
≤1 GB; up to 15 min |
Queue holds work; scales to zero when empty |
Returns 202 Accepted; output to S3; optional SNS. Supports VPC/GPU. No live caller blocked. |
| Batch Transform |
Pre-assembled dataset, scheduled bulk |
S3 input/output; no endpoint |
Compute starts, processes, stops |
No live caller. Not for dynamic individual requests. Pay only during job. |
Exam gotcha: Real-time and Serverless are synchronous with a 60-second timeout. If requirement says >6 MB or >60 s, choose Async (individual requests) or Batch (dataset).
Exam gotcha: Serverless = no VPC, no GPU, no multi-model. Real-time scale-in to zero requires inference components and a scale-out policy; otherwise it stays warm.
3. Endpoint Packing: Single vs Multi-Model vs Multi-Container
| Strategy |
Assumption |
Use when |
Trade-off |
| Single-model |
One model, dedicated fleet |
High throughput, strict latency, isolation, unique framework |
Highest cost for many models |
| Multi-model (MME) |
Many models, same framework/container, shared fleet |
Hundreds/thousands of small models, uneven traffic |
Dynamic load/unload from S3; cold start on first invoke for rare models; not on Serverless |
| Multi-container (MCE) |
Up to 15 different containers/frameworks |
Different frameworks or inference pipeline |
Not for many small models; more complex; containers can run sequentially or be invoked by name |
Exam gotcha: MME shares one serving container → same framework only. MCE supports different containers → use for framework diversity or pipeline. A model with much higher throughput or tighter latency than neighbors belongs on its own endpoint.
4. Foundation Model Deployment Targets
| Target |
Best for |
Capacity model |
Gotchas |
| Bedrock on-demand |
Managed FM API, variable traffic, no endpoint ops |
Pay per token/request |
Model access must be enabled; no instance sizing; quotas apply. |
| Bedrock Provisioned Throughput |
Reserved FM capacity, steady throughput, latency predictability |
Reserved model units |
Required for custom models fine-tuned inside Bedrock. Higher cost commitment. |
| Bedrock Custom Model Import |
Bring open-weight custom model from S3 into Bedrock |
On-demand invocation |
Only supported architectures/file layouts. No endpoint ops; served via Bedrock API. |
| SageMaker JumpStart |
Pretrained model catalog, quick deploy |
SageMaker endpoint instances |
Preconfigured; still manage endpoint/scaling. |
| SageMaker self-hosted FM |
Full control over accelerator, container, scaling, VPC |
Instance/accelerator |
You manage ops, patching, scaling, cost. |
| EC2/ECS/EKS |
Custom containers, existing orchestrator, special networking |
Self-managed |
No SageMaker endpoint features; you manage everything. |
Exam gotcha: Bedrock fine-tuned custom models require Provisioned Throughput. Custom Model Import is on-demand. Confirm supported architectures before migration.
5. Self-Managed Compute Targets
| Target |
Best for |
Avoid when |
| Lambda |
Event-driven glue, lightweight inference, short tasks |
Long ML inference, GPU, large payloads, >15 min |
| ECS |
Long-lived container services, scheduled tasks, self-managed inference |
You want managed endpoint features or no Kubernetes requirement |
| EKS |
Existing Kubernetes manifests/controllers, multi-team cluster |
Simple model serving or no K8s expertise |
| EC2 |
Full control, custom accelerators, special networking, legacy |
Managed hosting satisfies latency/cost/ops and you don't need control |
6. Security, Networking & Compliance
- IAM: Execution roles for SageMaker, Bedrock, ECS/EKS tasks, Lambda. Least privilege. Separate training vs inference roles.
- KMS: Encrypt model artifacts, S3 input/output, EBS volumes, Bedrock custom models. Use CMK for compliance.
- VPC: SageMaker real-time/async/batch support VPC. Serverless does not. Use VPC endpoints (PrivateLink) for SageMaker, Bedrock, S3, ECR, CloudWatch.
- Network isolation: Disable internet access for containers; use VPC endpoints.
- Bedrock: Model access, guardrails, VPC endpoints. Custom Model Import requires S3 access.
- Agents: AgentCore provides runtime, memory, gateway (tools via MCP), identity, observability.
7. Anti-Patterns & Exam Traps
- Real-time for long jobs: 60 s timeout → use Async or Batch.
- Serverless for strict latency: Cold start after idle; no VPC/GPU/MME.
- Batch for individual requests: Batch expects S3 dataset, not dynamic queue.
- MME for different frameworks: MME shares serving container; use MCE or separate endpoints.
- MCE for many small models: MCE up to 15 containers; MME is for many same-framework models.
- Self-hosting when Bedrock suffices: Bedrock removes instance sizing; use when model is supported and no custom container needed.
- Ignoring payload limits: Real-time 6 MB, Serverless 4 MB, Async 1 GB.
- Forgetting Provisioned Throughput for Bedrock custom models: Fine-tuned in Bedrock requires reserved capacity.
- Assuming Serverless supports VPC: It does not.
- Scaling real-time to zero without inference components: Requests fail until capacity returns.
8. Quick Decision Matrix
| Requirement |
Best target |
| Interactive, ms latency, steady/spiky |
SageMaker Real-time + Auto Scaling |
| Intermittent, idle gaps, cold start OK |
SageMaker Serverless |
| Queued requests, large payload, long processing |
SageMaker Async |
| Bulk dataset, scheduled, no live caller |
SageMaker Batch Transform |
| Many small models, same framework, uneven traffic |
SageMaker Multi-Model Endpoint |
| Different frameworks/containers, pipeline or direct invoke |
SageMaker Multi-Container Endpoint |
| Managed FM API, variable traffic |
Bedrock on-demand |
| Reserved FM capacity, steady throughput |
Bedrock Provisioned Throughput |
| Bring open-weight custom FM to Bedrock |
Bedrock Custom Model Import |
| Pretrained model quick deploy |
SageMaker JumpStart |
| Event-driven glue, lightweight inference |
Lambda |
| Long-lived container service, self-managed |
ECS |
| Existing Kubernetes control plane |
EKS |
| Full control, custom accelerator/network |
EC2 |