Skill 3.1.2: Select deployment orchestrators and multi-model or multi-container deployment strategies
Task 3.1: Deployment and Orchestration of ML and AI Workflows - Manage deployment infrastructure for ML and AI model types
Core Mental Model
- Hosting target = where the model runs: SageMaker endpoint, Bedrock FM API, ECS/EKS, Lambda, EC2.
- Orchestrator = what coordinates deployment: SageMaker Pipelines, Step Functions, EventBridge, ECS/EKS controllers, Lambda.
- Packing strategy = how many models/containers share one endpoint: single-model, multi-model, multi-container, Inference Components.
- Deciding signals: model count, framework homogeneity, latency/throughput isolation, existing control plane, cost, cold-start tolerance.
SageMaker Endpoints are a deployment target, not an orchestrator. Pipelines, Step Functions, EventBridge, ECS/EKS, and Lambda orchestrate endpoint creation, updates, and traffic shifts.
Deployment Orchestrator Selection
| Orchestrator | Best For | Avoid When |
|---|---|---|
| SageMaker Pipelines | ML-native CI/CD: train → evaluate → register → deploy; Model Registry approval | General multi-service workflows, non-ML DAGs |
| Step Functions | Multi-service workflows, retries, parallel branches, human approval, long-running jobs | Simple scheduled batch; ML-only lineage |
| EventBridge | Event-driven triggers, schedules, reacting to S3/CloudWatch/SageMaker events | Complex stateful orchestration |
| Lambda | Lightweight inference, glue code, event processing, invoking endpoints | Long-running >15 min, GPU, large model hosting |
| ECS | AWS-native container orchestration, long-lived services, scheduled tasks | Kubernetes-native teams |
| EKS | Kubernetes-native deployments, custom controllers, GPU scheduling, KServe/Kubeflow | Simple managed ML hosting |
| Bedrock | Managed foundation model API; no endpoint sizing | Custom traditional ML models; full serving control |
| Bedrock AgentCore | Production agents: runtime, memory, gateway, identity, observability | Simple single-model inference |
| SageMaker JumpStart | Deploy pretrained models to a preconfigured endpoint | Custom training/serving stacks |
Endpoint Packing: Single vs Multi-Model vs Multi-Container
| Strategy | Model/Framework Assumption | Invocation | Cost/Scaling | Best For | Gotchas |
|---|---|---|---|---|---|
| Single-model endpoint | One model per endpoint; any framework | Standard endpoint invoke | Dedicated resources; auto scaling; predictable latency | High-throughput, latency-sensitive, isolated models | Expensive for many low-traffic models |
| Multi-model endpoint (MME) | Many models share same framework/container | Select model via TargetModel | Shared fleet; models loaded/unloaded on demand | Hundreds/thousands of models, uneven traffic | Cold-start/load penalty; memory pressure; not serverless |
| Multi-container endpoint (MCE) | Up to 15 containers; different frameworks | Serial pipeline or direct invocation by TargetContainerHostname | Shared instance; per-container CPU/GPU/memory allocation | Pre/post-processing, ensemble, different frameworks | Not for mass model hosting; shared resource limits |
| Inference Components | Multiple models on one endpoint with independent containers | Per-component routing | Per-model scaling, resource allocation, scale-to-zero | Multi-model, A/B, canary, scale-to-zero needs | Requires endpoint + component design; newer feature |
MME assumes same framework/container. Different frameworks or dependencies → MCE or Inference Components. MCE max = 15 containers. Thousands of models → MME, not MCE. Serverless Inference does not support MME, MCE, VPC, GPU, or accelerated instances.
Multi-Model Endpoint (MME) Deep Dive
- Use case: many models, same framework, low/uneven traffic, cost reduction.
- Mechanism: artifacts in S3; endpoint loads model into memory on first invoke, unloads under pressure.
- Routing: TargetModel selects the model.
- Scaling: endpoint fleet scales; model load adds latency on cold start.
- Security: execution role needs S3 read access to model artifacts.
- Anti-pattern: different Python/CUDA/framework versions on same MME.
Multi-Container Endpoint (MCE) Deep Dive
- Serial inference pipeline: containers run in order, e.g., preprocess → model → postprocess. Final response from last container.
- Direct invocation: each container has a hostname; invoke specific container via TargetContainerHostname.
- Resource allocation: each container gets CPU/GPU/memory; total must fit instance.
- Use cases: different frameworks, custom preprocessing, ensembles, multi-stage inference.
- Limit: maximum 15 containers per endpoint.
Inference Components — Granular Multi-Model
- Purpose: deploy multiple models to one endpoint with independent scaling, resource allocation, and scale-to-zero.
- Use cases: multi-model hosting, A/B testing, canary deployments, cost optimization.
- Key advantage: per-model scaling policies; can scale to zero without dedicated endpoint per model.
- Gotcha: real-time endpoints generally cannot scale to zero without Inference Components + scale-to-zero policy.
Security & Compliance
- IAM: SageMaker execution role for S3, ECR, KMS, CloudWatch. MME needs S3; MCE needs ECR. EKS uses IRSA; ECS uses task role. Bedrock uses IAM for InvokeModel.
- KMS: encrypt model artifacts at rest in S3, EBS, ECR. TLS in transit for endpoint invocations.
- Network: VPC endpoints (PrivateLink) for SageMaker, S3, ECR, CloudWatch. Serverless inference does not support VPC.
- Audit: CloudTrail for API calls, CloudWatch for logs, Config for compliance, Bedrock Guardrails for FM safety.
Anti-Patterns & Exam Traps
MME for different frameworks → wrong; use MCE or Inference Components. MCE for thousands of models → wrong; use MME. Serverless for strict latency → cold starts; use real-time with auto scaling. Batch for dynamic requests → wrong; use async or real-time. Async for blocking user → wrong; use real-time. MME on serverless → not supported. MCE >15 containers → not allowed. Lambda for long-running/GPU → not supported. Real-time scale to zero without Inference Components → not possible. Step Functions for simple scheduled batch → use EventBridge + Batch Transform. SageMaker Pipelines for non-ML multi-service → use Step Functions. Bedrock Custom Model Import → on-demand only; specific open-weight architectures. Fine-tuned FM in Bedrock → requires Provisioned Throughput.
Quick Selection Matrix
| Requirement | Best Fit |
|---|---|
| Many models, same framework, uneven traffic | MME |
| Many models, different frameworks, independent scaling | Inference Components |
| Preprocessing + model + postprocessing | MCE serial inference pipeline |
| Independent containers invoked by name | MCE direct invocation |
| One high-throughput, low-latency model | Single-model endpoint |
| Managed foundation model API | Bedrock |
| Import open-weight FM | Bedrock Custom Model Import |
| Fine-tuned FM in Bedrock | Provisioned Throughput |
| ML CI/CD | SageMaker Pipelines |
| Multi-service workflow | Step Functions |
| Kubernetes-native | EKS |
| Simple container service | ECS |
| Event-driven glue | Lambda |