Skill 3.2.5: Deploy and host models programmatically (for example, by using the SageMaker AI SDK for Python, AWS CLI, Boto3)
Task 3.2: Deployment and Orchestration of ML and AI Workflows - Provision and configure resources for ML and AI workloads based on existing architecture and requirements.
Core Objective & Mental Model
- Business goal: Replace manual console hosting with repeatable, auditable, versioned deployments that match existing architecture.
- Deployment unit: Model artifact + inference container + endpoint configuration + endpoint (or async/serverless/batch job).
- Programmatic success: Idempotent, parameterized, CI/CD-ready, with approval gates and rollback.
- Exam lens: Know which hosting option supports which requirement; know immutable/config constraints that force rebuilds.
Programmatic Surfaces: Choose the Right Layer
| Surface | Best for | Trade-off / Trap |
|---|---|---|
| High-level SageMaker AI SDK | Notebooks, quick deploys, pipelines | Abstracts model/config/endpoint; less per-field control |
| Low-level SDK / Boto3 client | CI/CD, custom orchestration, fine control | Imperative; must poll status/waiters; handle idempotency |
| AWS CLI | Scripts, admin, simple automation | No state; drift if not idempotent; not declarative |
| CloudFormation / CDK | Repeatable environments, compliance, cross-stack | Slower for model iteration; template/CDK synthesis |
| SageMaker Pipelines / Step Functions | Multi-step ML workflows, approvals, retries | Orchestration layer, not a hosting runtime |
Deployment Sequence & Key Config
| Step | API operation | What to configure | Exam-critical nuance |
|---|---|---|---|
| Model | CreateModel | Container image, model artifacts S3, env vars, execution role, VPC | Role needs ECR pull, S3 read, CloudWatch logs, ENI create/attach for VPC |
| Endpoint config | CreateEndpointConfig | Instance type/count, variant weights, data capture, async/serverless config, inference components | Serverless config cannot include VPC; async needs S3 output; multi-model uses components |
| Endpoint | CreateEndpoint | Real-time/async/serverless endpoint, tags, KMS | Endpoint creation is not instant; poll/wait for InService |
| Update | UpdateEndpoint | Blue/green, canary, linear, all-at-once, shadow | Traffic shifting via variants; rollback on alarms |
| Invoke | InvokeEndpoint, InvokeEndpointAsync, CreateTransformJob | Real-time, async, batch transform | Batch is offline; async for large/long; real-time for low latency |
Hosting Options Decision Matrix
| Requirement | Best option | Why / Trap |
|---|---|---|
| Low-latency real-time | Real-time endpoint | VPC support, auto scaling, GPU options |
| Intermittent / scale-to-zero | Serverless endpoint | No VPC config; cold starts; provisioned concurrency mitigates |
| Large payload / long processing | Async endpoint | Queue + S3 input/output; 202 response |
| Offline batch scoring | Batch Transform | No endpoint, no auto scaling |
| Private network only | Real-time or async + VPC | Serverless cannot join VPC |
| Many models, shared instance | Multi-model endpoint / inference components | Scale per component; pack models |
| GPU-heavy inference | Real-time GPU instances | Scale on GPU utilization, not CPU |
| Agent runtime | Bedrock AgentCore | Memory, gateway, identity, observability |
Networking, Security & Compliance
- IAM execution role: Least privilege; must allow ECR image pull, S3 model artifact read, CloudWatch Logs, and VPC ENI actions.
- VPC: Private subnets in at least two AZs; security groups; route tables.
- Private endpoints: S3 gateway endpoint, ECR/CloudWatch interface endpoints; otherwise endpoint fails to pull artifacts.
- Encryption: KMS for S3 artifacts, ECR images, EBS volumes; TLS in transit; use PrivateLink to avoid public internet.
- No public exposure: Put endpoint in private subnets; no internet gateway route needed if VPC endpoints exist.
Exam Gotcha: Serverless endpoints do not accept VPC configuration. If private networking is required, choose real-time or async endpoint.
Exam Gotcha: VPC subnets must span at least two Availability Zones even for a single hosting instance.
Exam Gotcha: Execution role must permit ENI create/describe/delete for VPC endpoints.
Scaling Programmatically
| Scaling signal | Use when | Trap |
|---|---|---|
| InvocationsPerInstance | General request-count driven | 1-minute resolution; slow for bursts |
| ConcurrentRequestsPerModel (high-res) | Streaming/long responses, rapid surges | Sub-minute; scales out faster |
| CPU/GPU utilization | Cost driver is compute, not requests | Custom metric; must validate proportionality |
| Scheduled scaling | Predictable calendar traffic | Does not react to unexpected spikes |
| Provisioned concurrency | Serverless cold-start sensitive | Not for real-time endpoints |
- Application Auto Scaling registers endpoint variants or inference components as scalable targets.
- Cooldowns prevent flapping; scale-out cooldown should be shorter than scale-in for bursty workloads.
- GPU workloads: Use GPU utilization or custom metrics; choose accelerator instance families; consider inference components for packing.
Automation & IaC
- CloudFormation/CDK: Declarative; export network outputs and import in hosting stack to avoid hardcoded subnet IDs.
- Cross-stack references: Keep network stack and hosting stack decoupled; rename in network stack does not silently break hosting.
- Orchestration: Step Functions for multi-service retries/approvals; SageMaker Pipelines for ML workflow; EventBridge for triggers.
- CI/CD: Build container in CodeBuild, push to ECR, register model in Model Registry, approve, deploy via pipeline.
- Idempotency: Use unique names/tags; avoid duplicate endpoints; poll/wait for terminal states.
Containers & ECR
- Prebuilt framework container: Fastest; use when framework and version match.
- Extended prebuilt container: Add custom dependencies; still inherit serving stack.
- Custom container from scratch: Full control; must implement required health and inference endpoints.
- ECR: Version images; avoid moving tags; endpoint pulls image at creation/update.
- Container requirements: Serve HTTP on expected port; health check path; inference path; write logs to stdout/stderr.
Deployment Strategies
| Strategy | Use case | Risk / Trap |
|---|---|---|
| All-at-once | Dev/test, no downtime requirement | Downtime; no rollback safety |
| Blue/green | Production, full validation before shift | Duplicate capacity cost |
| Canary | Small traffic to new model | Requires monitoring/alarms |
| Linear | Gradual shift over time | Slower rollout |
| Shadow | Test new model on live traffic without impact | Extra cost; no user impact |
Anti-pattern: Console-only deployments cause drift and are not auditable. Use IaC or pipelines for production.
Anti-pattern: Hardcoding subnet IDs or security group IDs across stacks. Use exported outputs/imports.
Anti-pattern: Using moving container tags like
latest. Version images in ECR for reproducible redeploys.Anti-pattern: Scaling GPU workloads on CPU utilization. Use GPU/custom metrics.