Skill 3.2.1: Optimize resource provisioning between on-demand and provisioned resources for performance and cost efficiency
Task 3.2: Deployment and Orchestration of ML and AI Workflows - Provision and configure resources for ML and AI workloads based on existing architecture and requirements
Core Objective: Match capacity model to traffic predictability and criticality. On-demand = elastic, pay-per-use, higher unit cost. Provisioned = committed capacity, lower unit cost, predictable performance.
Hosting Options: Provisioning & Scaling Trade-offs
| Option | Provisioning Model | Scaling | Cost Profile | Best For |
|---|---|---|---|---|
| SageMaker Real-time | Provisioned instances (min ≥1) | Application Auto Scaling (target tracking, step, scheduled) | Per instance-hour; Savings Plans | Low-latency, steady/high traffic |
| SageMaker Serverless | On-demand; optional Provisioned Concurrency | Automatic, scales to zero | Per ms + per request; PC adds cost | Intermittent/spiky, cold start tolerable |
| SageMaker Asynchronous | Provisioned instances; can scale to zero | Auto Scaling; queue-based | Per instance-hour when running | Large payloads, long processing |
| SageMaker Batch Transform | On-demand instances | Job-based | Per instance-hour; no endpoint | Offline batch, cost-sensitive |
| Bedrock On-Demand | On-demand | Automatic | Per token | Variable/low throughput |
| Bedrock Provisioned Throughput | Dedicated model units | Fixed (commitment) | Hourly per model unit; ⅙-month term | High, predictable throughput |
Training Provisioning
- On-Demand: No interruption, higher cost. Use for critical, time-sensitive, or short jobs.
- Spot (Managed Spot Training): Up to 90% savings, 2-min interruption notice. Requires checkpointing to resume. Use for fault-tolerant, long training.
- Savings Plans / Reserved: Commit to steady-state usage for lower hourly rates.
- HyperPod: Provisioned resilient clusters; auto-replaces faulty hardware; for long training runs.
Auto Scaling Signals & Policies
- Target tracking: SageMakerVariantInvocationsPerInstance (general), ConcurrentRequestsPerModel (high-resolution, sub-minute), GPU utilization (custom).
- High-resolution concurrency metrics enable rapid scale-out for streaming/long inference.
- Cooldowns: Scale-out and scale-in cooldowns prevent thrashing. Set based on workload ramp-up.
- Scale to zero: Serverless and Asynchronous only. Real-time requires ≥1 instance.
- Right-size first: Use SageMaker Inference Recommender to choose instance type/count before committing.
Decision Matrix
| Traffic Pattern | Provisioning Choice | Rationale |
|---|---|---|
| Predictable high 24/7 | Provisioned / Savings Plans / Provisioned Throughput | Lowest unit cost, guaranteed capacity |
| Spiky / unpredictable | On-demand / Serverless | Pay per use, elastic |
| Fault-tolerant training | Spot | Cost savings with checkpointing |
| Critical training | On-Demand | No interruption |
| Low-latency inference | Real-time + Auto Scaling | Consistent latency |
| Intermittent inference | Serverless | Scale to zero, cost-efficient |
Exam Gotchas - Provisioned Throughput (Bedrock) requires 1- or 6-month commitment; not for spiky traffic. - Serverless scales to zero; Real-time does not (min 1 instance). - Spot instances can be interrupted; always checkpoint. - Savings Plans apply to SageMaker instance usage, not Serverless Inference or Bedrock tokens. - Auto Scaling requires registering a scalable target and attaching a policy; cooldowns are mandatory to avoid thrashing.
Anti-Patterns
- Over-provisioning for peak 24/7 → wasted cost.
- On-demand for steady high utilization → higher unit cost.
- Provisioned for highly variable traffic → unused capacity.
- No auto scaling on real-time endpoints → manual over-provisioning.
- Spot without checkpointing → lost work on interruption.
- Ignoring cold starts for Serverless when low latency is required.
Security & Compliance (Provisioned Resources)
- IAM roles for execution; least privilege.
- VPC private subnets + VPC endpoints for private access to S3, ECR, etc.
- KMS encryption at rest; TLS in transit.
- Network isolation may dictate hosting option (e.g., Serverless has no VPC config).