Skip to content

Skill 3.2.10: Implement AI-specific resource scaling for GPU workloads

Task 3.2: Deployment and Orchestration of ML and AI Workflows — Provision and configure resources for ML and AI workloads based on existing architecture and requirements.

The Scaling Engine (mental model)

  • Application Auto Scaling is the only scaler for SageMaker endpoints/inference components — not EC2 Auto Scaling groups, not CloudFormation.
  • Flow: register scalable target (variant or inference component) → attach scaling policy → policy reads a CloudWatch metric → adjusts instance count or copy count between MinCapacity/MaxCapacity.
  • Business objective: hold the smallest GPU fleet that still meets latency SLO — GPU capacity is the dominant cost, and GPU quota is the dominant constraint.
  • Two scaling dimensions: instance count (endpoint variant / HyperPod nodes) vs copy count (inference components sharing one GPU host).

Scalable Targets & Metrics Decision Matrix

Target Predefined metric Resolution Best for
Endpoint variant (real-time GPU) SageMakerVariantInvocationsPerInstance 1 min Request-count-driven, short-completing inference
Inference component SageMakerInferenceComponentInvocationsPerCopy; ...ConcurrentRequestsPerCopy 1 min / 10 s Multi-model GPU packing, per-model scaling, scale-to-zero
Async inference endpoint ApproximateBacklogSizePerInstance (custom, from SQS queue depth) 1 min Queue/batch GPU jobs with bursty arrival
Serverless endpoint None (no scalable target) — No GPU support; only ProvisionedConcurrency (manual/scheduled)
HyperPod cluster DCGM/GPU util, node metrics as published Long training/fine-tuning runs needing resilient accelerator nodes
EKS/ECS GPU service HPA / Application Auto Scaling on DCGM util as published Self-managed containerized GPU inference
Scaling policy Use when Trap
Target tracking Default; metric scales proportionally with load Needs a high-resolution metric to react in seconds
Step scaling Need tiered/burst response; requires a CloudWatch alarm Slower than target tracking; alarm evaluation period adds lag
Scheduled scaling Predictable peaks (e.g., nightly batch scoring window) Does not react to unexpected spikes — combine with target tracking
Predictive scaling — Not supported for SageMaker targets (distractor)

ConcurrentRequestsPerModel / ConcurrentRequestsPerCopy are the only high-resolution (10-second) SageMaker metrics. They are the answer whenever the stem says streaming responses, long-lived requests, or "scales too late on a per-minute metric."

GPU-Specific Mechanics

  • Accelerator families to recognize: ml.g4dn/ml.g5 (T4/A10G — cost-efficient inference), ml.p4d/ml.p4de/ml.p5 (A100/H100 — large-model training & inference), ml.inf2/ml.trn1 (Inferentia/Trainium — cheapest per-inference, not GPUs).
  • GPU utilization is not published by SageMaker. To scale on it you must publish a custom metric (CloudWatch agent + NVIDIA DCGM exporter, or Container Insights). Validate the metric tracks capacity before production — nothing guarantees proportionality.
  • Multi-Instance GPU (MIG) on A100/H100 partitions one physical GPU into isolated slices → pair with inference components to host several small models per accelerator.
  • Inference components are the recommended GPU-efficiency pattern: multiple models/copies on one instance, independent per-model scaling, and scale-to-zero when MinInstanceCount = 0.
  • Multi-model endpoints (MME) pack many models behind one container for GPU cost amortization — but scale at the endpoint level, not per model.
  • Async inference decouples GPU burst load via SQS; scale on backlog size per instance, not invocations.

Acquiring GPU Capacity (Cost & Availability)

Option When Exam signal
On-Demand Unpredictable, short-lived Default; highest cost
Savings Plans / Reserved Instances Steady 1–3 yr baseline Predictable, committed traffic
On-Demand Capacity Reservation Guarantee capacity in an AZ now "Must guarantee GPU availability in this AZ"
Capacity Blocks for ML Future-dated reserved GPU window (P4d/P5/Trn1, day-granular) "Training run scheduled weeks out, need guaranteed GPUs"
Spot / Managed Spot Training Interruption-tolerant training Checkpointing required
  • Scale-out reacts to load; scale-in follows a cooldown. Default cooldown is 300 s in both directions.
  • Set scale-in cooldown longer than scale-out to prevent flapping — a burst that ends immediately should not tear down warm GPU capacity that the next burst needs.

Prerequisites & Networking (scaling fails without these)

  • Private networking: real-time GPU endpoints take subnets + security groups; at least 2 subnets in 2 different AZs, even for one instance.
  • Execution role needs ec2:CreateNetworkInterface, DescribeNetworkInterfaces, DeleteNetworkInterface; private subnets need an S3 gateway endpoint (artifacts) and ECR/interface endpoints (image pull) or a NAT gateway.
  • Images must live in ECR with immutable/versioned tags — a moving tag breaks reproducible redeploys; enable ECR KMS encryption for compliance.
  • Quotas: GPU instance quotas are per instance family and frequently 0 or very low on new accounts — request the increase before the deployment, not during the incident.

Serverless Inference accepts no VPC configuration and offers no GPU instances. Any requirement for private networking or accelerators eliminates serverless before any other option is considered.

HyperPod & Training-Side Scaling

  • SageMaker HyperPod provisions resilient accelerator clusters with automatic faulty-node replacement and job auto-resume — the answer for long training runs that must survive hardware failure.
  • Orchestration choice: Amazon EKS (Kubernetes) or Slurm — scaling via Karpenter/Cluster Autoscaler + HPA (EKS) or Slurm node management.
  • Task governance on HyperPod enforces fair-share priority across teams competing for the same GPU pool.

Anti-Patterns & Distractor Traps

  • Scaling on GPU memory utilization — memory is not released between requests; it will scale out and never in.
  • Target tracking on a 1-minute invocation metric for a streaming LLM — reacts a minute too late; use high-resolution concurrency.
  • Increasing cooldown or raising the target value to "fix slow scaling" — both make scaling slower; they are the wrong-direction answers.
  • Scheduled scaling as the answer to an unexpected spike — scheduled handles known peaks only.
  • Copying subnet IDs as stack parameters instead of exported stack outputs imported cross-stack — parameters drift; exports stay linked.
  • EC2 Auto Scaling groups / CloudFormation / ECR offered as the scaling mechanism — none of them monitor or adjust runtime capacity.
  • Deploying GPU workloads without a quota increase, or ignoring multi-AZ subnet requirements — the two most common "endpoint won't come up / can't scale" causes.