Skip to content

Skill 3.2.6: Select specific metrics for auto scaling implementations

Task 3.2: Deployment and Orchestration of ML and AI Workflows – Provision and configure resources for ML and AI workloads based on existing architecture and requirements.

Core objective: Choose the CloudWatch metric that best represents the workload’s scaling driver so Application Auto Scaling adds/removes SageMaker endpoint capacity at the right speed without over/under-provisioning.

Scaling Building Blocks

  • Application Auto Scaling — registers a scalable target: endpoint variant or inference component. Not EC2 Auto Scaling.
  • Target tracking — maintains metric at a target value. Default choice.
  • Step scaling — CloudWatch alarm thresholds with steps. Aggressive/custom control.
  • Scheduled scaling — time-based. Predictable peaks only.
  • Cooldowns — scale-out and scale-in are separate; prevent flapping. Default ~300s for target/step.

Metric Selection Matrix

Metric Resolution Best for Avoid when Scales
SageMakerVariantInvocationsPerInstance Standard (1 min) General real-time, steady request rate, short requests Streaming/long requests, sub-minute bursts Endpoint variant
SageMakerVariantConcurrentRequestsPerModel High-res (10 s) Latency-sensitive, streaming, long-running requests, rapid surges Stable low-traffic workloads where cost > latency Endpoint variant
SageMakerInferenceComponentConcurrentRequestsPerCopy High-res (10 s) Multi-model/inference components, per-copy concurrency Single-model variant Inference component
ApproximateBacklogSizePerInstance Standard Async inference, queue-based workloads Synchronous real-time Async endpoint variant
Custom CPU/GPU/memory utilization Custom Compute-bound workloads where request count misleads Without load testing proving correlation Variant/inference component
Scheduled action N/A Known daily/weekly peaks Unpredictable spikes Any scalable target

Workload → Metric Decision

  • Steady real-time endpoint → InvocationsPerInstance. Target = avg invocations per instance per minute.
  • Streaming / long-running inference → ConcurrentRequestsPerModel (high-res). Invocation count can stay low while in-flight concurrency is high.
  • Async inference → ApproximateBacklogSizePerInstance. Scale on queue depth per instance, not invocation count.
  • Inference components / multi-model → ConcurrentRequestsPerCopy or per-copy invocation metric. Scale copies.
  • GPU compute-bound → Custom GPU utilization (and memory). Validate that utilization moves with capacity need.
  • Predictable peaks → Scheduled scaling + target tracking for baseline.
  • Serverless inference → auto scales natively; do not use instance-count metrics. Provisioned concurrency addresses cold starts.

Cooldowns & Target Values

  • Scale-out cooldown — too long delays burst response; shorten for streaming/high-res metrics.
  • Scale-in cooldown — too short causes flapping; longer protects stability but costs more.
  • Target value — set from load testing. Too high = under-provisioning; too low = over-provisioning.
  • High-resolution metrics — 10s cadence lets target tracking scale out much sooner than 1-min invocation metrics.

Anti-Patterns & Traps

  • InvocationsPerInstance for streaming — metric lags concurrency; scale-out too late.
  • CPU/GPU utilization alone — may not correlate with request load; can scale on idle GPU or miss queue growth.
  • Scheduled scaling for unpredictable spikes — no reaction to actual demand.
  • Long cooldowns on bursty workloads — capacity arrives too late.
  • Scaling serverless on instance metrics — serverless auto scales; use provisioned concurrency.
  • Scaling the endpoint as a whole — scalable target is variant or inference component.
  • Ignoring scale-in — cost waste; configure scale-in cooldown and target.

Security / Networking Note

  • Predefined SageMaker metrics are emitted by the service; endpoint containers do not need outbound access for them.
  • Custom metrics published from a VPC-isolated endpoint require a path to CloudWatch (VPC endpoint or NAT).
  • IAM: Application Auto Scaling uses a service-linked role; endpoint execution role still needs ECR/S3/KMS permissions.

Exam Gotchas

Application Auto Scaling ≠ EC2 Auto Scaling. SageMaker scalable targets are variants or inference components. Standard vs high-resolution: InvocationsPerInstance = 1 min. ConcurrentRequestsPerModel/Copy = 10 s. Use high-res for sub-minute surges. Async ≠ real-time: Async uses backlog per instance; real-time uses invocations or concurrency. Serverless endpoints auto scale; they do not use instance-count metrics. Metric must match cost driver: request rate → invocations; in-flight load → concurrency; queue → backlog; compute-bound → custom utilization. Target tracking is default; step scaling for aggressive/custom; scheduled for predictable peaks.

Quick Recall

  • Steady real-time → InvocationsPerInstance
  • Streaming/long → ConcurrentRequestsPerModel
  • Async → ApproximateBacklogSizePerInstance
  • Inference components → ConcurrentRequestsPerCopy
  • GPU compute-bound → Custom GPU utilization
  • Predictable peaks → Scheduled + target tracking
  • Cold start → Provisioned concurrency, not scaling metric