Skill 3.2.6: Select specific metrics for auto scaling implementations

Question 1 - High-Resolution Concurrency Metrics for Streaming Inference

A financial services company has deployed a fraud detection model on Amazon SageMaker AI real-time endpoint. The model processes credit card transactions in real time, where each inference request streams a response back to the client over a period of 30-60 seconds as new data points are analyzed.

During peak transaction periods, concurrent requests increase from 50 to 500 within seconds. The current auto scaling policy uses a target-tracking policy on SageMakerVariantInvocationsPerInstance, which only adds capacity after several minutes, causing request timeouts and rejected transactions. The company needs a scaling solution that reacts within seconds to sudden increases in simultaneous in-flight requests.

Which scaling configuration should the ML engineer implement to meet this requirement? (Select ONE)

A) Continue using SageMakerVariantInvocationsPerInstance with a lower target value of 5 invocations per instance.
B) Implement a target-tracking policy on the high-resolution metric SageMakerVariantConcurrentRequestsPerModel with a target value based on model capacity.
C) Create a custom CloudWatch alarm on CPUUtilization that triggers a step scaling policy when CPU exceeds 50%.
D) Implement scheduled scaling actions that increase instance count 10 minutes before known peak transaction periods.
E) Use Application Auto Scaling with a target-tracking policy on Average statistic of GPUUtilization aggregated over 5-minute periods.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: Amazon SageMaker AI publishes high-resolution concurrency metrics (such as ConcurrentRequestsPerModel and ConcurrentRequestsPerCopy) at 10-second intervals. When paired with Application Auto Scaling target tracking, these metrics allow the policy to detect rapid increases in in-flight concurrent requests and scale out within seconds—before existing instances are overwhelmed. This directly addresses the constraint of sub-minute traffic surges from streaming, long-lived requests.

Distractor Analysis

A is Incorrect: Lowering the target value for InvocationsPerInstance still relies on a per-minute metric, meaning the policy cannot detect rapid bursts before the one-minute aggregation window completes, leaving the endpoint unprotected during the first critical seconds of a traffic surge.
C is Incorrect: While CPUUtilization can reflect load, it is a custom resource-utilization metric that may not track request concurrency or in-flight request capacity accurately. It also aggregates over longer periods (1-minute minimum for standard CloudWatch), causing delayed reactions compared to native high-resolution concurrency metrics.
D is Incorrect: Scheduled scaling is deterministic and works only for predictable, calendar-based traffic patterns (e.g., daily 9 AM office traffic). It cannot react dynamically to sudden, unpredictable spikes in concurrent requests.
E is Incorrect: GPU utilization is not directly correlated with the number of concurrent streaming requests, and a 5-minute aggregation window makes the policy even slower to react than the existing invocation metric.

Key Takeaway: For streaming or long-lived inference workloads where concurrent in-flight requests spike in seconds, always use high-resolution concurrency metrics like ConcurrentRequestsPerModel with target tracking rather than per-minute invocation counts.

Question 2 - Cooldown Configuration to Prevent Scaling Flapping

A media streaming company hosts a recommendation model on Amazon SageMaker AI using Application Auto Scaling. The current scaling policy uses a target-tracking policy on SageMakerVariantInvocationsPerInstance with a target value of 50.

The engineering team observes that during normal traffic fluctuation, the endpoint frequently scales out and then scales back in within minutes, leading to unnecessary instance churn and increased operational cost. They want to maintain the same response time to genuine load changes but prevent rapid, oscillating scale-out/scale-in cycles caused by short-lived traffic bursts.

Which configuration change should the ML engineer implement? (Select ONE)

A) Remove the scale-in cooldown entirely to allow faster scale-in when traffic drops.
B) Increase the scale-in cooldown period to allow the newly added instances to serve requests for a minimum period before they can be removed.
C) Increase the target value from 50 to 500 invocations per instance to reduce the frequency of scaling actions.
D) Switch the metric to SageMakerVariantConcurrentRequestsPerModel with a target of 500 concurrent requests.
E) Disable target tracking and rely solely on manual scaling during business hours.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: Application Auto Scaling supports scale-in cooldowns that enforce a minimum interval between scale-out and scale-in actions. Increasing the scale-in cooldown ensures that once new instances are added due to a traffic burst, they remain active for a longer period (e.g., 10-15 minutes) before being removed—even if traffic briefly dips. This prevents scaling flapping and stops unnecessary instance churn from short-lived traffic fluctuations.

Distractor Analysis

A is Incorrect: Removing the scale-in cooldown makes the problem worse—instances would be terminated immediately after scale-out as soon as traffic dips even slightly, causing even more frequent oscillation.
C is Incorrect: Increasing the target value to 500 invocations per instance would severely overload each instance before scale-out occurs, causing unacceptable latency and potential request failures during genuine traffic growth, violating the requirement to maintain response time.
D is Incorrect: Switching to a high-resolution concurrency metric without adjusting cooldown settings would still result in frequent scale-in actions after bursts dissipate; the core issue is oscillation from short-lived traffic bursts, not the metric type itself.
E is Incorrect: Manual scaling requires constant operational oversight and cannot dynamically react to traffic changes, imposing significant operational overhead and risk of under-provisioning.

Key Takeaway: Scale-in cooldowns are the correct tool to prevent flapping from short-lived traffic bursts—they keep newly added capacity online for a minimal duration without affecting the sensitivity of scale-out decisions.

Question 3 - GPU Workload Scaling with Custom Metric

A pharmaceutical research firm is running a protein folding model for inference on Amazon SageMaker AI with GPU-backed instances (ml.g5.2xlarge). The model's inference time is highly dependent on GPU memory bandwidth and compute utilization rather than the raw number of API requests. Under normal load, the endpoint receives 20 requests per minute, but each request may complete in 1-3 minutes, and all requests consume significant GPU resources.

The company wants to scale the endpoint based on actual GPU resource saturation to ensure that new requests are not queued too long when GPUs are saturated. They have confirmed that GPU utilization increases proportionally to the number of concurrent inferences the model can handle.

Which scaling configuration should the ML engineer implement? (Select ONE)

A) Use a target-tracking policy on SageMakerVariantInvocationsPerInstance with a target value of 20.
B) Use a target-tracking policy on SageMakerVariantConcurrentRequestsPerModel with a target of 10 concurrent requests.
C) Publish GPU utilization as a custom metric to CloudWatch at 1-minute intervals, then configure a target-tracking policy on that custom metric.
D) Use a scheduled scaling action that provisions maximum GPU instances at 8:00 AM daily.
E) Configure a step scaling policy on CPUUtilization across the endpoint's GPU instances.

Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

C is Correct: When the scaling signal is a resource like GPU utilization that is not exposed as a built-in SageMaker metric, the correct approach is to publish it as a custom CloudWatch metric from the model container (e.g., using the CloudWatch agent or the put_metric_data API), then configure a target-tracking policy on that custom metric. The company has verified that GPU utilization moves proportionally to capacity, satisfying the requirement to scale based on actual GPU saturation.

Distractor Analysis

A is Incorrect: InvocationsPerInstance counts API calls, not GPU resource consumption. Since each request may take 1-3 minutes, a low request count hides the real GPU saturation issue, making this metric unsuitable for scaling GPU workloads based on resource utilization.
B is Incorrect: While high-resolution concurrency metrics detect in-flight requests, they do not directly measure GPU memory bandwidth or compute utilization. A few long-running requests could saturate the GPU without showing as high concurrency, causing the policy to under-scale.
D is Incorrect: Scheduled scaling is only suitable for predictable, calendar-based traffic patterns. Protein folding workloads are likely to be spiky and unpredictable, requiring dynamic scaling based on real-time resource utilization rather than fixed time windows.
E is Incorrect: CPUUtilization is a poor indicator of GPU-heavy workloads. GPU instances may have low CPU utilization even when the GPU is saturated by model inference, causing the scaling policy to miss critical scaling events.

Key Takeaway: For GPU-accelerated inference where the bottleneck is GPU resource saturation (not request count or CPU), publish a custom CloudWatch metric from the container and apply target tracking—but always validate that the custom metric correlates with scaling needs before production use.

Question 4 - Scaling for Provisioned Concurrency on Serverless Endpoints

A startup has deployed a chatbot model on Amazon SageMaker AI Serverless Inference with provisioned concurrency to eliminate cold-start latency. The current configuration has a fixed provisioned concurrency of 10, but the team observes that during business hours (9 AM - 6 PM), concurrent user requests peak at 50, while outside business hours they drop to nearly zero.

The startup wants to minimize operational cost while maintaining fast response times. They need to automatically adjust provisioned concurrency based on actual concurrent request patterns, without maintaining a fleet of always-on instances.

Which scaling configuration should the ML engineer implement? (Select ONE)

A) Migrate the model to real-time inference and use a target-tracking policy on SageMakerVariantInvocationsPerInstance.
B) Use Application Auto Scaling on the serverless endpoint's provisioned concurrency with a target-tracking policy on ConcurrentRequestsPerModel.
C) Set provisioned concurrency to 50 permanently to handle peak demand, accepting higher cost.
D) Use a scheduled scaling policy that sets provisioned concurrency to 50 at 9 AM and 0 at 6 PM on weekdays.
E) Disable provisioned concurrency entirely and rely on on-demand serverless scaling with cold starts.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: Amazon SageMaker AI supports Application Auto Scaling for serverless endpoints with provisioned concurrency. You can register the endpoint as a scalable target and apply a target-tracking policy on high-resolution concurrency metrics (e.g., ConcurrentRequestsPerModel). The policy will automatically adjust provisioned concurrency up during peak hours and down during off-peak hours based on actual concurrent requests, optimizing both latency and cost without scheduled actions or manual intervention.

Distractor Analysis

A is Incorrect: Migrating to real-time inference introduces unnecessary operational complexity and requires managing a fleet of always-on instances, whereas the serverless endpoint with auto-scaled provisioned concurrency already satisfies the requirements with lower overhead.
C is Incorrect: Setting provisioned concurrency to 50 permanently handles peak demand, but it incurs continuous cost during off-peak hours when concurrency is near zero, violating the cost-minimization constraint.
D is Incorrect: Scheduled scaling to specific values at fixed times is only approximate and cannot handle unpredictable or weekend traffic shifts. For example, if a late-night promo campaign causes unexpected traffic, the schedule would not adapt. Dynamic target tracking is the more flexible and operationally efficient approach.
E is Incorrect: Disabling provisioned concurrency entirely causes cold starts during traffic spikes and when the endpoint scales up from zero capacity, degrading response time and violating the requirement for fast response times.

Key Takeaway: Serverless endpoints support Application Auto Scaling on provisioned concurrency with target tracking on concurrency metrics—use this for dynamic, cost-optimized scaling without managing instances.

Question 5 - Multi-Model Endpoint Scaling with Inference Components

A retail analytics company has deployed multiple machine learning models on a single Amazon SageMaker AI endpoint using inference components. Each inference component serves a different model (e.g., product recommendation, price forecasting, inventory prediction) with different traffic patterns and resource requirements. The company wants to scale each model independently based on its own traffic.

Which configuration should the ML engineer implement to meet this requirement? (Select ONE)

A) Scale the endpoint instance count using a single target-tracking policy on the aggregate SageMakerVariantInvocationsPerInstance metric.
B) Register each inference component as a separate scalable target in Application Auto Scaling, and configure individual scaling policies per component.
C) Deploy each model to a separate real-time endpoint and scale each endpoint independently.
D) Configure a single step scaling policy for the entire endpoint based on combined CPU utilization.
E) Use a fixed instance count and rely on the multi-model server's internal routing to balance load.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: Amazon SageMaker AI with inference components allows each component (model) on a multi-model endpoint to be registered as an independent scalable target in Application Auto Scaling. Each component can have its own scaling policy based on component-specific metrics such as invocations per copy or concurrent requests per copy. This provides fine-grained, independent scaling of each model based on its specific traffic pattern without needing separate endpoints.

Distractor Analysis

A is Incorrect: Scaling the entire endpoint instance count based on aggregate invocations across all models is too coarse—if one model receives a traffic spike while others are idle, the entire endpoint scales out, incurring unnecessary cost for all models. It also cannot scale individual model copies independently.
C is Incorrect: Deploying each model to a separate real-time endpoint requires managing multiple endpoint configurations, security groups, and scaling policies separately, increasing operational overhead significantly compared to using inference components on a shared endpoint.
D is Incorrect: CPU utilization is a coarse aggregate across all models and does not distinguish which model is under load. A single step scaling policy for the entire endpoint cannot independently scale individual models.
E is Incorrect: A fixed instance count does not adapt to traffic changes and relies on internal routing, which may cause thread pool exhaustion or queue delays for high-traffic models while low-traffic models consume idle resources. This violates the requirement for independent scaling.

Key Takeaway: For multi-model endpoints with inference components, register each component as an independent scalable target with its own policy using component-specific metrics (e.g., invocations per copy) to scale models independently.

Question 6 - Scaling Metrics for Batch Transform Jobs

A data engineering team at an insurance company processes nightly claims data through Amazon SageMaker AI Batch Transform. The batch processing job reads from an S3 bucket, processes millions of records through a risk scoring model, and writes predictions back to S3. The team notices that a single batch transform job with 2 instances takes 8 hours to complete, which exceeds the overnight processing window.

The team wants to reduce processing time by automatically sizing the batch transform cluster based on the data volume. They need a solution that scales compute resources proportionally to the number of input records.

Which approach should the ML engineer implement? (Select ONE)

A) Use Application Auto Scaling with a target-tracking policy on SageMakerVariantInvocationsPerInstance to scale the batch transform cluster.
B) Use the MaxPayloadInMB parameter to allow each instance to process larger payloads, reducing processing time.
C) Increase the InstanceCount parameter and use MaxConcurrentTransforms to tune parallelism, optimizing for the data volume and instance capacity.
D) Register the batch transform job as a scalable target with a step scaling policy on CPU utilization.
E) Use AWS Glue to process the data instead of SageMaker Batch Transform.

Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

C is Correct: SageMaker Batch Transform jobs do not support Application Auto Scaling. Batch jobs are sized at job submission time using the InstanceCount parameter, and parallelism is controlled by MaxConcurrentTransforms (number of concurrent HTTP requests per instance). The correct approach is to increase InstanceCount to add more compute and tune MaxConcurrentTransforms to optimize throughput per instance. Additionally, using BatchStrategy (e.g., MultiRecord) can improve throughput for certain data formats.

Distractor Analysis

A is Incorrect: Application Auto Scaling does not support batch transform jobs—it only works with real-time endpoints, serverless endpoints with provisioned concurrency, and inference components. Batch transform is a finite, offline job that runs to completion rather than serving continuous traffic.
B is Incorrect: MaxPayloadInMB limits the size of individual payloads sent to a model, but increasing it does not necessarily improve overall throughput and may even cause issues with large payloads. It does not address the core issue of insufficient compute capacity across the cluster.
D is Incorrect: Batch Transform jobs are not registered as scalable targets in Application Auto Scaling, and CPU utilization-based step scaling policies are not applicable to finite batch jobs.
E is Incorrect: While AWS Glue can process large datasets, migrating the ML inference workload from SageMaker Batch Transform to AWS Glue would require rewriting the inference pipeline and may not support the same ML model inference capabilities without additional integration (e.g., invoking SageMaker endpoints from Glue jobs), introducing unnecessary complexity.

Key Takeaway: Batch Transform is a finite offline job that must be sized at submission time—use InstanceCount and MaxConcurrentTransforms to optimize throughput, not auto scaling policies.

Question 7 - Scaling Based on Model Queue Depth with Custom Metric

A logistics company has deployed a demand forecasting model on a SageMaker AI real-time endpoint. The endpoint receives requests from various internal applications, and each request can take 5-20 seconds to complete depending on the model's complexity for the given region and time window.

The operations team monitors the endpoint and finds that when the per-instance request queue depth exceeds 50 (i.e., more than 50 requests are waiting to be processed), response latency degrades significantly. The company wants to scale out when queue depth grows, not when the per-minute invocation count reaches a threshold (because that still causes backlog).

Which scaling approach should the ML engineer implement? (Select ONE)

A) Use a target-tracking policy on SageMakerVariantConcurrentRequestsPerModel with a target value of 50.
B) Use a target-tracking policy on SageMakerVariantInvocationsPerInstance with a target value of 50.
C) Publish the per-instance queue depth as a custom CloudWatch metric from the model container, then configure a target-tracking policy on that metric.
D) Increase the instance count to a fixed value of 10 instances to handle the queue depth.
E) Use a step scaling policy on ModelLatency with a threshold of 20 seconds.

Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

C is Correct: Queue depth is not a built-in SageMaker metric. To scale based on it, the model container must publish the per-instance queue depth as a custom CloudWatch metric (e.g., using the CloudWatch agent or the put_metric_data API call from the inference container). A target-tracking policy can then be configured on the custom metric. The company has identified queue depth as the direct indicator of backlog, so this targeted approach provides precise scaling.

Distractor Analysis

A is Incorrect: SageMakerVariantConcurrentRequestsPerModel measures in-flight concurrent requests (currently being processed), not the number of requests waiting in a queue. Requests in the queue waiting for processing are not "in-flight" and are not captured by this high-resolution metric.
B is Incorrect: SageMakerVariantInvocationsPerInstance measures the rate of incoming requests per minute, which is a leading indicator of load but does not directly measure backlog or waiting requests. It can also react too slowly (per-minute aggregation) to capture immediate queue saturation.
D is Incorrect: A fixed instance count of 10 may be over-provisioned during low traffic and under-provisioned during spikes, incurring unnecessary cost without dynamic adaptation. This violates the requirement to scale based on queue depth, not a fixed capacity.
E is Incorrect: ModelLatency measures the time taken to process a request, not the number of waiting requests. While high latency may correlate with queue depth, it is an indirect signal that can be skewed by other factors (e.g., model complexity), and the 20-second threshold is scenario-specific rather than directly tied to the queue depth issue.

Key Takeaway: For scaling signals that SaaS metrics don't expose (like queue depth), publish a custom CloudWatch metric from the container and apply target tracking—but only after validating that the custom metric moves proportionally to capacity in staging.

Question 8 - Choosing the Correct Scaling Metric for a Chatbot with Long-Lived Connections

A healthcare startup has deployed an AI-powered patient triage chatbot on SageMaker AI real-time inference. Each user session maintains a persistent connection to the endpoint for 5-10 minutes while the chatbot asks follow-up questions and streams responses. During this time, the model instance handles the session state and processes multiple turns of conversation.

The startup's current scaling policy uses SageMakerVariantInvocationsPerInstance with a target of 100 invocations per minute, but they observe that during peak hours, patients experience long wait times before receiving responses. They discover that each instance can only handle 20 simultaneous active sessions before performance degrades, regardless of the total number of API calls per minute.

Which metric should the ML engineer use for scaling to ensure each instance doesn't exceed 20 simultaneous active sessions? (Select ONE)

A) SageMakerVariantInvocationsPerInstance with a target value of 20.
B) SageMakerVariantConcurrentRequestsPerModel (or PerCopy) with a target value of 20.
C) CPUUtilization with a target value of 50%.
D) ModelLatency with a target value of 1 second.
E) SageMakerModelCacheHitRate with a target value of 90%.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: The high-resolution concurrency metrics SageMakerVariantConcurrentRequestsPerModel and SageMakerVariantConcurrentRequestsPerCopy measure the number of simultaneous in-flight requests currently being processed by the endpoint (or per instance/model copy). Because each patient session maintains a persistent connection and multiple requests during a session, the number of concurrent in-flight requests directly reflects the number of active sessions. Setting a target value of 20 ensures Application Auto Scaling adds instances when concurrent requests approach the per-instance handling capacity, preventing response delays.

Distractor Analysis

A is Incorrect: SageMakerVariantInvocationsPerInstance measures the rate of requests per minute, not the number of simultaneous active sessions. Multiple API calls during a single session could yield a high invocation count even when few sessions are active, or a low count when many sessions are in a holding pattern between turns. The target value of 20 would trigger scaling too frequently based on request rate, not on concurrent session load.
C is Incorrect: CPUUtilization does not directly map to concurrent session capacity. The model may consume relatively little CPU while holding conversational state (e.g., waiting for user input), causing under-scaling. Conversely, CPU spikes during brief computation may trigger over-scaling.
D is Incorrect: ModelLatency measures the time per inference request, not the number of concurrent sessions. While high latency is a symptom of overload, it is an indirect signal that reacts after performance has already degraded, and it is not tied to the specific per-instance session capacity limit.
E is Incorrect: SageMakerModelCacheHitRate is a metric for multi-model endpoints measuring model cache effectiveness, not a scaling signal for concurrent sessions or request load. It is unrelated to the patient session bottleneck described in the scenario.

Key Takeaway: For long-lived, stateful connections (e.g., chatbots, streaming sessions), the constraint is concurrent in-flight requests, not per-minute invocations—use high-resolution concurrency metrics for scaling.

Question 9 - Multi-Metric Scaling Policy for Real-Time Inference

An e-commerce platform runs a real-time product recommendation model on SageMaker AI endpoint. The model's performance depends on both request throughput and processing latency. The current scaling policy uses a target-tracking policy on SageMakerVariantInvocationsPerInstance with target 100, but the team notices that during flash sales, the endpoint becomes CPU-bound even when invocation rates remain moderate, causing latency to spike.

The team wants to scale out based on whichever signal indicates saturation first—request throughput or CPU utilization—to provide maximum protection during traffic spikes. Which configuration should the ML engineer implement? (Select ONE)

A) Use two separate target-tracking policies on the same scalable target: one on SageMakerVariantInvocationsPerInstance and one on CPUUtilization, each with their own target values.
B) Use a single target-tracking policy on SageMakerVariantConcurrentRequestsPerModel only.
C) Configure a step scaling policy with multiple steps based on SageMakerVariantInvocationsPerInstance thresholds.
D) Publish a composite custom metric that combines invocation rate and CPU utilization, then use target tracking on the composite.
E) Use scheduled scaling during flash sales and manual scaling otherwise.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

A is Correct: Application Auto Scaling supports multiple target-tracking policies on the same scalable target. Each policy independently evaluates its metric and triggers scaling actions when the metric crosses its target. By configuring one target-tracking policy on SageMakerVariantInvocationsPerInstance (to handle throughput saturation) and another on CPUUtilization (to handle compute saturation), Application Auto Scaling will scale out when either metric indicates saturation—whichever happens first. The most aggressive scaling action is applied when multiple policies are triggered.

Distractor Analysis

B is Incorrect: A single policy on concurrency only tracks in-flight requests and would still miss CPU-bound scenarios where a few heavy requests saturate the processor without high concurrency (e.g., complex inference for a single user during a flash sale).
C is Incorrect: Step scaling policies are based on CloudWatch alarm thresholds and are less responsive than target tracking for gradual environmental changes. They also require manual tuning of threshold levels and only respond to a single metric (invocations), missing CPU-bound saturation.
D is Incorrect: Publishing a composite metric requires custom code in the container to combine signals, adds operational complexity, and may hide the precise cause of saturation. Two separate target-tracking policies achieve the same effect natively without custom metric logic.
E is Incorrect: Scheduled scaling is deterministic and cannot adapt to unexpected flash sale traffic patterns or irregular demand. Manual scaling during off-peak also requires constant operational oversight and risks under-provisioning.

Key Takeaway: Application Auto Scaling supports multiple target-tracking policies on the same target—use this for scenarios where different resources (e.g., request rate and CPU) can independently bottleneck performance.

Question 10 - Scaling Based on SageMakerVariantInvocations vs Custom Predefined Metrics

A data science team at a logistics company has deployed a route optimization model on SageMaker AI real-time inference. The model is a CPU-heavy workload where inference time varies between 0.5 and 5 seconds depending on the complexity of the route (e.g., number of stops, traffic conditions, depot constraints). The current scaling policy uses a target-tracking metric on SageMakerVariantInvocationsPerInstance with a target of 50.

The team observes that during periods of complex route requests (5-second inference), the per-minute invocation count remains low (e.g., 12 requests per minute) because each request takes longer to process. However, the endpoint's CPU utilization reaches 90%, and additional requests queue up. With 50 invocations per instance as the target, the current policy never triggers scale-out during these complex-request periods. The team needs a scaling policy that reacts to CPU saturation rather than request count.

Which configuration change should the ML engineer implement? (Select ONE)

A) Lower the target value of SageMakerVariantInvocationsPerInstance to 10 to trigger scale-out during complex requests.
B) Switch to a target-tracking policy on the CPUUtilization metric with a target value of 60%.
C) Switch to a target-tracking policy on SageMakerVariantConcurrentRequestsPerModel with target 30.
D) Increase the instance type from CPU-optimized to memory-optimized to handle longer inference times.
E) Use a step scaling policy on MemoryUtilization with multiple thresholds.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: CPUUtilization is a valid target-tracking metric for SageMaker endpoints in Application Auto Scaling. The team has observed that CPU utilization reaches 90% during complex inference requests even when invocation rates are low. By setting a target-tracking policy on CPUUtilization with a target value of 60%, the policy will scale out when CPU usage exceeds 60%, directly addressing the resource saturation caused by compute-heavy requests. This is the correct metric for workloads where inference time varies significantly and CPU is the scaling bottleneck.

Distractor Analysis

A is Incorrect: Lowering the invocation target to 10 would cause premature scale-out during normal traffic (where invocation rate is high but CPU is low), incurring unnecessary cost, and it would still not directly measure CPU saturation. The root cause is resource utilization, not request rate.
C is Incorrect: SageMakerVariantConcurrentRequestsPerModel measures in-flight requests. While this would capture some of the load, it does not reflect the varying CPU intensity of each request. A few complex 5-second requests might saturate the CPU while concurrency remains low, causing under-scaling.
D is Incorrect: Changing the instance type from CPU-optimized to memory-optimized does not address the CPU-bound nature of the workload. The route optimization model is CPU-heavy, so memory-optimized instances would not improve inference throughput and would increase cost without performance benefit.
E is Incorrect: The scenario states CPU is the bottleneck, not memory. Targeting MemoryUtilization would not trigger scale-out when CPU is saturated, causing continued queueing and latency issues.

Key Takeaway: When inference time varies significantly and requests are compute-heavy, per-invocation metrics may remain low even under saturation—use resource-based metrics like CPUUtilization (or custom GPU metrics) to capture the true load.

Question 11 - Target Tracking vs Step Scaling Policy Selection

A banking application uses a real-time transaction classification model on SageMaker AI. The traffic pattern is highly variable: for most of the day, the endpoint receives 10-20 requests per minute, but during end-of-month billing cycles, traffic surges to 500 requests per minute for several hours, then drops back to baseline. The team wants a scaling solution that adapts smoothly to both gradual traffic growth and sudden spikes, without requiring manual intervention or frequent threshold adjustments.

Which scaling policy type should the ML engineer implement? (Select ONE)

A) A step scaling policy with three steps based on CloudWatch alarms on SageMakerVariantInvocationsPerInstance.
B) A target-tracking policy on SageMakerVariantInvocationsPerInstance with a target value based on desired per-instance throughput.
C) A simple scaling policy with a single alarm threshold at 500 invocations.
D) Scheduled scaling actions for the end-of-month period combined with manual scaling otherwise.
E) A fixed instance count of 10 to handle peak load at all times.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: Target-tracking policies are designed to maintain a specified target value for a metric (e.g., 50 invocations per instance per minute) by automatically adjusting capacity proportionally. When traffic increases gradually, the policy scales out smoothly; when traffic surges suddenly, the policy reacts by adding sufficient capacity to bring the metric back to the target. Target tracking requires minimal configuration, adapts to changing traffic patterns without manual intervention, and avoids the constant threshold tuning required by step policies. This is the recommended approach for dynamic workloads.

Distractor Analysis

A is Incorrect: Step scaling policies require configuring multiple CloudWatch alarms at specific thresholds and manual tuning of the step adjustments. While they can handle sudden spikes, they do not adapt as smoothly to gradual traffic growth and require ongoing maintenance as traffic patterns evolve.
C is Incorrect: A simple scaling policy with a single alarm at 500 invocations is too coarse—it would only trigger at extreme traffic levels and might under-provision during moderate growth (e.g., 200-400 invocations), or over-provision at the peak, wasting cost.
D is Incorrect: Scheduled scaling works for the predictable end-of-month spike, but manual scaling for baseline traffic is operationally intensive and cannot adapt to unexpected surges. This violates the requirement for minimal manual intervention.
E is Incorrect: A fixed instance count of 10 is over-provisioned for most of the month when traffic is low (10-20 requests/minute), incurring significant unnecessary cost, and may still under-provision during extreme spikes if 10 instances cannot handle the load.

Key Takeaway: Target-tracking policies are the default recommendation for dynamic workloads because they automatically adjust capacity proportionally to maintain a target metric value, requiring minimal ongoing tuning.

Question 12 - Scaling Metric Selection for Multi-Model Endpoint with Heterogeneous Traffic

A technology company hosts a multi-model endpoint on SageMaker AI using inference components, serving both a text summarization model and an image classification model. The text model receives high-frequency, short-lived requests (100 requests/minute, 200ms each), while the image model receives low-frequency, long-lived requests (5 requests/minute, 3 seconds each). The current single scaling policy on the endpoint uses SageMakerVariantInvocationsPerInstance with a target of 100.

The company wants to scale each model independently based on its specific traffic characteristics to optimize cost and performance. Which approach should the ML engineer take? (Select ONE)

A) Use a single target-tracking policy on the endpoint with SageMakerVariantConcurrentRequestsPerModel target 100.
B) Register each inference component as an independent scalable target in Application Auto Scaling, with separate scaling policies per component using appropriate metrics (invocation count for text model, concurrency for image model).
C) Deploy the text and image models to separate endpoints with separate scaling policies.
D) Use a step scaling policy on aggregate CPU utilization for the entire multi-model endpoint.
E) Manually adjust instance counts for each model based on weekly traffic reports.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: With inference components, each model copy can be registered as an independent scalable target. Application Auto Scaling allows different scaling policies per component. The text model (high-frequency, short requests) benefits from a target-tracking policy on SageMakerComponentInvocationsPerCopy or similar invocation-based metrics, while the image model (low-frequency, long-lived requests) benefits from SageMakerComponentConcurrentRequestsPerCopy. This approach scales each model according to its unique traffic pattern, optimizing for both cost and performance.

Distractor Analysis

A is Incorrect: A single concurrency target for the entire endpoint conflates the two models' traffic patterns. The text model's high concurrency would trigger scale-out even when the image model is under-loaded, and vice versa. The single metric cannot independently optimize each model's capacity.
C is Incorrect: Deploying each model to a separate endpoint achieves independent scaling, but requires managing twice the endpoint configurations, twice the VPC/security group settings, and twice the operational overhead. Inference components provide the same scaling granularity with a single endpoint.
D is Incorrect: Aggregate CPU utilization across all models cannot isolate which model is causing CPU saturation. A CPU spike from image processing could trigger unnecessary scale-out for the text model, and aggregate CPU does not reflect per-model queueing or request patterns.
E is Incorrect: Manual scaling based on weekly traffic reports cannot react to real-time traffic fluctuations and requires constant operational oversight, resulting in either under-provisioning (causing latency during spikes) or over-provisioning (incurring unnecessary cost).

Key Takeaway: For multi-model endpoints with heterogeneous traffic, use inference components and register each as an independent scalable target with its own scaling policy tailored to its specific workload characteristics.

Summary of Key Metrics for Auto Scaling on SageMaker AI

Metric	Type	Best Use Case
`SageMakerVariantInvocationsPerInstance`	Predefined, per-minute	General-purpose, request-count-driven scaling for real-time endpoints
`SageMakerVariantConcurrentRequestsPerModel` / `PerCopy`	High-resolution (10s), concurrency	Streaming, long-lived, or session-based inference with rapid concurrency spikes
`CPUUtilization`	Resource, 1-minute	Compute-bound workloads where CPU saturation drives performance
Custom CloudWatch metric (e.g., queue depth, GPU utilization)	Custom, user-defined	Workloads where no predefined metric captures the scaling signal
`SageMakerComponentInvocationsPerCopy`	Component-level, per-minute	Multi-model endpoints with per-component invocation scaling
`SageMakerComponentConcurrentRequestsPerCopy`	Component-level, high-resolution	Multi-model endpoints with per-component concurrency scaling

Exam Tip: For the MLA-C02 exam, memorize that high-resolution concurrency metrics (like ConcurrentRequestsPerModel) publish every 10 seconds and are essential for rapid-reaction scaling on streaming endpoints, while invocation and CPU metrics publish at 1-minute intervals and are better for steady-state or compute-bound workloads.