Skill 3.2.10: Implement AI-specific resource scaling for GPU workloads

Question	Answer & Explanation
### Question 1 - High-Resolution Concurrency Metrics for Streaming LLMs A machine learning team is hosting a Large Language Model (LLM) on an Amazon SageMaker real-time endpoint using `ml.g5.2xlarge` GPU instances. The model generates long streaming responses, and traffic spikes can occur rapidly within seconds. The current auto scaling policy uses a target tracking policy on the standard `SageMakerVariantInvocationsPerInstance` metric, which has a 1-minute aggregation period. This causes latency spikes because new instances are provisioned too late during sudden bursts. The team needs the fastest reaction time with the least operational overhead. Which scaling configuration addresses this requirement? (Select ONE) - A) Configure an Asynchronous endpoint and scale on the `ApproximateBacklogSize` metric. - B) Configure a target tracking policy on a high-resolution concurrency metric, such as `ConcurrentRequestsPerModel`. - C) Increase the target value of the existing `SageMakerVariantInvocationsPerInstance` metric to trigger scaling earlier. - D) Decrease the scale-in cooldown period to allow faster recovery from scale-out events.	Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker supports high-resolution CloudWatch concurrency metrics (like `ConcurrentRequestsPerModel` or `ConcurrentRequestsPerCopy`) emitted at sub-minute intervals (e.g., 10 seconds). Configuring a target tracking policy on this high-resolution metric allows Application Auto Scaling to detect in-flight request surges almost instantaneously and trigger scale-out events in seconds, which is critical for long-running streaming LLM inferences. Distractor Analysis: - A is Incorrect: Asynchronous endpoints are for large payloads or long processing times where immediate response is not required, but they do not solve the need for real-time streaming responses with low latency. - C is Incorrect: Increasing the target threshold forces each instance to handle more traffic before triggering a scale-out, which delays scaling even further and risks overwhelming the existing GPU instances. - D is Incorrect: Modifying the scale-in cooldown affects how quickly instances are terminated after traffic drops, not how quickly the system scales out in response to a sudden traffic spike. > Key Takeaway: For workloads with rapid traffic surges and long-running inferences (like LLM streaming), use high-resolution concurrency metrics instead of standard 1-minute invocation metrics for faster auto scaling reactions.
### Question 2 - Scaling Model Copies with SageMaker Inference Components A company deploys a large text-to-image generation model on a single `ml.p4d.24xlarge` SageMaker real-time endpoint. The model does not fully saturate the instance's GPU resources. The engineering team wants to maximize GPU utilization by running multiple copies of the model on the same instance and scaling the number of copies automatically based on traffic, minimizing the need to provision entirely new instances. Constraint: Least operational overhead. Which configuration satisfies this requirement? (Select ONE) - A) Deploy a SageMaker Asynchronous endpoint and increase the instance count. - B) Use SageMaker Inference Components with Application Auto Scaling on the `ConcurrentRequestsPerCopy` metric. - C) Create multiple endpoint variants on the same instance and use weighted routing. - D) Deploy the model on a SageMaker Serverless endpoint and configure provisioned concurrency.	Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker Inference Components allow you to deploy one or more models (or copies of a model) onto a SageMaker real-time endpoint. You can use Application Auto Scaling to scale the number of copies of a specific inference component up and down on the underlying instance(s) based on concurrency, fully utilizing the GPU hardware before scaling out the endpoint instance count itself. Distractor Analysis: - A is Incorrect: Provisioning more instances does not maximize the utilization of the existing large `ml.p4d.24xlarge` instance and introduces unnecessary hardware costs. - C is Incorrect: Endpoint variants are used for A/B testing or traffic splitting between different model versions, not for scaling multiple copies of the same model on a single instance. - D is Incorrect: SageMaker Serverless Inference does not support GPU instances; it is limited to CPU configurations with a maximum of 6 GB of memory. > Key Takeaway: Use SageMaker Inference Components to pack multiple copies of a model onto a single large GPU instance and scale those copies independently based on concurrent requests.
### Question 3 - Custom GPU Utilization Metrics for Scaling An AI team is running a GPU-intensive computer vision inference workload on a SageMaker real-time endpoint. They want to scale the endpoint capacity based on the actual GPU memory utilization to prevent Out-Of-Memory (OOM) errors during peak traffic, as different image resolutions consume varying amounts of GPU VRAM. The team needs to use a native CloudWatch metric for their target tracking policy. Which predefined SageMaker metric should they use? (Select ONE) - A) `SageMakerVariantInvocationsPerInstance` - B) `CPUUtilization` - C) `GPUMemoryUtilization` - D) `ConcurrentRequestsPerModel`	Correct Answer: C Correct Answer Analysis: - C is Correct: SageMaker natively publishes hardware utilization metrics to CloudWatch, including `CPUUtilization`, `MemoryUtilization`, `GPUUtilization`, and `GPUMemoryUtilization`. By configuring a target tracking policy on `GPUMemoryUtilization`, the auto scaler will add more GPU instances when VRAM usage approaches the defined threshold, preventing OOM errors when processing large, memory-intensive payloads. Distractor Analysis: - A is Incorrect: `SageMakerVariantInvocationsPerInstance` tracks the number of invocations, not the hardware resource consumption, making it unsuitable for preventing OOM errors if payload sizes vary drastically. - B is Incorrect: `CPUUtilization` tracks CPU usage, which may not correlate directly with GPU VRAM consumption during inference. - D is Incorrect: `ConcurrentRequestsPerModel` tracks active requests, but a single large request could trigger an OOM before the concurrency threshold is reached. > Key Takeaway: SageMaker publishes `GPUMemoryUtilization` and `GPUUtilization` metrics to CloudWatch, allowing Application Auto Scaling to scale based on actual hardware resource constraints rather than just request counts.
### Question 4 - Configuring Scale-In Cooldowns for Heavy Model Artifacts A generative AI startup is configuring Application Auto Scaling for their SageMaker real-time endpoint backed by `ml.g5` instances. The model artifacts are over 20 GB, meaning newly launched instances take several minutes to download and load the model into GPU memory. The team wants to ensure that newly added instances are not terminated prematurely if traffic momentarily drops during a model warm-up phase. Constraint: Minimize operational overhead. Which scaling configuration should they apply? (Select ONE) - A) Set a longer scale-in cooldown period. - B) Set a longer scale-out cooldown period. - C) Decrease the target tracking metric threshold. - D) Enable detailed monitoring on the underlying EC2 instances.	Correct Answer: A Correct Answer Analysis: - A is Correct: The scale-in cooldown period dictates how long Application Auto Scaling waits before terminating an instance after a scaling policy triggers. Setting a longer scale-in cooldown (e.g., 10-15 minutes) ensures that newly provisioned GPU instances have adequate time to download artifacts and warm up before being terminated by a momentary dip in traffic. Distractor Analysis: - B is Incorrect: The scale-out cooldown controls how long the system waits before adding more instances after a scale-out event. It does not prevent the termination of newly added instances. - C is Incorrect: Decreasing the target tracking threshold makes the endpoint scale out faster, but does not protect new instances from premature scale-in. - D is Incorrect: Detailed monitoring changes the CloudWatch metric resolution to 1-minute for EC2, but SageMaker endpoints already emit metrics at high resolution, and this does not affect the auto scaling cooldown logic. > Key Takeaway: For ML endpoints with heavy model artifacts and long initialization times, configure a longer scale-in cooldown to prevent Application Auto Scaling from prematurely terminating instances during brief traffic dips.
### Question 5 - Decoupling Queue Scaling with Asynchronous Endpoints A team is deploying an audio transcription model that takes long audio files as input. The endpoint uses GPU instances, and payloads can be up to 500 MB. The team needs to decouple request queuing from processing so users receive an immediate acknowledgment, and they need to scale the GPU instances based on the number of pending requests in the queue. Which scaling configuration satisfies this? (Select ONE) - A) Real-time endpoint with target tracking on `ConcurrentRequestsPerModel`. - B) Serverless endpoint with provisioned concurrency. - C) Asynchronous endpoint with target tracking on `ApproximateBacklogSize`. - D) Batch Transform job scheduled every hour.	Correct Answer: C Correct Answer Analysis: - C is Correct: SageMaker Asynchronous endpoints are designed for large payloads (up to 1 GB) and long processing times. They queue incoming requests and return an inference ID immediately. You can configure Application Auto Scaling to scale based on the `ApproximateBacklogSize` metric, which tracks the number of pending requests in the queue, provisioning GPU capacity precisely when the backlog grows. Distractor Analysis: - A is Incorrect: Real-time endpoints have a 60-second timeout limit and do not queue large payloads natively in the same way. 500 MB payloads would fail on a real-time endpoint without custom chunking. - B is Incorrect: Serverless endpoints do not support GPU instances and have a payload limit of 100 MB, making them unsuitable for 500 MB audio files. - D is Incorrect: Batch Transform is for offline, bulk processing of large datasets, not for handling individual, user-initiated on-demand inference requests. > Key Takeaway: Use SageMaker Asynchronous endpoints for large payload GPU workloads and scale based on `ApproximateBacklogSize` to manage queue depth dynamically.
### Question 6 - Resilient Scaling for Distributed Training with HyperPod A team is using Amazon SageMaker HyperPod for distributed training of a large language model across hundreds of GPU instances. The training process takes multiple weeks, and the cluster must survive the failure of individual GPU instances without interrupting the training job. Constraint: Ensure high availability and automatic recovery with the least operational overhead. Which configuration meets these requirements? (Select ONE) - A) Deploy the cluster in a single Availability Zone with large EBS volumes. - B) Configure SageMaker HyperPod with Slurm and enable automatic node replacement. - C) Use AWS Batch to monitor the training job and automatically restart it upon failure. - D) Configure a custom CloudWatch alarm to trigger an AWS Lambda function to re-provision failed nodes.	Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon SageMaker HyperPod is specifically designed to provide resilient accelerator clusters for distributed training. When orchestrated with a workload manager like Slurm, HyperPod automatically detects faulty hardware and replaces the faulty nodes, allowing the long-running training job to survive node failures without manual intervention or job restarts. Distractor Analysis: - A is Incorrect: Deploying in a single Availability Zone reduces resilience. Standard EBS volumes do not automatically replace faulty GPU hardware or reintegrate nodes into the training cluster. - C is Incorrect: AWS Batch can manage batch jobs but restarting a multi-week training job from scratch is extremely costly and does not provide node-level fault tolerance for distributed training. - D is Incorrect: Writing custom CloudWatch alarms and Lambda functions to handle node replacement introduces massive operational overhead and does not natively reintegrate nodes into the distributed training framework as seamlessly as HyperPod. > Key Takeaway: Amazon SageMaker HyperPod provides automated node replacement and fault tolerance for long-running distributed GPU training clusters, managed via Slurm or Kubernetes.
### Question 7 - Scaling to Zero with VPC Isolation for GPU Endpoints An enterprise has a real-time endpoint serving an internal generative AI model on `ml.g5.12xlarge` instances. They need to scale the endpoint to zero during off-hours to minimize costs, while strictly maintaining VPC network isolation for compliance. The team wants to use native SageMaker and Application Auto Scaling features. Which endpoint type and configuration meet these requirements? (Select ONE) - A) Serverless endpoint with VPC configuration and min capacity of 0. - B) Real-time endpoint with Application Auto Scaling min capacity set to 0, deployed in private subnets. - C) Asynchronous endpoint with a custom AWS Lambda function to terminate the instance. - D) Real-time endpoint with a scheduled scaling action to set capacity to 0, deployed in public subnets.	Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker real-time endpoints support VPC network isolation (private subnets and security groups) and can be registered with Application Auto Scaling to set the minimum capacity to 0. This allows the endpoint to scale down to zero instances when not in use, saving costs while keeping all traffic within the AWS private network. Distractor Analysis: - A is Incorrect: SageMaker Serverless Inference does not support VPC network configuration (PrivateLink/subnets), violating the compliance requirement, and it also does not support GPU instances. - C is Incorrect: Using a custom Lambda function to terminate instances introduces unnecessary operational overhead and does not leverage the native scaling-to-zero capability of Application Auto Scaling. - D is Incorrect: While scheduled scaling can set capacity to 0, deploying in public subnets violates the strict VPC network isolation compliance requirement. > Key Takeaway: For GPU workloads requiring VPC isolation and cost savings, use real-time endpoints in private subnets with Application Auto Scaling configured with a minimum capacity of 0.
### Question 8 - General-Purpose Predefined Metric for Scaling A machine learning engineer is configuring an Application Auto Scaling target tracking policy for a SageMaker endpoint variant running on `ml.g4dn` GPU instances. The team wants a general-purpose scaling metric that scales out when the number of invocations per instance exceeds the target value. The workload consists of short, synchronous API calls with predictable processing times. Which metric is the default general-purpose choice for this? (Select ONE) - A) `SageMakerVariantInvocationsPerInstance` - B) `Invocations` - C) `ModelLatency` - D) `OverheadLatency`	Correct Answer: A Correct Answer Analysis: - A is Correct: `SageMakerVariantInvocationsPerInstance` is the standard, predefined general-purpose metric for SageMaker endpoint scaling. It tracks the average number of invocations handled by each instance. A target tracking policy on this metric evenly distributes traffic across instances and scales out when the invocations per instance exceed the defined target. Distractor Analysis: - B is Incorrect: The `Invocations` metric tracks the total number of invocations across the entire endpoint, not normalized per instance. Using this for target tracking would not account for the current instance count. - C is Incorrect: `ModelLatency` measures the time it takes for the model to respond, which can fluctuate based on payload size. While it can be used for custom scaling, it is not the default general-purpose metric. - D is Incorrect: `OverheadLatency` measures the time taken by SageMaker infrastructure (routing, queueing) rather than the actual model load, making it a poor indicator for scaling compute capacity. > Key Takeaway: For general-purpose, predictable workloads, use the predefined `SageMakerVariantInvocationsPerInstance` metric in target tracking policies to scale based on average request load per instance.
### Question 9 - High-Resolution Metric Configuration Requirement A team is using a SageMaker real-time endpoint with `ml.g5.12xlarge` instances to serve an LLM. They want to scale based on the `ConcurrentRequestsPerModel` metric. The standard 1-minute CloudWatch aggregation is too slow for their highly variable traffic. They want to ensure the auto scaling policy evaluates the metric at a 10-second resolution to trigger scale-outs faster. Which action must they take in Application Auto Scaling to meet this requirement? (Select ONE) - A) Enable detailed monitoring on the underlying EC2 instances. - B) Choose the high-resolution metric in the Application Auto Scaling target tracking policy. - C) Increase the CloudWatch metric retention period to 15 months. - D) Create a custom CloudWatch dashboard to monitor the 10-second resolution.	Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker emits concurrency metrics (like `ConcurrentRequestsPerModel` and `ConcurrentRequestsPerCopy`) natively at high resolution (10 seconds). When creating the target tracking policy in Application Auto Scaling, the engineer must explicitly specify the metric and its high-resolution aggregation period (e.g., 10 seconds) so the auto scaler polls CloudWatch at this faster cadence. Distractor Analysis: - A is Incorrect: Enabling detailed monitoring on EC2 instances sets CloudWatch metrics to 1-minute resolution for standard EC2 metrics, but it does not affect SageMaker's custom high-resolution ML metrics, which are already emitted at 10 seconds. - C is Incorrect: The CloudWatch metric retention period dictates how long metrics are stored, not how frequently the Application Auto Scaling service evaluates the metric for scaling actions. - D is Incorrect: A CloudWatch dashboard is a visualization tool and has zero impact on the configuration or execution of Application Auto Scaling policies. > Key Takeaway: Application Auto Scaling can use high-resolution (10-second) CloudWatch metrics for target tracking, allowing near-instantaneous scale-out reactions for generative AI workloads.
### Question 10 - Scaling Multi-Model Endpoints on GPU An AI team deploys multiple versions of a generative text model on a SageMaker multi-model endpoint (MME) backed by GPU instances. Different models are invoked dynamically based on the request payload. The team wants to scale the endpoint capacity based on the overall GPU compute utilization across all models, rather than tracking invocations per instance, as some models are significantly more compute-intensive than others. Which scaling metric should they use for the target tracking policy? (Select ONE) - A) `ConcurrentRequestsPerModel` - B) `ApproximateBacklogSize` - C) `CPUUtilization` - D) `GPUUtilization`	Correct Answer: D Correct Answer Analysis: - D is Correct: `GPUUtilization` is a SageMaker CloudWatch metric that measures the actual compute load on the GPU hardware. For a Multi-Model Endpoint (MME) where different models consume different amounts of GPU compute, scaling based on `GPUUtilization` ensures new instances are provisioned when the hardware is genuinely saturated, regardless of which model is handling the request. Distractor Analysis: - A is Incorrect: `ConcurrentRequestsPerModel` scales based on concurrent requests for a specific model. In an MME, this doesn't account for the aggregate compute load across all loaded models on the GPU. - B is Incorrect: `ApproximateBacklogSize` is used for Asynchronous endpoints to measure queue depth, not real-time endpoint compute utilization. - C is Incorrect: `CPUUtilization` tracks CPU usage. Since the inference workload is GPU-bound, the CPU might remain at low utilization while the GPU is maxed out, leading to under-provisioning. > Key Takeaway: When hosting multiple models with varying compute profiles on a single GPU endpoint, scale based on hardware utilization metrics like `GPUUtilization` rather than request-count metrics.