Skip to content

Skill 3.2.10: Implement AI-specific resource scaling for GPU workloads

Question 1 - GPU Cluster Provisioning for Foundation Model Training

A financial services company is training a multi-billion parameter foundation model for transaction anomaly detection. The training run requires a cluster of interconnected GPU instances that can survive hardware failures without losing checkpoint progress. The architecture team requires the cluster to automatically replace faulty GPU instances, support the Slurm workload manager, and minimize operational overhead for infrastructure management.

Which AWS service should the machine learning engineer use to provision this GPU cluster? (Select ONE)

  • A) Amazon SageMaker Training with GPU-optimized hyperparameters
  • B) Amazon EKS with Nvidia GPU operator and a custom node group
  • C) Amazon SageMaker HyperPod for Slurm-based resilient and self-healing GPU clusters
  • D) Amazon EC2 Auto Scaling group with GPU instance types and spot capacity
  • E) Amazon SageMaker Inference with GPU-based real-time endpoints
Click to reveal Answer & Explanation

Correct Answer: C

Correct Answer Analysis

  • SageMaker HyperPod is purpose-built for distributed training of large-scale foundation models on GPU and accelerated instances. It provides resilient clusters that automatically detect and replace faulty nodes, supports both Slurm and EKS orchestration, and manages infrastructure lifecycles to minimize operational load.

Distractor Analysis

  • A is Incorrect: SageMaker Training jobs lack automatic node-level hardware fault recovery; if a single training instance fails mid-run, the job can be interrupted and restarted, losing progress unless manual checkpoint recovery is implemented.
  • B is Incorrect: Running EKS with the Nvidia GPU operator provides GPU orchestration but requires managing node replacement, job monitoring, and resilience logic manually. This adds significant operational complexity compared to managed HyperPod.
  • D is Incorrect: EC2 Auto Scaling can replace instances when they fail health checks, but it does not natively provide the task-aware resilient orchestration or integrated Slurm support necessary for long-running distributed GPU training jobs.
  • E is Incorrect: Inference endpoints are for serving predictions, not distributed model training. This option also conflates hosting with training.

Exam Tip: Choose SageMaker HyperPod for long-running distributed training workloads that demand hardware-failure resilience and workload manager support.

Question 2 - Scaling Policy for Variable GPU Inference Traffic

A generative AI startup hosts a Stable Diffusion image generation model on a SageMaker real-time endpoint backed by one GPU instance. Traffic is highly variable and bursts unpredictably (e.g., after launching influencer marketing campaigns). The team wants to minimize GPU idle spend during quiet periods while still handling bursts without intervention. The solution must respond to rapid traffic increases within seconds, not minutes.

Which scaling configuration should the team implement? (Select TWO)

  • A) Target tracking policy on SageMakerVariantInvocationsPerInstance
  • B) Target tracking policy on a high-resolution concurrency metric such as ConcurrentRequestsPerModel
  • C) Target tracking policy on GPUUtilization custom metric published every 60 seconds
  • D) Scheduled scaling actions for peak business hours only
  • E) Scale-in cooldown longer than scale-out cooldown to prevent flapping
Click to reveal Answer & Explanation

Correct Answers: B and E

Correct Answer Analysis

  • B is Correct: For bursty GPU inference workloads where requests climb in seconds, high-resolution concurrency metrics (published at 10-second granularity) allow Application Auto Scaling to react fast enough before GPU memory or queue depths are exhausted. This directly addresses "bursts in seconds."
  • E is Correct: GPU workloads are expensive. Setting a longer scale-in cooldown (e.g., 15–30 minutes) prevents rapid scale-in/out oscillations ("flapping") that would repeatedly deallocate and allocate expensive GPU instances, saving cost and reducing deployment churn during spiky traffic patterns.

Distractor Analysis

  • A is Incorrect: The standard invocation metric is aggregated at 1-minute intervals. For highly variable, second-level bursty GPU traffic, a per-minute metric causes delayed scale-out, risking dropped requests or prolonged latency during spikes.
  • C is Incorrect: A custom GPUUtilization metric is a valid fallback for cost drivers not based on request count. However, publishing it only every 60 seconds is incompatible with the "respond in seconds" requirement. Custom metrics also require validation and overhead.
  • D is Incorrect: Scheduled scaling is useful for entirely predictable (e.g., 9 AM EST) daily business-hour traffic, but cannot dynamically adapt to random marketing-driven or viral traffic spikes outside of those fixed schedules.

Exam Tip: For GPU inference, choose high-resolution concurrency metrics for fast burst response and tune cooldown asymmetrically (quick scale-out, slower scale-in) to protect expensive GPU capacity.

Question 3 - Multi-AZ GPU Endpoints for Model Artifact Access

A media company deploys a real-time video upscaling model (using NVIDIA GPU) on SageMaker with network isolation enabled. The endpoint is configured with private subnets in a VPC with no internet gateway. When attempting to deploy, the endpoint creation fails. The CloudWatch logs indicate the container cannot download the model artifacts from its S3 bucket.

What is the most likely cause of this failure, and what should the engineer do? (Select ONE)

  • A) The S3 bucket is in a different AWS Region, so the engineer should create a cross-region replication rule.
  • B) The endpoint's execution role lacks s3:GetObject, so the engineer should attach an S3 read-only policy.
  • C) The private subnets cannot reach Amazon S3 without an S3 Gateway VPC Endpoint. The engineer should create and attach one to the VPC route table.
  • D) SageMaker GPU instances do not support VPC network isolation, so the engineer should use a serverless endpoint instead.
  • E) The security group attached to the endpoint does not have an outbound rule allowing HTTPS to port 443.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Answer Analysis

  • C is Correct: When a SageMaker endpoint is placed in private subnets with no NAT or internet gateway, all outbound internet traffic is blocked. The container cannot reach the public Amazon S3 endpoint to pull model artifacts. Creating an S3 Gateway VPC Endpoint provides a private, AWS-backbone route from the subnets to S3, resolving the artifact download failure without internet access.

Distractor Analysis

  • A is Incorrect: SageMaker model artifacts are typically pulled from the account’s S3 bucket in the same region as the endpoint. Cross-region replication is used for disaster recovery or multi-region consistency, not for VPC egress to S3.
  • B is Incorrect: If the execution role lacked S3 permissions, the failure would be an AccessDenied error rather than a timeout or inability to reach the endpoint. VPC network isolation issues typically manifest as connection timeouts (no route).
  • D is Incorrect: SageMaker real-time endpoints do support VPC network isolation with GPU instance types. Serverless endpoints are a different hosting option, but the problem is network path, not hosting type.
  • E is Incorrect: Security Groups are stateful; an implicit outbound deny is typically not the issue unless explicitly configured. VPC endpoint access is controlled at the route table level. A missing route to S3 is the root cause.

Exam Tip: For private-subnet SageMaker deployments with no NAT, ensure S3 Gateway VPC Endpoints exist; without them, the inference or training containers cannot access bucket artifacts.

Question 4 - Provisioning Durable GPU Clusters with Self-Healing

A research lab uses Amazon SageMaker HyperPod to accelerate distributed training of a complex 70B parameter LLM. The lab wants to ensure that when a GPU node fails mid-training, the job does not get permanently halted; instead the cluster should swap in a healthy GPU node and resume from a checkpoint automatically, without the research team manually intervening.

Which two configurations or practices will satisfy this requirement? (Select TWO)

  • A) Configure SageMaker HyperPod with cluster self-healing enabled for node failure detection and replacement.
  • B) Use Amazon EC2 Spot Instances for the entire cluster to reduce costs, because Spot Instances automatically get replaced without job interruption when capacity is unavailable.
  • C) Integrate the training framework with Amazon SageMaker Checkpointing to asynchronously persist model state to S3 on a periodic basis.
  • D) Use Amazon CloudWatch Application Insights to alert the team when a node fails, so they can manually relaunch the training job with a fresh cluster.
  • E) Split the model across multiple individual SageMaker Training jobs (one per node), each with its own checkpoint in a local /opt/ml/model directory.
Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Answer Analysis

  • A is Correct: SageMaker HyperPod provides automatic self-healing for distributed training clusters. When a GPU node fails, HyperPod detects the unhealthy instance, replaces it, and can integrate with the training scheduler (Slurm or EKS) to resume the job. This removes manual infrastructure intervention.
  • C is Correct: Checkpointing is critical for fault-tolerant distributed training. Persisting model weights and optimizer states (e.g., to Amazon S3) at regular intervals allows the training process to resume from the last saved state after a node replacement, rather than starting from scratch.

Distractor Analysis

  • B is Incorrect: Spot Instances are reclaimed when AWS needs capacity back, and although HyperPod supports spot, uninterrupted robustness comes from health-based replacement plus checkpointing, not from spot’s inherent behavior. Spot interruption notices do not auto-replace and resume distributed training jobs without coordination.
  • D is Incorrect: Manual intervention via CloudWatch alerts violates the "no manual intervention" requirement. The goal is automatic self-healing and resume, not "alert and manually relaunch."
  • E is Incorrect: Splitting a 70B model across individual SageMaker Training jobs is an invalid distributed training approach for this context; it would not provide collective communication between nodes or a cohesive model state. Local /opt/ml/model directories are ephemeral and wiped after instance replacement.

Exam Tip: For long-running GPU training jobs, pair HyperPod self-healing with periodic S3 checkpointing to achieve auto-resume after node failures.

Question 5 - GPU Scaling Metric for Streaming Token Generation

A startup hosts a custom LLM streaming inference endpoint on SageMaker with a single GPU. Each request streams tokens back to the user over 30+ seconds. During load tests, they observe that near-concurrent requests quickly overwhelm the GPU, but InvocationsPerInstance stays low and normal. The endpoint’s scale-out reaction is too slow, causing queue buildup and elevated time-to-first-token (TTFT).

What is the most appropriate remediation? (Select ONE)

  • A) Lower the target value for SageMakerVariantInvocationsPerInstance to a very small number (e.g., 0.5) to trigger more aggressive scaling.
  • B) Switch the scaling metric to a high-resolution concurrency metric (e.g., ConcurrentRequestsPerModel) with target tracking based on concurrent request count.
  • C) Increase the endpoint instance size to the largest available GPU instance and keep the same scaling policy.
  • D) Change the storage volume from gp3 to io2 to reduce inference latency and better support streaming.
  • E) Configure scheduled scaling actions to add capacity every 30 minutes and remove every hour.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: For long-running, streaming LLM requests, invocation count is a poor signal because a single invocation may hold a GPU for tens of seconds. High-resolution concurrency metrics measure the number of simultaneous in-flight requests and are published at 10-second intervals, enabling faster scale-out when many tokens are being generated concurrently.

Distractor Analysis

  • A is Incorrect: Lowering the target of InvocationsPerInstance does not address the fundamental mismatch between "number of invocations" and "GPU occupancy time." Even with more scaling, the metric reacts slowly (1-minute aggregation) for second-level traffic bursts.
  • C is Incorrect: A larger GPU instance can handle more concurrent requests but does not solve auto-scaling timing problems. It also increases cost, especially if traffic remains spiky and unpredictable.
  • D is Incorrect: Storage volume performance (IOPS) is not the bottleneck for streaming LLM token generation, which is primarily memory-bandwidth and compute-bound on the GPU. This is a non-architectural distraction.
  • E is Incorrect: Scheduled scaling every 30 minutes is far too coarse for bursty LLM workloads and does not respond dynamically to traffic changes.

Exam Tip: Use concurrency-based metrics, not invocation-based metrics, for streaming or long-duration generative AI endpoint scaling.

Question 6 - Reserved GPU Capacity vs. On-Demand for Stable Inference Traffic

A healthcare diagnostics company runs an X-ray anomaly detection model on a SageMaker GPU endpoint. The traffic is steady and predictable across the year (hospital radiologists use it during regular clinical hours), and the organization wants to reduce costs while ensuring capacity is always available. The model is critical and cannot tolerate cold starts due to patient safety.

Which provisioning option is most suitable for this GPU workload? (Select ONE)

  • A) Use SageMaker Serverless Inference for high flexibility and zero cold-start latency, since traffic is predictable.
  • B) Use SageMaker Savings Plans or Reserved Instances for provisioned real-time GPU capacity with always-warm endpoints.
  • C) Use spot instances for all endpoint variants to reduce cost, and rely on auto replacement because the workload has fault tolerance.
  • D) Use AWS Fargate with GPU tasks for elastic scalable inference.
  • E) Use a batch transform job to run hourly to process all patient studies at once.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: When GPU inference traffic is steady and predictable, purchasing SageMaker Savings Plans or Reserved Instances commits to consistent GPU capacity at a significant discount (up to 64%+). Real-time endpoints stay warm, eliminating cold-start risk for latency-sensitive medical applications.

Distractor Analysis

  • A is Incorrect: SageMaker Serverless Inference has no provisioned GPU support today; it uses CPU-based compute and has cold-start times when the model is inactive. This conflicts with the GPU requirement and zero cold-start constraints.
  • C is Incorrect: Spot instances can be reclaimed at any time, causing endpoint termination and cold-start disruption. This violates the "critical and cannot tolerate cold starts" and patient safety requirement.
  • D is Incorrect: AWS Fargate does not support GPU workloads; GPUs require EC2-based container orchestration (EKS or ECS with GPU AMIs). This option refers to a non-existent Fargate GPU feature.
  • E is Incorrect: Batch transform jobs run asynchronously on a schedule and are unsuitable for real-time clinical diagnostic workloads that require near-instant inference results.

Exam Tip: Stable, always-on GPU inference workloads benefit from committed-use pricing (Savings Plans/Reserved) rather than serverless or spot for critical production systems.

Question 7 - Auto Scaling for GPU-Specific Metrics in SageMaker

A gaming analytics company hosts a reinforcement learning (RL) inference model on SageMaker GPU endpoint variants. They notice that GPU memory utilization consistently exceeds 90% before the Application Auto Scaling policy triggers a scale-out, causing occasional out-of-memory errors and dropped requests. The team wants to scale before GPU memory becomes the bottleneck.

Which approach should the company take? (Select ONE)

  • A) Increase the endpoint's TargetModel size in Application Auto Scaling to force more capacity before memory runs out.
  • B) Publish a custom CloudWatch metric for GPUUtilization or GPUMemoryUtilization from the instance, then create a target tracking policy scaling on that metric with a target value below the memory saturation threshold.
  • C) Use InvocationsPerInstance with a very large target value to reduce false scale-out triggers.
  • D) Use Amazon EventBridge to detect OutOfMemoryError events and then invoke a Lambda that manually increases instance count.
  • E) Move the RL model to CPU instances to avoid GPU memory pressure entirely.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: AWS Application Auto Scaling supports custom CloudWatch metrics. For GPU-bound reinforcement learning inference, publishing GPUMemoryUtilization (or GPUUtilization) from the NVIDIA driver or a monitoring agent to CloudWatch, then creating a target tracking scaling policy with an appropriate target (e.g., 70%), enables proactive scaling before resources become saturated and requests fail.

Distractor Analysis

  • A is Incorrect: TargetModel is not a valid Application Auto Scaling parameter for SageMaker endpoint variants. Target values apply to metrics (e.g., InvocationsPerInstance), not model size.
  • C is Incorrect: Increasing the target for InvocationsPerInstance makes scaling less aggressive, causing more memory exhaustion, not less. This is counterproductive for the stated problem.
  • D is Incorrect: Reacting to OutOfMemoryError events is a reactive, non-scalable workaround. Adding a custom pipeline with EventBridge and Lambda increases operational overhead and still results in dropped requests before intervention occurs.
  • E is Incorrect: Moving to CPU instances eliminates the GPU workload entirely, but may drastically reduce throughput and performance for RL inference models. The requirement is to scale GPU instances before memory bottlenecks, not abandon GPUs.

Exam Tip: For GPU-bound workloads, leverage custom CloudWatch metrics (e.g., GPU memory) with target tracking to scale proactively before hitting hardware limits.

Question 8 - Multi-Node GPU Scaling with View of Cluster Health

A machine learning platform team runs distributed GPU training across multiple SageMaker HyperPod nodes using Kubernetes (EKS). They need to monitor cluster health, GPU utilization, and job status from a single pane of glass. They want a solution that integrates with Amazon CloudWatch for alerts and dashboards without writing custom observability code.

Which combination of services and configurations meets these requirements? (Select TWO)

  • A) Deploy a Prometheus and Grafana stack entirely on an EC2 instance, scraped by a custom Python script that writes to S3.
  • B) Use Amazon Managed Service for Prometheus (AMP) to collect GPU and node metrics from an EKS cluster via the AWS Distro for OpenTelemetry (ADOT) collector.
  • C) Rely solely on Amazon CloudWatch Container Insights with nvidia-smi output written to a local file system; no need for metric export.
  • D) Use Amazon CloudWatch Container Insights with enhanced observability for EKS to collect pod-level and node-level GPU metrics.
  • E) Store all metrics in an Amazon RDS MySQL database and build a custom Node.js dashboard.
Click to reveal Answer & Explanation

Correct Answers: B and D

Correct Answer Analysis

  • B is Correct: Amazon Managed Service for Prometheus (AMP) is a managed observability service that natively integrates with EKS. The AWS Distro for OpenTelemetry (ADOT) collector can scrape Kubernetes metrics, including GPU-specific data exposed by NVIDIA DCGM exporters, and send them to AMP for persistent storage and querying. This eliminates manual Prometheus management.
  • D is Correct: Amazon CloudWatch Container Insights with enhanced observability for EKS provides GPU metrics (e.g., nvidia_smi_utilization_gpu, nvidia_smi_memory_used) at a cluster level, with built-in dashboards and alarm integration, directly matching the "single pane of glass + CloudWatch" requirement.

Distractor Analysis

  • A is Incorrect: Running a self-managed Prometheus/Grafana EC2 instance with custom S3 scripts introduces unnecessary operational overhead and fails the "without writing custom observability code" constraint. Managed services are preferred.
  • C is Incorrect: CloudWatch Container Insights does collect GPU metrics when configured with the NVIDIA GPU monitoring add-on, but simply writing nvidia-smi output to a local file without exporting it to CloudWatch does not meet the dashboarding requirement.
  • E is Incorrect: Building a custom MySQL/RDS-backed monitoring system is excessive, adds operational complexity, and is not natively integrated with CloudWatch for alerts, dashboards, and threshold-based alarms.

Exam Tip: For GPU metric observability on EKS, prefer managed solutions like CloudWatch Container Insights enhanced observability or Amazon Managed Service for Prometheus with OTel collectors.

Question 9 - Warm Pool Strategy for GPU Training Jobs

A biotech company runs weekly distributed GPU training jobs on Amazon SageMaker using HyperPod with a fixed number of nodes. Between training runs, the nodes remain idle but there is a high setup cost to re-provisioning and re-attaching storage volumes, network interfaces, and container images. The team wants to keep the GPU cluster "warm" for rapid job submission while minimizing idle compute cost.

Which strategy best meets these requirements using SageMaker HyperPod? (Select ONE)

  • A) Use AWS Batch with GPU compute environments where jobs are submitted weekly.
  • B) Use a HyperPod cluster with a KeepAlive period configured to retain idle nodes between jobs, so桙that subsequent training jobs can reuse the cluster without re-provisioning.
  • C) Delete the HyperPod cluster after each job and launch a new one from an AWS CloudFormation template.
  • D) Use EC2 Spot Fleet for all GPU instances with persistent spot requests and no maintenance windows.
  • E) Use Amazon SageMaker Automatic Model Tuning jobs triggered by Amazon EventBridge rules every week.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: SageMaker HyperPod supports KeepAlive periods for cluster nodes. A KeepAlive period defines how long the cluster infrastructure (e.g., GPU nodes, network attachments, instance storage) remains provisioned and ready for the next job after a training job completes. This enables rapid job reuse with reduced re-provisioning overhead, while still allowing rare maintenance windows. The team can balance warm capacity with idle cost based on the chosen KeepAlive duration.

Distractor Analysis

  • A is Incorrect: AWS Batch is a valid alternative for GPU job scheduling, but it does not integrate with SageMaker HyperPod’s self-healing and cluster-level management. It also introduces a separate orchestration layer that is not needed here.
  • C is Incorrect: Deleting and rebuilding the HyperPod cluster weekly introduces significant operational overhead and defeats the purpose of keeping capacity warm. Provisioning new HyperPod clusters is time-consuming and may involve replacing network and storage.
  • D is Incorrect: EC2 Spot Fleets are not integrated with HyperPod’s scheduling and self-healing. Relying solely on spot instances for mission-critical weekly jobs risks capacity interruptions and does not provide the managed cluster experience requested.
  • E is Incorrect: Automatic Model Tuning jobs are used for hyperparameter optimization, not for managing cluster warm pools or idle GPU infrastructure between training jobs.

Exam Tip: For weekly or periodic GPU training loads, use HyperPod KeepAlive settings to recycle cluster capacity between jobs, avoiding cold-start re-provisioning overhead.

Question 10 - Serverless GPUs and Provisioned Capacity Scaling

A startup building a real-time AI coding assistant (based on a code-generation model) wants to host inference on Amazon Bedrock with an inference profile using provisioned throughput. During peak development hours (weekdays), demand is high but during off-hours, it drops significantly. The team wants fully managed GPU scaling with low operational overhead and the ability to scale provisioned capacity based on demand.

Which AWS capability should the engineering team use for this Bedrock-hosted model? (Select ONE)

  • A) Use SageMaker Inference Components to attach GPUs to a serverless endpoint
  • B) Use a Bedrock provisioned throughput model with Application Auto Scaling registered on InvocationsPerModel
  • C) Use Amazon Bedrock's built-in model invocation scaling by setting max_concurrency to auto
  • D) Use a custom AWS Lambda function that invokes IncreaseProvisionedThroughput API during peak hours
  • E) Use AWS EC2 Auto Scaling with a custom GPU inference server on EC2 instances that call the Bedrock model over the internet
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: Amazon Bedrock provisioned throughput can be registered as a scalable target with Application Auto Scaling. By attaching a target tracking policy to the provisioned throughput model, Bedrock can automatically scale the provisioned capacity (in model units) based on utilization metrics, maintaining low latency for the coding assistant during peak hours without manual infrastructure operations.

Distractor Analysis

  • A is Incorrect: SageMaker Inference Components are a feature for multi-model endpoints on SageMaker, not for Bedrock provisioned throughput. Serverless endpoints do not support GPU attachments currently, and this option incorrectly mixes services.
  • C is Incorrect: Amazon Bedrock does not expose a max_concurrency auto parameter to users. Bedrock manages throttling and quality of service via provisioned throughput and rate limits, not a single auto-scaling toggle for concurrency.
  • D is Incorrect: Building a custom Lambda to call IncreaseProvisionedThroughput introduces unnecessary operational complexity and does not react dynamically to traffic. Native Application Auto Scaling is the managed, low-overhead answer.
  • E is Incorrect: Running a custom GPU inference server on EC2 and calling Bedrock over the internet adds multiple layers of complexity, network egress, and defeats the purpose of using Bedrock as a managed service. Bedrock provisioned throughput natively supports scaling.

Exam Tip: Amazon Bedrock provisioned throughput integrates with Application Auto Scaling, so register the model as a scalable target and use target tracking policies for automatic, managed GPU capacity scaling.