Skill 3.2.1: Optimize resource provisioning between on-demand and provisioned resources for performance and cost efficiency

Question 1 - On-Demand vs. Provisioned Concurrency for Serverless Endpoints

A financial services company is deploying a transaction risk-scoring model on Amazon SageMaker AI Serverless Inference. The model analyzes individual transactions in real-time and must return predictions within 100 milliseconds to avoid delaying payment processing.

Traffic analysis shows the endpoint receives approximately 2,500 requests per hour during business hours, with sudden spikes of up to 800 requests within a 30-second window during market opening and closing periods. The ML engineering team needs to minimize cold-start latency during these spikes while keeping costs predictable for the steady-state traffic.

Which configuration should the team implement to meet the latency requirements with the least operational overhead?

A) Configure a real-time endpoint with ml.g4dn.xlarge instances and a target-tracking scaling policy on SageMakerVariantInvocationsPerInstance.
B) Configure a serverless endpoint with 0 provisioned concurrency to allow automatic scaling based on incoming traffic.
C) Configure a serverless endpoint with provisioned concurrency set to the expected steady-state request rate, allowing on-demand concurrency to absorb spikes.
D) Configure a serverless endpoint with provisioned concurrency set to the peak spike rate, ensuring all traffic is handled without cold starts.

Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

C is Correct: Serverless endpoints support provisioned concurrency to pre-warm capacity for predictable baseline traffic. Setting provisioned concurrency to the steady-state request rate (approximately 2,500 requests/hour ≈ 42 requests/min) ensures the endpoint can handle normal business-hours traffic without cold starts. When traffic spikes (800 requests/30s ≈ 1,600 requests/min), the serverless endpoint automatically scales using on-demand concurrency to absorb the surge. This balances latency requirements with cost efficiency, as the company pays a discounted rate for provisioned concurrency and only pays standard rates for the burst capacity.

Distractor Analysis

A is Incorrect: Real-time endpoints require managing instance fleets, scaling policies, and cooldown periods. This introduces significant operational overhead compared to serverless inference and is unnecessary for a serverless-friendly workload with predictable traffic.
B is Incorrect: Zero provisioned concurrency means the endpoint starts with no warmed capacity. During business hours, cold starts on initial requests could occur, potentially exceeding the 100ms latency requirement. Provisioned concurrency is specifically designed to eliminate cold-start latency for predictable traffic.
D is Incorrect: While provisioning to the peak spike rate eliminates all cold starts, it forces the company to pay for high concurrency capacity (1,600 requests/min capacity) at all times—even during low-traffic periods. This is cost-inefficient compared to using on-demand concurrency for short-duration spikes.

Key Takeaway: Use provisioned concurrency for predictable baseline traffic and on-demand concurrency for bursty spikes—this is the optimal cost/latency balance for serverless endpoints.

Question 2 - Reserved Instances vs. On-Demand for Real-Time Endpoint

A healthcare analytics company is hosting a clinical note classification model on Amazon SageMaker AI Real-Time Inference using ml.m5.2xlarge instances. The model has been in production for 18 months with consistent traffic patterns, and the company has committed to running this workload for at least another 2 years due to regulatory approval cycles.

The endpoint currently runs 4 instances continuously to handle the steady-state workload. The finance team is pressuring the ML team to reduce infrastructure costs, while maintaining identical performance characteristics. The team can sign a managing savings agreement with AWS.

Which TWO actions would provide the most cost-effective solution while maintaining the exact same inference performance? (Select TWO)

A) Switch from SageMaker Real-Time Inference to SageMaker Multi-Model Endpoints to share capacity across multiple models.
B) Purchase SageMaker Savings Plans for the committed usage over the next 1-3 years.
C) Convert the endpoint to use provisioned Inferential components with purchased reserved capacity.
D) Deploy the model on SageMaker Serverless Inference to eliminate instance management costs.
E) Reduce the instance fleet to 2 instances during off-peak hours using a scheduled scaling policy.

Click to reveal Answer & Explanation

Correct Answers: B and C

Correct Options Analysis

B is Correct: SageMaker Savings Plans (or AWS Compute Savings Plans applicable to SageMaker) provide significant discounts (typically 42-64%) in exchange for a 1 or 3-year commitmented spend. Given the company has committed to 2+ years of stable production, this is the most straightforward cost optimization with zero operational changes.
C is Correct: For SageMaker Real-Time Inference, Reserved Instance capacity provides significant discounts and guarantees capacity reservations. Since the workload is stable and predictable, purchasing reserved capacity for the equivalent instance count/family directly reduces cost without performance changes.

Distractor Analysis

A is Incorrect: Multi-Model Endpoints share instances across multiple models, useful for cost savings when many models have low individual traffic. However, this introduces resource contention risk and requires re-architecting the endpoint—violating the "maintain identical performance" requirement.
D is Incorrect: Serverless Inference reduces operational overhead but does not guarantee identical performance (cold starts, max concurrency limits, different latency profiles). For an established real-time endpoint with stable traffic, reserved capacity on real-time is more cost-predictable.
E is Incorrect: While scheduled scaling saves cost, it changes performance characteristics during off-peak hours (cold starts when scaling back up, potential capacity issues during unexpected traffic). This violates the "maintain identical performance" requirement.

Key Takeaway: For committed, predictable workloads on real-time inference, SageMaker Savings Plans or Reserved Instances provide significant cost savings without changing architecture or performance.

Question 3 - GPU Provisioned Throughput for Bedrock Foundation Models

A document-processing company is building an Amazon Bedrock Agent that extracts structured fields from legal contracts using Amazon Bedrock Custom Model Import with a fine-tuned Llama 3.1 8B model.

The agent processes an average of 15 documents per hour during business hours (9 AM - 6 PM, Monday through Friday). Each document requires approximately 120 seconds of processing time, and each document processing task allows for up to 10 minutes of latency. Users do not require concurrent document processing—documents queue sequentially in the agent workflow.

The ML engineer must provision the model with the most cost-efficient option that meets the throughput demands.

Which provisioned throughput configuration should the engineer choose?

A) Purchase Amazon Bedrock Provisioned Throughput with a 1-month commitment, set to 1 model unit, and configure the agent to process documents sequentially.
B) Use Amazon Bedrock On-Demand for the imported custom model to avoid any committed spend.
C) Purchase Amazon Bedrock Provisioned Throughput with a 6-month commitment, set to 4 model units to handle worst-case load.
D) Use Amazon Bedrock On-Demand during business hours and manually switch to Provisioned Throughput after hours.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

A is Correct: Amazon Bedrock custom imported models require Provisioned Throughput—on-demand access applies to native Bedrock foundation models only. One model unit provides adequate processing capacity for 15 documents/hour (each taking 120s). A 1-month commitment is the shortest available and allows for re-evaluation, fitting the "most cost-efficient" requirement for a light, intermittent workload.

Distractor Analysis

B is Incorrect: Amazon Bedrock Custom Model Import models cannot be accessed via on-demand pricing. Provisioned Throughput is mandatory for custom imported models, making this option technically impossible.
C is Incorrect: Four model units provides far more capacity than required (the workload can process sequentially with 1 MU handling approximately 45 requests/hour). A 6-month commitment locks in costs for longer than necessary for a light workload.
D is Incorrect: As with Option B, custom imported models cannot use on-demand. Additionally, a manual switching process introduces operational overhead and a 1-month provisioned throughput commitment cannot be paused/resumed arbitrarily.

Key Takeaway: Custom imported models in Amazon Bedrock are always provisioned throughput—a 1-month commitment with minimal model units is the cost-efficient choice for low-throughput workloads.

Question 4 - SageMaker HyperPod vs. Standard SageMaker Training

An autonomous vehicle startup is training a large vision transformer model for object detection using distributed data parallelism across 32 GPU instances. The training run is estimated to take 3 weeks to complete.

During previous training runs, the team experienced 2-3 node failures per week due to hardware issues, each requiring manual intervention (identify failed node, remove from cluster, restart training on remaining nodes). This increased total training time by over 35%.

The lead ML engineer must choose a training infrastructure that automatically recovers from node failures while maintaining the existing training code and framework compatibility.

Which solution should the engineer implement?

A) Use standard SageMaker Training Jobs with ml.p4d.24xlarge instances and implement a custom checkpoint/restart script using AWS Lambda to detect failures.
B) Use Amazon SageMaker HyperPod with auto-resume capabilities for the SageMaker-managed Slurm orchestrator.
C) Use Amazon EKS with the SageMaker operator for Kubernetes to manage distributed training, accepting manual node replacement.
D) Use AWS Batch with GPU instance types and retry policies to resubmit failed training jobs.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: SageMaker HyperPod is specifically designed to provide resilient training infrastructure for large-scale, long-running foundation model training jobs. It automatically detects faulty instances (GPU, CPU, memory issues), replaces them, and resumes training from the last checkpoint. HyperPod supports both SageMaker-managed Slurm and Amazon EKS orchestrators, and maintains compatibility with existing training frameworks (PyTorch, TensorFlow, etc.).

Distractor Analysis

A is Incorrect: Standard SageMaker Training Jobs do not provide automatic node recovery. If a node fails, the training job fails. Building custom Lambda-based failure detection and restart logic introduces significant operational overhead and doesn't fully automate the recovery process as cleanly as HyperPod.
C is Incorrect: Amazon EKS with the SageMaker operator provides Kubernetes-based orchestration but requires manual or custom node replacement when hardware fails. There is no built-in automatic node recovery mechanism.
D is Incorrect: AWS Batch retry policies can resubmit a failed job, but this restarts training from the beginning unless the training script handles checkpoints manually. Additionally, AWS Batch lacks built-in health monitoring for GPU instances during large-scale distributed training.

Key Takeaway: SageMaker HyperPod is the only AWS service that provides automatic node replacement and job auto-resume for long-running distributed training workloads.

Question 5 - Application Auto Scaling Metrics for Streaming Responses

A generative AI startup is hosting a Llama 3.1 70B model on SageMaker Real-Time Inference using ml.g5.12xlarge instances. The model generates conversational responses that stream token-by-token back to the client, with average response times of 30 seconds per request.

The engineering team has configured a target-tracking scaling policy on SageMakerVariantInvocationsPerInstance with a target of 20 invocations per instance. However, during traffic spikes, the endpoint responds extremely slowly due to instance overload, and the scaling policy reacts too late to maintain acceptable latency.

The model's main bottleneck is GPU memory utilization due to concurrent streaming responses. Which scaling approach would provide the most responsive scaling for this workload?

A) Increase the target value on SageMakerVariantInvocationsPerInstance to 40 to prevent overloading.
B) Configure target tracking on the high-resolution ConcurrentRequestsPerModel metric to scale out when concurrent streaming requests exceed the instance capacity.
C) Configure step scaling on GPUUtilization CloudWatch metric with a scale-out step adding 1 instance when GPU utilization exceeds 90%.
D) Configure a scheduled scaling policy that adds 3 instances during known high-traffic hours.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: For streaming generative AI workloads with long-response windows, orchestration metrics like InvocationsPerInstance are weak indicators of current choke capacity because they count completed requests. The ConcurrentRequestsPerModel metric (or ConcurrentRequestsPerCopy) is a high-resolution CloudWatch metric that tracks in-flight concurrent requests, ensuring the fastest possible auto-scaling responses for long-running inference requests. Target tracking on concurrent requests scales rapidly when many streaming responses overlap.

Distractor Analysis

A is Incorrect: Increasing the target threshold only makes each instance handle more requests before scaling, which would worsen the overload situation and increase latency further.
C is Incorrect: While GPU utilization can be a valid custom scaling metric, step scaling with static thresholds is slower to react than target tracking on high-resolution concurrency metrics. Additionally, GPU memory—not GPU compute—is the indicated constraint, making utilization possibly an incomplete signal.
D is Incorrect: Scheduled scaling only adds capacity at fixed calendar events, but traffic spikes are not strictly predictable. This approach cannot dynamically respond to sudden, unexpected global traffic.

Key Takeaway: For streaming or long-running inference, use high-resolution concurrency metrics (e.g., ConcurrentRequestsPerModel) for scaling—not invocation counts.

Question 6 - On-Demand vs. Reserved SageMaker Hosting for Batch Workloads

A retail analytics team processes nightly sales reports using SageMaker Batch Transform to generate product demand forecasts. The batch jobs run every night at 11 PM and process approximately 500,000 records (total ~2 GB of data) in about 45 minutes using ml.m5.xlarge instances.

The ML engineer is reviewing infrastructure costs and wants to minimize spending on inference while ensuring the batch jobs complete before the 6 AM business start time.

Which approach provides the most cost-effective inference solution?

A) Replace Batch Transform with a SageMaker Real-Time endpoint running ml.m5.xlarge instances with reserved capacity.
B) Use SageMaker Batch Transform with on-demand instances, running only during the nightly processing window.
C) Use SageMaker Batch Transform with SageMaker Savings Plans for the nightly compute.
D) Replace Batch Transform with SageMaker Serverless Inference and invoke the endpoint 500,000 times nightly.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: SageMaker Batch Transform with on-demand instances is the ideal solution for nightly, offline batch workloads. The compute resources are provisioned only for the 45-minute processing window and terminate afterward, meaning the company only pays for actual usage. No committed or reserved spend is required for a workload that runs less than 10 hours per week.

Distractor Analysis

A is Incorrect: A Real-Time endpoint running 24/7 is vastly over-provisioned for a batch job that only needs 45 minutes of processing per night. Reserved capacity would lock in costs for infrastructure that's idle over 97% of the time.
C is Incorrect: SageMaker Savings Plans are designed for predictable, continuous compute usage (e.g., real-time endpoints, long-running training jobs). For only 45 minutes of nightly inference, the committed usage pattern is too small to benefit meaningfully from Savings Plans pricing.
D is Incorrect: Invoking a Serverless Inference endpoint 500,000 times nightly would incur per-request charges and potentially exceed concurrency limits, making it more expensive than Batch Transform for large batch scoring workloads. Batch Transform is specifically optimized for large batch inference.

Key Takeaway: Batch Transform with on-demand instances is the most cost-effective solution for periodic, large-scale batch inference—no committed capacity required.

Question 7 - Provisioned Throughput vs. On-Demand for Bedrock Knowledge Bases

A legal services company is building a conversational document Q&A system using Amazon Bedrock Knowledge Bases with the Anthropic Claude 3.5 Sonnet foundation model for answer generation.

The system is used by paralegals during business hours (9 AM - 6 PM, Monday through Friday) to answer client queries. Peak usage analysis shows that approximately 1,500 queries per hour are processed during normal operations. Each query invokes the FM once for answer generation (excluding retrieval calls). The company is sensitive to cost and is evaluating Bedrock pricing options.

The ML engineer must choose a provisioning strategy for the foundation model access. Which statement accurately describes the most cost-efficient approach?

A) Purchase Provisioned Throughput for the Claude 3.5 Sonnet model because it provides a lower per-token cost than on-demand at all usage levels.
B) Use On-Demand pricing for Claude 3.5 Sonnet, as the moderate, intermittent query volume likely does not justify committed Provisioned Throughput spend.
C) Purchase Provisioned Throughput with a 6-month commitment to guarantee model availability during peak hours.
D) Migrate the workload to a custom imported model accessed via on-demand pricing to reduce token costs.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: For moderate, non-continuous workloads (1,500 queries/hour over a 9-hour workday), on-demand pricing is the correct default. On-demand provides instant access to models like Claude 3.5 Sonnet with no committed spend or capacity management. Provisioned Throughput is intended for high-throughput, sustained workloads where predictable throughput and latency SLAs justify the 1-6 month committed investment.

Distractor Analysis

A is Incorrect: While Provisioned Throughput can offer lower per-token costs at scale, it requires a minimum committed spend (e.g., level-tied model units) that makes it expensive for moderate usage. On-demand is superior when total token volume is below the break-even threshold.
C is Incorrect: On-demand Bedrock models are always available; Provisioned Throughput is not required for "availability guarantees" at 1,500 queries/hour.
D is Incorrect: Custom imported models require Provisioned Throughput and cannot be accessed via on-demand. Additionally, Claude 3.5 Sonnet is not a custom importable model—custom imports apply to select open-weight models like Llama or Mistral.

Key Takeaway: On-demand Bedrock pricing is optimal for moderate, intermittent workloads; Provisioned Throughput is for sustained, high-throughput production environments.

Question 8 - Multi-Model Endpoints for Cost Optimization

A mid-sized SaaS company has developed 12 domain-specific classification models (each performing text classification for different customer verticals). Each model receives an average of 50-100 inference requests per hour, far below the capacity of a single ml.m5.large instance.

Currently, each model is deployed on its own SageMaker Real-Time endpoint, resulting in 12 separate endpoints and 12 running instances. The finance team has requested significant infrastructure cost reduction.

The ML engineer must consolidate these models while maintaining separate API endpoints for each model for client compatibility.

Which solution provides the most cost-effective deployment with the least architectural change?

A) Deploy all 12 models to a SageMaker Multi-Model Endpoint sharing one instance, maintaining the same invocation URL with a model-specific suffix.
B) Deploy all 12 models to a single SageMaker endpoint as a single merged model that routes to the appropriate classifier based on input.
C) Migrate all 12 models to SageMaker Serverless Inference to eliminate infrastructure costs entirely.
D) Reduce each endpoint to the smallest possible instance type (ml.t2.medium) to minimize per-endpoint costs.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

A is Correct: SageMaker Multi-Model Endpoints (MME) are the classic solution for consolidating many low-traffic models onto shared compute infrastructure. MME keeps each model deployable independently and accessible via distinct TargetModel parameters in the invocation URL, maintaining client-facing API compatibility while reducing from 12 instances to as few as 1 instance.

Distractor Analysis

B is Incorrect: Merging 12 models into a single binary requires re-training a routing layer combined with all original model artifacts—significant architectural re-work and potential model drift.
C is Incorrect: While Serverless Inference would eliminate instance management, it may not be the "least architectural change" compared to MME. MME allows the same real-time endpoint configuration and artifact storage pattern while Serverless Inference has cold start characteristics and inference size/concurrency limits to evaluate.
D is Incorrect: Reducing instance sizes only reduces per-instance cost marginally—you still pay for 12 separate endpoints with 12 separate instances. This does not solve the fundamental over-provisioning problem.

Key Takeaway: Multi-Model Endpoints are the optimal consolidation pattern for fleets of low-traffic models, reducing cost while preserving per-model API isolation.

Question 9 - Auto Scaling with Custom GPU Memory Metrics

A computer vision startup is running an image segmentation model on SageMaker Real-Time Inference using ml.g4dn.xlarge instances (1 GPU per instance). The model processes high-resolution satellite imagery with batch sizes of 1 image per request.

The team has noticed that the endpoint becomes unstable when GPU memory utilization exceeds 85%, causing out-of-memory errors. The engineering team wants to configure auto scaling based on a custom metric that directly reflects the constrained resource.

What is the most effective approach to configure this scaling?

A) Configure a target-tracking policy on SageMakerVariantInvocationsPerInstance with a low target value.
B) Configure a scaling policy using the GPUUtilization predefined CloudWatch metric from the SageMaker endpoint.
C) Publish custom GPUResidentMemoryUtilization metrics (via CloudWatch custom metrics from within the inference container) and configure target tracking on that metric.
D) Configure step scaling on CPUUtilization with subsequent adjustment steps based on memory pressure indicators.

Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

C is Correct: When the constraint is a resource metric not directly exposed by default (e.g., GPU resident memory utilization), the best practice is to publish a custom CloudWatch metric from within the inference container and then configure target tracking on that custom metric. This gives a scaling signal directly proportional to capacity utilization of the bottleneck resource.

Distractor Analysis

A is Incorrect: InvocationsPerInstance is a request-rate metric; it does not measure GPU memory pressure directly. Different images can cause different GPU memory utilization based on size and model complexity.
B is Incorrect: As of current AWS documentation, SageMaker's predefined CloudWatch metrics include GPUUtilization (GPU compute utilization) but not GPUResidentMemoryUtilization as a built-in metric. The memory metric must be published as a custom metric.
D is Incorrect: CPU utilization is poorly correlated with GPU memory pressure in GPU-intensive workloads, making it an ineffective scaling trigger for this scenario.

Key Takeaway: Use custom CloudWatch metrics for bottleneck resources (like GPU memory) not covered by built-in metrics, and configure target tracking for dynamic scaling.

Question 10 - Bedrock Foundation Model Cost Optimization with Model Selection

A customer service automation company is building a ticket classification and summarization pipeline using Amazon Bedrock.

The pipeline has two stages:

Stage 1: A fine-tuned Llama 3.1 8B model (imported as a custom model) classifies incoming support tickets into categories (30-minute average latency tolerance).
Stage 2: A Claude 3.5 Sonnet foundation model generates brief customer-facing summaries (5-minute average latency tolerance).

The pipeline processes approximately 300 tickets per day during weekdays. The ML architect must minimize foundation model inference cost while meeting all performance requirements.

Which AWS Bedrock configuration should the architect select for the Llama 3.1 8B classified model?

A) Access the custom imported model via Bedrock on-demand pricing since traffic volume is low.
B) Purchase 1 model unit of Provisioned Throughput for the custom imported model with a 1-month commitment.
C) Deploy the fine-tuned model on SageMaker Real-Time Inference with on-demand instances to avoid Bedrock pricing.
D) Use Amazon Bedrock Model Distillation to convert the custom model into a smaller, on-demand accessible model.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: All custom imported models in Amazon Bedrock require Provisioned Throughput. A 1-month commitment with 1 model unit is the minimum, most cost-effective configuration for a low-throughput custom model workload (300 tickets/day). Provisioned Throughput offers instant, bounded latency for the classifier stage.

Distractor Analysis

A is Incorrect: Amazon Bedrock does not support on-demand pricing for custom imported models—this is only available for native Bedrock foundation models.
C is Incorrect: While technically feasible, deploying on SageMaker Real-Time Inference introduces significantly more operational overhead (instance management, scaling policies, versioning) compared to using Bedrock's managed Provisioned Throughput. This is a less efficient use of engineering effort when Bedrock can already host the model.
D is Incorrect: Amazon Bedrock Model Distillation is not a service feature. Distillation is a model training technique that requires custom training infrastructure (e.g., SageMaker). Additionally, distilling a model would require retraining and potentially impact classification accuracy.

Key Takeaway: Custom imported models in Bedrock require Provisioned Throughput; use the minimum model units and shortest commitment for low-throughput workloads to minimize cost.