Skip to content

Skill 4.2.1: Select inference instance families to optimize performance and cost

Question 1 - GPU vs CPU Instance Family Selection for Transformer Inference

A financial services company is deploying a BERT-based transformer model for real-time document classification on Amazon SageMaker AI. The model has approximately 110 million parameters, serves roughly 50 requests per second with strict latency requirements under 100 milliseconds, and processes sequences with an average length of 128 tokens. The engineering team must choose an instance family that balances inference latency, cost-efficiency, and operational simplicity for production deployment.

Which instance family should the team select as the primary choice for this workload? (Select ONE)

  • A) ml.c5.xlarge — Compute Optimized CPU instance with 4 vCPUs and 8 GiB of memory
  • B) ml.g4dn.xlarge — GPU instance with NVIDIA T4, 4 vCPUs, and 16 GiB of memory
  • C) ml.inf1.xlarge — AWS Inferentia-based instance with 4 vCPUs and 8 GiB of memory
  • D) ml.t3.medium — General purpose Burstable CPU instance with 2 vCPUs and 4 GiB of memory
  • E) ml.p4d.24xlarge — High-end GPU instance with 8x NVIDIA A100 GPUs and 96 vCPUs
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: The ml.g4dn.xlarge with an NVIDIA T4 GPU provides the most cost-effective accelerated compute for a BERT-based transformer model serving 50 TPS at sub-100ms latency. T4 GPUs are optimized for inference workloads and provide an excellent price-performance ratio for models of this size.

Distractor Analysis

  • A is Incorrect: While ml.c5.xlarge could technically serve a 110M parameter model, CPU inference for transformer models at 50 TPS with sub-100ms latency is unlikely to be achievable without spilling into higher compute, and would require more instances than a single GPU instance for equivalent throughput.
  • C is Incorrect: ml.inf1 instances are compelling for cost-optimization but require compiling the model to AWS Neuron SDK format. The operational overhead of compilation may be justified for larger models, but for a simple BERT-base model, the T4 GPU offers simpler deployment with excellent performance.
  • D is Incorrect: ml.t3.medium is a burstable general-purpose instance with insufficient CPU and memory to reliably serve a 110M parameter transformer model within 100ms latency at 50 TPS.
  • E is Incorrect: ml.p4d.24xlarge is a massive, expensive training-oriented instance. It provides 8 A100 GPUs, which is over-provisioned for a 110M parameter inference workload and would incur significant unnecessary cost.

Key Takeaway: For small-to-medium transformer models requiring real-time inference, the g4dn GPU family typically offers the right balance of latency, throughput, and cost; inf1 becomes attractive when you're willing to invest in Neuron compilation for substantial cost savings at scale.

Question 2 - Graviton vs x86 for Lightweight ML Inference

A retail analytics company operates a gradient-boosted decision tree (XGBoost) model for fraud detection on Amazon SageMaker AI. The model is small (25 MB), receives consistent traffic of approximately 200 requests per second with a PLAT latency requirement of 10 milliseconds, and the team wants to minimize compute costs while maintaining simplicity. The model does not require GPU acceleration and has been containerized with SageMaker's built-in XGBoost container.

Which instance family should the team use to achieve the lowest cost per inference for this CPU-bound, real-time workload? (Select ONE)

  • A) ml.m5.large — General Purpose x86 instance with 2 vCPUs and 8 GiB of memory
  • B) ml.c5.large — Compute Optimized x86 instance with 2 vCPUs and 4 GiB of memory
  • C) ml.g4dn.xlarge — GPU-accelerated instance with NVIDIA T4
  • D) ml.m6g.large — General Purpose AWS Graviton2 instance with 2 vCPUs and 8 GiB of memory
  • E) ml.inf1.xlarge — AWS Inferentia-based instance
Click to reveal Answer & Explanation

Correct Answer: D

Correct Options Analysis

  • D is Correct: The ml.m6g.large with AWS Graviton2 processors provides up to 40% better price-performance compared to equivalent x86 instances (m5). For a lightweight XGBoost model that is CPU-bound and uses SageMaker's built-in containers (which support Graviton), this gives the lowest cost per inference while meeting the latency requirement.

Distractor Analysis

  • A is Incorrect: ml.m5.large works correctly but costs approximately 20-40% more than the Graviton2 equivalent for the same vCPU and memory configuration, failing to minimize costs.
  • B is Incorrect: ml.c5.large is compute-optimized but offers no meaningful advantage over m6g for a small tree-based model at 200 TPS. Its smaller memory footprint (4 GiB) offers less headroom for request batching.
  • C is Incorrect: GPU acceleration provides little-to-no benefit for small gradient-boosted tree models, whose inference is primarily memory-latency-bound and rarely utilizes GPU parallelism effectively at this scale. This adds unnecessary cost per inference.
  • E is Incorrect: AWS Inferentia is optimized for deep learning models that can be compiled to the Neuron SDK. XGBoost models do not benefit from Inferentia's architecture and would require unnecessary conversion effort.

Key Takeaway: For small, non-neural-network models (e.g., XGBoost, LightGBM), the Graviton-based m6g/c7g families often deliver the lowest cost per inference while using SageMaker's built-in containers that support ARM natively.

Question 3 - Multi-Model Endpoint Instance Selection

A healthcare analytics platform has deployed 80 different specialized models (frailty prediction, readmission risk, medication interaction scoring) to Amazon SageMaker AI. Each model is individually small (under 50 MB), is invoked infrequently (a few requests per hour each), and shares the same container framework and dependencies. The team must consolidate these onto a multi-model endpoint to reduce hosting costs while keeping p95 latency under 200 milliseconds when models are loaded on demand.

Which instance configuration should the team select? (Select ONE)

  • A) 40 separate endpoints, each on ml.t3.small instances with one model per endpoint
  • B) One multi-model endpoint on a single ml.c5.xlarge instance with all 80 models stored in Amazon S3
  • C) One multi-model endpoint on a single ml.g4dn.xlarge GPU instance
  • D) One multi-model endpoint on ml.m5.large with auto-scaling configured for up to 4 instances
  • E) One multi-model endpoint on ml.m5.24xlarge with all 80 models pre-loaded into memory
Click to reveal Answer & Explanation

Correct Answer: D

Correct Options Analysis

  • D is Correct: A single multi-model endpoint on ml.m5.large with auto-scaling provides the right balance. It minimizes initial cost while allowing automatic scale-up when concurrent model loads push CPU utilization beyond threshold. For infrequently invoked small models, one ml.m5.large can handle dozens of models concurrently, and auto-scaling provides headroom for batch arrivals.

Distractor Analysis

  • A is Incorrect: Running 40 endpoints each on ml.t3.small incurs per-endpoint overhead costs (each endpoint bills a minimum number of instance hours) and fails to leverage multi-model endpoint savings. The cost would be 40x the base endpoint charge.
  • B is Incorrect: A single ml.c5.xlarge without auto-scaling might be sufficient initially, but as model invocations rise or cold-start penalties spike during concurrent loads, it lacks the headroom that auto-scaling provides. The oversight is the lack of scalability for reliability.
  • C is Incorrect: GPU acceleration is unnecessary for small, non-neural-network models (or even small neural networks at low TPS). Multi-model endpoints on GPU instances add cost without meaningful latency benefit for these model types.
  • E is Incorrect: ml.m5.24xlarge with all models pre-loaded eliminates cold-start latency but incurs massive cost for models that are only called a few times per hour. This fails to optimize for cost.

Key Takeaway: Multi-model endpoints (MMEs) consolidate many small, infrequently invoked models onto shared compute; choose a modest CPU instance with auto-scaling to balance cold-start latency against hosting costs.

Question 4 - High-Throughput Large Language Model Inference on Bedrock vs SageMaker

An e-commerce company is running a large language model (LLM) with 7 billion parameters for product recommendation generation, and usage has grown to a steady 500 requests per minute during business hours. The team currently uses Amazon Bedrock On-Demand and has seen monthly costs rise 3x in the past quarter. Leadership wants to reduce inference costs while maintaining p99 latency under 1 second. The traffic is predictable and consistently high during work hours, with minimal overnight traffic.

Which TWO strategies would accomplish the cost reduction goal with minimal architectural change? (Select TWO)

  • A) Migrate to Amazon SageMaker Real-Time inference with ml.g5.12xlarge instances and auto-scaling
  • B) Purchase Amazon Bedrock Provisioned Throughput for the LLM during business hours
  • C) Switch to a smaller foundation model that meets the quality bar
  • D) Move to Amazon Bedrock Batch Inference for all requests
  • E) Switch to EC2 Reserved Instances for the currently used instance family
Click to reveal Answer & Explanation

Correct Answer: B and C

Correct Options Analysis

  • B is Correct: For predictable, high-volume, real-time inference on Amazon Bedrock, Provisioned Throughput reserves dedicated model units at a fixed hourly rate. This can significantly reduce costs (up to 50%+ compared to On-Demand token pricing) when the volume is steady and predictable.
  • C is Correct: The "right-sizing" concept transfers to model selection. If a smaller foundation model (e.g., 3B parameters) can meet the quality requirements for product recommendations, moving to it reduces per-token cost and latency across all requests.

Distractor Analysis

  • A is Incorrect: Migrating to SageMaker Real-Time would introduce substantial architectural change. The scenario asks for minimal change, and migrating from managed Bedrock to self-managed SageMaker inference adds operational overhead and engineering effort that fails the "minimal architectural change" constraint.
  • D is Incorrect: Batch inference does not support real-time request/response patterns. Product recommendations are needed immediately as users browse, so asynchronously processing requests in batch is architecturally incompatible with the latency requirement.
  • E is Incorrect: EC2 Reserved Instances cover EC2 resource usage but do not cover Amazon Bedrock Provisioned Throughput or On-Demand token charges. Bedrock does not run on EC2 instances that you can reserve.

Key Takeaway: Foundation model cost optimization uses model selection (right-sizing), Provisioned Throughput (reserved capacity), and prompt discipline (token control)—not EC2-style instance reservations.

Question 5 - Choosing Between Real-Time and Serverless Inference

A startup's data science team has built a recommendation model on Amazon SageMaker AI that serves internal dashboards with sporadic traffic — anywhere from 0 to 5 requests per hour, with occasional weeks of no traffic. The model runs once per month for a scheduled batch job as well. The team wants the lowest operational overhead and is willing to tolerate some cold-start latency (up to 2 seconds) for the rare real-time queries. They are not concerned about GPU acceleration.

Which SageMaker inference option should the team select for the real-time queries? (Select ONE)

  • A) Real-time endpoint on ml.m5.large with auto-scaling configured from 0 to 1 instances
  • B) Serverless inference with a modest MaxConcurrency setting (e.g., 5 concurrent invocations)
  • C) Asynchronous inference on ml.c5.xlarge
  • D) Real-time endpoint on ml.g4dn.xlarge with 1 instance
  • E) Multi-model endpoint with 1 instance on ml.t3.medium
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: SageMaker Serverless Inference automatically provisions compute capacity only when requests arrive, with zero capacity and zero cost when not in use. It has a small cold-start penalty (typically under a few seconds when the model is idle), which is acceptable given the team's tolerance. This minimizes both operational overhead and cost for sporadic traffic.

Distractor Analysis

  • A is Incorrect: While auto-scaling from 0 to 1 instances is possible with Application Auto Scaling on SageMaker (using CloudWatch alarm-based scaling), it incurs latency when triggering scale-out and introduces added configuration complexity compared to serverless inference. Serverless is a better fit for this scenario.
  • C is Incorrect: Asynchronous inference is designed for large payloads (up to 1 GB) or long processing times (up to 15 minutes). Sporadic real-time queries from an internal dashboard with small payloads don't require this complexity.
  • D is Incorrect: With only 0-5 requests per hour, a GPU instance represents extreme over-provisioning and unnecessary cost. GPU acceleration is not needed.
  • E is Incorrect: A multi-model endpoint with 1 instance on t3.medium still runs 24/7 (charging continuously) and is designed for consolidating many models onto one instance, not for handling sporadic low-traffic workloads.

Key Takeaway: SageMaker Serverless Inference is ideal for intermittent or low-traffic workloads because it scales to zero capacity when idle, eliminating always-on endpoint charges.

Question 6 - Inferentia vs GPU for Deep Learning Vision Model

A manufacturing company runs a computer vision model for defect detection on Amazon SageMaker AI. The model is a convolutional neural network (CNN) with 50 million parameters, with a throughput requirement of 1,000 images per second across the production line. The team wants to reduce inference cost by at least 40% compared to their current ml.g4dn.xlarge GPU instances, and they are willing to invest engineering effort into model compilation. The model is built in PyTorch and uses standard CNN operators.

Which option best meets the cost-reduction demand while maintaining the 1,000 images/second throughput? (Select ONE)

  • A) Use ml.inf1.6xlarge instances with AWS Neuron SDK compilation
  • B) Use ml.c5.9xlarge CPU instances with Intel MKL optimization
  • C) Use ml.p3.2xlarge GPU instances (older NVIDIA V100 generation)
  • D) Use ml.g5.xlarge GPU instances (newer NVIDIA A10G generation)
  • E) Continue using ml.g4dn.xlarge but purchase 1-year SageMaker Savings Plans
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: AWS Inferentia (ml.inf1.6xlarge) is custom silicon designed specifically for deep learning inference. For CNN and transformer models that can be compiled with the AWS Neuron SDK, Inferentia delivers up to 70% lower cost per inference compared to comparable GPU instances. The team's willingness to invest in compilation makes this the right choice.

Distractor Analysis

  • B is Incorrect: A 50M parameter CNN processing 1,000 images per second would require extensive CPU instances to match GPU-level inference throughput. CPU inference for vision models at this scale is significantly less cost-effective than Inferentia or even GPU.
  • C is Incorrect: ml.p3.2xlarge with V100 GPUs is an older generation, typically slower than newer T4 (g4dn) for inference workloads. It would not reduce cost.
  • D is Incorrect: ml.g5.xlarge with A10G GPUs provides better performance than g4dn, but at a higher cost. It does not meet the explicit 40% cost-reduction requirement compared to g4dn.
  • E is Incorrect: Savings Plans would provide up to 64% discount over On-Demand rates for g4dn, but the question asks for cost reduction relative to the current g4dn. If the team is already on On-Demand pricing, this is a valid alternative; however, Inferentia provides additional cost reduction on top of any Savings Plan discount, and the scenario context implies they want architectural cost reduction.

Key Takeaway: AWS Inferentia (ml.inf1/ml.inf2) is purpose-built for high-throughput deep learning inference, offering the lowest cost per inference when models are compiled with the Neuron SDK.

Question 7 - Selecting Instance Size Based on Measured Utilization

A financial analytics firm runs a real-time fraud detection model on Amazon SageMaker AI with ml.c5.4xlarge instances. CloudWatch metrics over the past 30 days show:

  • Average CPU Utilization: 18%
  • Maximum CPU Utilization: 55% (during brief market open spikes)
  • Average Memory Utilization: 30 GiB out of 32 GiB (93.75%)
  • Throughput: Stable at 300 requests/second during peak

The team wants to reduce hosting costs while maintaining sufficient headroom for the market-open spikes. Which change should they make? (Select ONE)

  • A) Downscale to ml.c5.xlarge (4 vCPU, 8 GiB)
  • B) Downscale to ml.c5.2xlarge (8 vCPU, 16 GiB) with auto-scaling to 2 instances during spikes
  • C) Downscale to ml.r5.2xlarge (8 vCPU, 64 GiB)
  • D) Keep ml.c5.4xlarge but add auto-scaling to scale down overnight
  • E) Move to ml.m5.4xlarge (same size general-purpose)
Click to reveal Answer & Explanation

Correct Answer: C

Correct Options Analysis

  • C is Correct: The key constraint is memory utilization at 30 GiB out of 32 GiB (93.75%). CPU is under-utilized at 18% average, but memory is nearly exhausted. The correct move is to select an instance family with adequate memory headroom while reducing CPU over-provisioning. ml.r5.2xlarge (8 vCPU, 64 GiB) provides substantial memory while reducing CPU cost — appropriate since the workload is memory-bound, not CPU-bound.

Distractor Analysis

  • A is Incorrect: ml.c5.xlarge with only 8 GiB of memory would not accommodate the model's 30 GiB memory requirement, causing out-of-memory errors.
  • B is Incorrect: ml.c5.2xlarge with 16 GiB is still insufficient to hold the 30 GiB model, even with auto-scaling. The model simply won't fit on this instance.
  • D is Incorrect: The issue is not a diurnal (day/night) traffic pattern but rather an over-provisioned CPU on the current instance with insufficient memory headroom. Auto-scaling does not fix the instance type mismatch.
  • E is Incorrect: ml.m5.4xlarge has the same 32 GiB memory with balanced CPU (16 vCPU vs c5's 8 vCPU), but the workload is memory-bound, not CPU-bound. r5 provides more memory per dollar.

Key Takeaway: Always check multiple CloudWatch metrics before downscaling. High memory utilization with low CPU means downsizing by instance family (C to R for memory), not just by size.

Question 8 - Multi-Instance Real-Time Endpoint with Traffic Spikes

A video streaming platform uses a recommendation model on Amazon SageMaker AI Real-Time Inference with ml.g4dn.xlarge instances. During new content releases, traffic spikes from 100 to 2,000 requests per second within minutes, then drops back. The team wants to auto-scale instances from 1 to 10 based on traffic, with lowest latency impact during scale-out events. GPU instance startup typically takes 3-4 minutes.

Which configuration best meets these requirements? (Select ONE)

  • A) Use Application Auto Scaling with Step Scaling policy triggered when InvocationsPerInstance exceeds 100, with scale-out of +2 instances and scale-in of -1 instance
  • B) Use Application Auto Scaling with Target Tracking policy targeting InvocationsPerInstance = 500, with scale-out of +1 instance and scale-in of -1 instance
  • C) Pre-warm instances during peak hours by setting minimum capacity to 10 instances 24/7
  • D) Use Application Auto Scaling with Step Scaling triggered on CPU Utilization > 80%, with scale-out of +5 instances and scale-in of -1 instance
  • E) Use a single ml.p4d.24xlarge instance to handle the maximum 2,000 TPS demand
Click to reveal Answer & Explanation

Correct Answer: D

Correct Options Analysis

  • D is Correct: For sudden traffic spikes (content releases), step scaling with aggressive scale-out (+5 instances) and conservative scale-in (-1 instance) helps quickly absorb burst demand on GPU instances. CPU utilization above 80% is a clear signal on GPU instances that the model inference is becoming compute-bound.

Distractor Analysis

  • A is Incorrect: Scaling out only +2 instances per step may be too slow to absorb a 20x traffic spike (from 100 to 2,000 TPS), leaving the endpoint throttled during the critical launch window.
  • B is Incorrect: Target tracking with InvocationsPerInstance = 500 is misconfigured. A single g4dn.xlarge for a recommendation model typically cannot handle 500 invocations per second, and the conservative +1/-1 scale-out pattern would not keep up with the demand spike.
  • C is Incorrect: Keeping 10 instances running 24/7 eliminates scaling concerns but incurs massive costs. Auto-scaling exists specifically to avoid this scenario.
  • E is Incorrect: A single ml.p4d.24xlarge (8x A100 GPUs) is massive overkill, extremely expensive, and provides no better latency than scaling multiple smaller GPU instances. It also creates a single point of failure.

Key Takeaway: For GPU-hosted endpoints with bursty traffic, aggressive step scaling (or faster target-tracking with a higher threshold) combined with conservative scale-in is the standard pattern to minimize both latency and cost.

Question 9 - Elastic Inference vs Purpose-Built Acceleration

A startup data science team built a natural language processing (NLP) model for sentiment analysis on Amazon SageMaker AI. The model is deployed on ml.m5.large CPU instances with SageMaker Elastic Inference (EI) attached for acceleration. The team is happy with the performance but notice that EI-attached instances now have a service end-of-life notification and are no longer supported for new model deployments. They need to migrate to a supported configuration that preserves cost-efficiency with minimal engineering change.

Which option should the team select to replace Elastic Inference? (Select ONE)

  • A) Move to ml.g4dn.xlarge GPU instances with the same model architecture
  • B) Move to ml.inf1.xlarge with AWS Neuron compilation
  • C) Move to ml.c5.xlarge CPU-only instances (removing acceleration)
  • D) Continue using EI on existing instances (it still works)
  • E) Move to SageMaker Serverless Inference
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: The most straightforward migration path from EI-attached instances is to dedicated GPU instances in the g4dn family. The model architecture remains unchanged (no recompilation needed since g4dn uses standard NVIDIA CUDA/cuDNN libraries), and the T4 GPU handles small NLP models well. This minimizes engineering effort while providing supported acceleration.

Distractor Analysis

  • B is Incorrect: ml.inf1 requires compiling the model to AWS Neuron SDK, a significant engineering effort. The scenario asks for minimal engineering change.
  • C is Incorrect: Removing GPU acceleration entirely will degrade inference latency significantly for an NLP transformer model, especially if the original architecture required EI to meet latency targets.
  • D is Incorrect: Amazon announced EI end-of-life for new generations of instances. Relying on unsupported infrastructure violates AWS best practices and leaves the team vulnerable to future service shutdowns.
  • E is Incorrect: Serverless Inference auto-scales to zero and works well for intermittent traffic, but it may not provide the same latency guarantees for an always-on NLP model and introduces cold-start considerations without addressing the acceleration requirement directly.

Key Takeaway: When migrating from deprecated acceleration technologies, choose the simplest supported equivalent (in this case, dedicated GPU instances) to minimize re-engineering.

Question 10 - Selecting Instance Family for Batch Transform with Large Dataset

An insurance company needs to run batch inference on 50 million records using a moderately complex neural network model (roughly 200 million parameters) on Amazon SageMaker Batch Transform. The job must complete within 4 hours and the team wants to minimize total job cost. The model was trained in PyTorch and has been packaged into a SageMaker-compatible container.

Which instance configuration should the team select? (Select ONE)

  • A) Use ml.g4dn.12xlarge GPU instances (4 GPUs each) with MaxPayloadInMB set to 10 and MaxConcurrentTransforms set to 8
  • B) Use ml.c5.9xlarge CPU instances with MaxPayloadInMB set to 10 and MaxConcurrentTransforms set to 8
  • C) Use ml.p4d.24xlarge GPU instances (8x A100 GPUs) with MaxConcurrentTransforms set to 1
  • D) Use ml.m5.24xlarge CPU instances with MaxPayloadInMB set to 1 and MaxConcurrentTransforms set to 4
  • E) Use ml.g5.2xlarge GPU instances with MaxPayloadInMB set to 10 and MaxConcurrentTransforms set to 4
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: For batch transform on a 200M parameter neural network processing 50M records in 4 hours, GPU instances offer significantly faster throughput than CPU for matrix-heavy operations. ml.g4dn.12xlarge (4x T4 GPUs) provides balanced cost-performance. Setting appropriate MaxConcurrentTransforms and MaxPayloadInMB values allows parallel processing of multiple records, improving throughput.

Distractor Analysis

  • B is Incorrect: While CPU instances are cheaper per hour, they would process records significantly slower for a 200M parameter neural network. To meet the 4-hour deadline, the team would need many more CPU instances, potentially increasing total cost beyond the GPU option.
  • C is Incorrect: ml.p4d.24xlarge with 8x A100 GPUs is overkill for a 200M parameter model, costing significantly more per hour and exceeding the necessary compute requirements.
  • D is Incorrect: ml.m5.24xlarge is memory-optimized, not compute-optimized, and sits at the high end of the family. It is also CPU-based which is slower for neural network inference. Setting MaxPayloadInMB to 1 also causes excessive overhead and slower processing.
  • E is Incorrect: While ml.g5.2xlarge with A10G GPUs is a reasonable option, the A10G GPU typically costs more than T4 while providing similar or only marginally better throughput for inference of this model size. The g4dn option provides better cost efficiency for this batch workload.

Key Takeaway: For batch transform workloads, choose GPU instances for neural network models and configure parallel processing parameters (MaxConcurrentTransforms, MaxPayloadInMB) to optimize both throughput and cost; the g4dn family is usually the cost-efficient choice for medium-sized models.