Skip to content

Skill 2.2.6: Combine multiple ML models to improve performance or reduce cost

Question 1 - Multi-Model Endpoint Cost Optimization

A financial services company has deployed 200 fraud detection models using Amazon SageMaker, each trained on transaction data for a different regional merchant. These models are lightweight gradient-boosted trees that are invoked infrequently, with only a few prediction requests per minute.

The current architecture runs each model on its own SageMaker endpoint, resulting in monthly infrastructure costs exceeding $50,000. The ML engineering team must reduce endpoint hosting costs while maintaining the ability to invoke each model independently with sub-second latency for real-time fraud screening.

Which solution should the team implement to be MOST cost-effective while meeting the latency requirements? (Select ONE)

  • A) Use a multi-model endpoint with a shared serving container, storing all 200 model artifacts in a single S3 prefix and invoking them by specifying TargetModel in the inference request.
  • B) Use SageMaker batch transform to process fraud prediction requests in scheduled hourly batches, eliminating the need for real-time endpoints.
  • C) Consolidate the models into a single ensemble model using SageMaker Autopilot, which automatically blends all 200 models into one endpoint.
  • D) Migrate the models to an Amazon OpenSearch k-NN index, which can store model parameters and perform real-time inference natively.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: Amazon SageMaker Multi-Model Endpoints (MME) allow many models to be served behind one endpoint using a shared serving container. All 200 model artifacts are stored in an Amazon S3 prefix (e.g., s3://bucket/models/), and SageMaker dynamically loads the requested model into memory when invoked via the TargetModel parameter. This dramatically reduces cost because the endpoint's underlying compute (instances) is shared across all models, rather than paying for 200 separate endpoints. Sub-second latency is preserved through SageMaker's intelligent model caching strategy where frequently accessed models remain loaded in memory.

Distractor Analysis

  • B is Incorrect: SageMaker Batch Transform processes large batches of data asynchronously and is intended for offline/periodic scoring, not real-time fraud screening. Shifting to hourly batches introduces unacceptable latency for fraud detection, where predictions must be made at the transaction moment.
  • C is Incorrect: SageMaker Autopilot is an AutoML service that builds classification/regression models from a dataset; it does not "merge" already-trained models into a single ensemble artifact. Consolidating 200 models into one would also discard the regional independence each merchant-specific model requires.
  • D is Incorrect: Amazon OpenSearch k-NN indexes store vector embeddings for approximate nearest neighbor search, not trained ML model parameters. It cannot execute arbitrary model inference for fraud detection tasks.

Key Takeaway: When many similar models have low invocations per minute, SageMaker Multi-Model Endpoints minimize costs by sharing a single endpoint and dynamically loading models from S3, while preserving real-time latency.

Question 2 - Model Caching for Multi-Model Endpoint

A retail intelligence platform has implemented a SageMaker multi-model endpoint hosting 50 product recommendation models, one per product category, each stored as a .tar.gz artifact in an S3 bucket. The endpoint is configured with a single ml.m5.xlarge instance.

During a flash sale event, the team observes high inference latency (P99 exceeding 5 seconds) for certain product categories, while other categories remain fast. Engineers determine that some models are being repeatedly loaded and unloaded from the instance's memory due to cache thrashing.

Which TWO actions would MOST effectively reduce latency for the frequently accessed models without increasing infrastructure cost? (Select TWO)

  • A) Delete unused models from the S3 prefix so that only the high-traffic models are available through the endpoint.
  • B) Configure a custom ResponseCache in the SageMaker endpoint's serving stack to store frequent inference responses in memory.
  • C) Add more instances to the multi-model endpoint using an inference recommender job to determine the optimal instance count.
  • D) Separate the models into two multi-model endpoints on different instance types, giving the high-traffic models dedicated compute resources.
  • E) Enable SageMaker MME time-based model caching to prioritize recently accessed models by setting the ModelCacheSetting to TimeBasedCache.
Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: The cache thrashing occurs because the single instance's memory cannot hold all 50 models simultaneously. By removing low-traffic models from the S3 prefix (or moving them to a separate endpoint), the remaining high-traffic models stay resident in memory and do not repeatedly compete for cache space, lowering latency without requiring additional infrastructure.
  • C is Correct: Scaling the multi-model endpoint horizontally by adding more instances increases the total model cache capacity across the fleet. SageMaker load-balances inference requests across instances using routing mechanisms, and with more memory available, more models can remain loaded per instance, decreasing cache misses and unloading frequency.

Distractor Analysis

  • B is Incorrect: ResponseCache is not a native SageMaker endpoint feature. SageMaker endpoints do not automatically cache inference responses at the serving layer; implementing response caching would require custom Lambda or API Gateway layers, which introduces additional complexity and does not address the root cause (model artifact loading/unloading).
  • D is Incorrect: While separating models into different endpoints on different instance types can solve memory constraints, hosting two separate endpoints increases cost—the opposite of the question's requirement. Multi-model endpoints exist to consolidate endpoints, not proliferate them.
  • E is Incorrect: SageMaker MME does not have a ModelCacheSetting or TimeBasedCache parameter. While MME does implement an LRU (Least Recently Used) cache eviction policy internally, it cannot be configured via a time-based cache setting.

Key Takeaway: In a multi-model endpoint, latency spikes for some models usually mean cache thrashing—fix it by reducing the model count per prefix or scaling instances horizontally to expand pooled memory, not by adding caching layers.

Question 3 - Ensemble Strategy for Predictive Maintenance

An automotive manufacturer is building a predictive maintenance system to forecast when industrial robots on the assembly line will fail. The data science team has trained three separate models:

  • Model 1: A gradient-boosted tree (XGBoost) trained on time-series sensor data, achieving high precision but low recall.
  • Model 2: A random forest trained on historical maintenance logs, achieving high recall but lower precision.
  • Model 3: A logistic regression model trained on machine age and usage metrics, serving as a simple baseline.

The team wants to combine these models to maximize overall F1-score (balanced precision and recall) for a weekly batch prediction job that runs over millions of records. They must use a native Amazon SageMaker approach with minimal custom coding.

Which ML technique should the team implement? (Select ONE)

  • A) Use SageMaker Autopilot to automatically search for and select the best single model from among the three, discarding the others.
  • B) Use a stacking ensemble implemented with SageMaker built-in XGBoost algorithm, where the predictions from Models 1, 2, and 3 are used as input features to a meta-model that learns the optimal combination.
  • C) Use a SageMaker Inference Pipeline that sequentially passes the data through Models 1, 2, and 3, averaging their prediction scores at the end using the built-in BlazingText algorithm.
  • D) Use a simple majority vote ensemble implemented by running all three models in parallel via SageMaker Batch Transform and combining their results with a Lambda function after each batch job completes.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: Stacking (also called stacked generalization) is an ensemble technique where the predictions of base models serve as input features to a higher-level "meta-model" that learns how to optimally weight or combine them. The SageMaker built-in XGBoost algorithm supports training a stacking meta-model on tabular data (the out-of-fold predictions from the three base models), and the entire workflow can be orchestrated natively within SageMaker without the need for custom container logic.

Distractor Analysis

  • A is Incorrect: SageMaker Autopilot selects a single best model architecture from its own predefined candidate algorithms, not from pre-existing models built by the team. It cannot consume already-trained models as candidates. Additionally, discarding models sacrifices the diversity of the ensemble, which is the primary reason for combining models in the first place.
  • C is Incorrect: SageMaker Inference Pipeline chains models sequentially (e.g., feature engineering to inference to post-processing), where one model's output feeds the next. It does not average predictions from parallel models. Also, BlazingText is a text processing/embedding algorithm, not a general-purpose prediction combiner.
  • D is Incorrect: While majority voting is a valid ensemble technique, implementing it with a custom Lambda function after every batch transform run introduces unnecessary operational complexity, especially at millions of records per week. SageMaker-native approaches (like stacking via XGBoost) keep the ensemble contained inside the managed training job and are more scalable.

Key Takeaway: Stacking ensembles turn individual model predictions into meta-features, letting a second-level model learn the best way to combine them—SageMaker's built-in XGBoost is ideal for this because it handles tabular inputs natively.

Question 4 - Distillation to Reduce Inference Cost

A large e-commerce company operates a customer support chatbot powered by a Llama 3 70B foundation model on Amazon Bedrock via on-demand inference. The chatbot handles simple FAQ inquiries (order tracking, return policies, payment issues), answering questions from a curated internal knowledge base of 2,000 articles.

The company's monthly inference cost has ballooned to over $120,000 due to the 70B model's high token pricing. The chatbot's response quality is being benchmarked against human agents and user satisfaction surveys. The ML team needs to drastically reduce inference cost while maintaining at least 95% of the current response quality. They have a static, comprehensive prompt set representing all typical customer questions.

Which solution is MOST cost-effective and aligns with AWS best practices? (Select ONE)

  • A) Use Amazon Bedrock Model Distillation to transfer knowledge from the 70B teacher model into a smaller student model (e.g., Llama 3 8B), then serve the student model for chatbot inference.
  • B) Switch from on-demand to provisioned throughput for the 70B model, which guarantees lower per-token costs for sustained workloads despite the high baseline.
  • C) Use Amazon Bedrock Knowledge Bases with a custom embedding model to replace the large foundation model entirely, avoiding any generation-based cost.
  • D) Continue using the 70B model but set the temperature parameter to 0.0 and constrain max_tokens to 100, which limits both generation unpredictability and token spend per response.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: Amazon Bedrock Model Distillation automates knowledge transfer from a large teacher model to a smaller, cheaper student model. The teacher generates responses to a curated prompt set (which the team already has), and the student model is fine-tuned on those responses. For simple FAQ-style queries from a bounded knowledge domain, a well-distilled 8B model can achieve near-parity with a 70B model at a fraction of the inference cost—typically 80-90% cheaper per token.

Distractor Analysis

  • B is Incorrect: Provisioned throughput offers predictable performance and lower effective per-token pricing at high sustained volume, but it does not address the fundamental cost problem—you are still running a 70B model with significantly more expensive compute than a smaller model. Provisioned throughput for a 70B model would also add substantial fixed charges regardless of actual traffic.
  • C is Incorrect: Amazon Bedrock Knowledge Bases handles retrieval (finding relevant articles from the internal wiki) but still requires a foundation model to generate the final customer-facing answer from the retrieved passages. It cannot completely replace the generator; it augments it. Additionally, RAG alone does not reduce per-token inference cost of the underlying LLM.
  • D is Incorrect: Lowering temperature can make outputs more deterministic for fact-based tasks, and limiting max_tokens does reduce per-request token spend. However, these are tuning tweaks that shave costs marginally; they do not change the underlying model size, which is the primary cost driver. The savings would be far smaller than those achieved through distillation to an 8B model.

Key Takeaway: Distillation is the AWS-native answer to "make the model smaller and cheaper while preserving quality" when the task is narrow—provide Bedrock with a prompt set, and it transfers knowledge from teacher to student automatically.

Question 5 - Model Parallel Distributed Training

A genomics research institute is training a protein structure prediction model using Amazon SageMaker with a custom PyTorch script. The model contains a multi-billion parameter transformer architecture, and it cannot fit into the memory of even the largest single GPU instance (ml.p4de.24xlarge with 8× 40GB A100 GPUs is insufficient).

The institute needs to train this model using model parallelism so that the full architecture can be processed across multiple GPUs. The team must implement a native SageMaker distributed training solution that requires minimal code changes to their existing PyTorch training loop.

Which option should the team use? (Select ONE)

  • A) Use SageMaker Distributed Data Parallel (SMDDP) to split the model's parameters across instances, with each GPU holding a subset of the layers.
  • B) Use SageMaker Model Parallel library with tensor parallelism and pipeline parallelism, wrapping the PyTorch model with the library's decorators and context managers.
  • C) Use SageMaker Training Compiler to automatically optimize and compress the model graph so it fits into a single GPU's memory during training.
  • D) Switch from PyTorch to SageMaker's built-in Object2Vec algorithm, which natively supports extremely large models with built-in parallelism.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: SageMaker's model parallel library (SMDMP/SMP) specifically supports model parallelism, where the model itself is partitioned across multiple GPUs. It supports both tensor parallelism (splitting individual layers across GPUs) and pipeline parallelism (splitting the sequence of layers across GPUs). The library is deeply integrated with PyTorch via minimal decorators (e.g., @smp.step, smp.init()), requiring only modest changes to an existing training loop.

Distractor Analysis

  • A is Incorrect: SageMaker Distributed Data Parallel (SMDDP) implements data parallelism, not model parallelism. In data parallelism, each GPU holds a full replica of the model and processes different mini-batches of data. This does not solve the problem because the model architecture cannot fit on any single device at all.
  • C is Incorrect: SageMaker Training Compiler optimizes the compute graph to speed up training (e.g., via operator fusion and memory reuse), but it cannot magically compress a multi-billion-parameter model into memory if the parameter count fundamentally exceeds available GPU memory. It is a performance optimizer, not a model-partitioning mechanism.
  • D is Incorrect: Object2Vec is a SageMaker built-in algorithm designed for learning vector representations from pairs of objects (e.g., product pairs or text pairs), not a general-purpose transformer architecture for protein structure prediction. It also does not support the custom model architecture the team has already developed.

Key Takeaway: When the model cannot fit on one GPU, use SageMaker Model Parallel (not Data Parallel) — it partitions layers across GPUs via pipeline/tensor parallelism with minimal PyTorch code changes.

Question 6 - Ensemble Diversity and Bias-Variance Tradeoff

A credit-scoring startup is developing a default-risk prediction model using Amazon SageMaker. The team has observed that their current single XGBoost model exhibits high variance: performance on the validation set varies significantly depending on the random seed used for data splitting and initialization. The model's AUC scores range from 0.71 to 0.78 across 10 different seeds.

The startup wants to reduce prediction variance and improve overall model stability without significantly increasing training time or inference complexity. They need a solution that runs natively in SageMaker.

Which technique should the team implement? (Select ONE)

  • A) Train 10 separate XGBoost models using different random seeds and random subsets of the training data, then average their prediction probabilities at inference time using a SageMaker Inference Pipeline.
  • B) Use SageMaker Automatic Model Tuning to find a single set of hyperparameters that minimizes variance across seeds by setting the objective metric to the standard deviation of AUC.
  • C) Use bagging (bootstrap aggregating) by training an ensemble of models on bootstrap samples of the training data and averaging their predictions, leveraging the SageMaker XGBoost built-in num_parallel_tree hyperparameter.
  • D) Increase the learning_rate and max_depth hyperparameters to force the model to capture more signal from the training data, reducing variance caused by under-specification.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Options Analysis

  • C is Correct: Bagging reduces variance by training multiple models on different bootstrap samples (random samples with replacement) of the training data and then aggregating their predictions (averaging for regression, majority vote for classification). XGBoost's built-in num_parallel_tree hyperparameter creates a forest of trees where each tree is trained on a different random subsample, effectively implementing bagging natively within the XGBoost framework without requiring separate training jobs or complex inference logic. This improves stability and reduces variance with minimal operational overhead.

Distractor Analysis

  • A is Incorrect: While averaging predictions from 10 models trained on different splits/seeds is a valid variance-reduction ensemble, orchestrating 10 separate training jobs and building a custom SageMaker Inference Pipeline to average probabilities is operationally heavy. Additionally, an Inference Pipeline chains models sequentially (each model's output feeds the next), it is not designed for parallel ensemble averaging. Custom inference logic would be required, which increases complexity.
  • B is Incorrect: SageMaker Automatic Model Tuning (AMT) optimizes hyperparameters against a single objective metric from a single training job. There is no built-in capability to treat "variance across multiple seeds" as the optimization objective. AMT does not inherently run multiple seeded replicates to estimate variance—it searches hyperparameter space using one metric per job.
  • D is Incorrect: Increasing learning_rate and max_depth typically increases model complexity, leading to further overfitting and higher variance. This is the opposite of what is needed when the model already suffers from high variance across seed variations. Larger learning rates also make training more unstable.

Key Takeaway: High variance means the model is too sensitive to idiosyncrasies in the training data—bagging (via XGBoost's num_parallel_tree) reduces variance by averaging diverse models trained on different data subsets, without the overhead of managing many training jobs.

Question 7 - Combining Models vs. Single Large Model

A healthcare AI company is building a medical diagnosis support system that analyzes chest X-rays. The team has two candidate approaches:

  • Approach 1: A single deep convolutional network (e.g., ResNet-152) trained end-to-end on 500,000 labeled chest X-rays.
  • Approach 2: An ensemble of three independently trained models—a ResNet-50, a DenseNet-121, and an EfficientNet-B4—trained on the same dataset, with prediction scores averaged at inference.

The team evaluates both approaches on a held-out test set. Approach 1 achieves an accuracy of 0.89, while Approach 2 achieves an accuracy of 0.91. However, Approach 2 requires 3× the inference compute and introduces higher latency.

Given the requirement to improve accuracy while keeping inference latency within clinical SLA (<2 seconds per image) and operational overhead low, which statement about model selection is correct? (Select ONE)

  • A) Approach 2 is always superior because ensemble methods reduce variance and bias simultaneously, guaranteeing better generalization in any scenario.
  • B) Approach 1 should be selected because a single deeper model with sufficient data can capture the same features as an ensemble, and its lower inference latency is safer for clinical deployment.
  • C) Approach 2 should be selected because the 2-percentage-point accuracy improvement in a medical setting outweighs latency concerns, even if it requires adding more GPU instances to maintain SLA.
  • D) The team should quantize the ensemble models (Approach 2) using SageMaker Neo to reduce inference latency, making the accuracy gain acceptable under SLA without increasing infrastructure cost.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: While ensembles typically outperform single models, the improvement is not guaranteed and comes at significant operational cost. In this scenario, the single model achieves 0.89 accuracy—only 2 percentage points below the ensemble—but requires ⅓ the inference compute and meets clinical latency SLA comfortably. In regulated, latency-sensitive domains like healthcare, an architecture that is simpler to maintain, monitor, and deploy is often the pragmatic and still highly accurate choice. The single deeper model captures sufficient representational power from 500K samples.

Distractor Analysis

  • A is Incorrect: This statement is false on several levels. Ensembles primarily reduce variance (not bias simultaneously—bias reduction often requires different model architectures or feature engineering), and their performance advantage is not "guaranteed" in any scenario. If base models are highly correlated in their errors, ensemble gains diminish. Furthermore, with sufficient data and regularization, a single well-tuned large model can match or exceed an ensemble.
  • C is Incorrect: While medical accuracy is critical, the statement ignores the operational and cost constraints stated in the problem. The SLA is explicitly mentioned (<2 seconds per image). Without quantizing or optimizing deployment, increasing GPU instances to support the heavier ensemble both increases latency risk and raises cost—directly violating stated business constraints.
  • D is Incorrect: SageMaker Neo compiles and optimizes models for inference on specific hardware targets (e.g., Intel, NVIDIA, ARM). While Neo can accelerate inference, quantization affects numerical precision and may degrade diagnostic model accuracy if not carefully validated. It does not eliminate the operational complexity of deploying and managing three separate models, and the accuracy degradation risk in a medical context requires rigorous re-validation.

Key Takeaway: Ensemble selection is not automatic—justify the added complexity, cost, and latency against the measured performance gain. When accuracy differences are marginal and SLA constraints are tight, a simpler single model is often the better choice.

Question 8 - Catastrophic Forgetting and Ensemble Regularization

A legal-tech company has fine-tuned a Claude 3 Haiku foundation model on Amazon Bedrock to specialize in contract clause extraction from real estate documents. After fine-tuning, the model excels at extracting lease terms, payment schedules, and renewal clauses (domain accuracy increased from 72% to 94%).

However, the legal team reports that the model now performs poorly on general legal queries it used to handle well—such as summarizing court opinions, identifying case citations, and explaining basic legal concepts. The model's general legal QA accuracy dropped from 88% to 61%.

The company needs to restore general legal proficiency while preserving the new domain specialization. Which TWO actions should the ML team take to address this issue? (Select TWO)

  • A) Include a mix of general legal instruction data and domain-specific fine-tuning data in the future fine-tuning dataset to prevent catastrophic forgetting.
  • B) Use Amazon Bedrock Model Distillation to transfer the specialized model's knowledge back into a pre-trained Claude 3 Haiku model, which should retain general capabilities.
  • C) Create a routing ensemble that directs general legal queries to the original pre-trained model and domain-specific contract extraction queries to the fine-tuned model.
  • D) Lower the temperature parameter during inference to make the model's responses more deterministic across both general and domain-specific queries.
  • E) Disable the fine-tuned model and revert to the original pre-trained model, relying on prompt engineering and in-context examples for domain-specific tasks.
Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: Catastrophic forgetting occurs when fine-tuning on a narrow dataset causes a model to overwrite previously learned general knowledge. The standard mitigation is data mixing—including a portion of general-domain samples (e.g., 10-30%) in the fine-tuning dataset alongside domain-specific examples. This helps the model retain broad capabilities while acquiring specialization.
  • C is Correct: A routing ensemble treats different models as specialized experts. A lightweight classifier (e.g., based on query keywords, embeddings, or a small DistilBERT model) can route incoming queries to the appropriate model: general queries go to the original foundation model, and contract extraction queries go to the fine-tuned model. This directly addresses the problem without requiring a retrained model, making it operationally feasible while alternative solutions are developed.

Distractor Analysis

  • B is Incorrect: Model distillation transfers knowledge from a teacher to a student model using generated response data. Distilling the fine-tuned (specialized) model into a fresh Claude 3 Haiku would transfer domain specialization, but it does not magically preserve general legal knowledge that the specialized model has already partially forgotten. The student model would simply learn the teacher's degraded general capabilities.
  • D is Incorrect: Temperature controls response randomness/determinism, not knowledge retention or model capability. Lowering temperature makes outputs more consistent but cannot restore general legal knowledge that was overwritten during fine-tuning.
  • E is Incorrect: Reverting to the original pre-trained model and relying solely on prompt engineering (few-shot in-context learning) accepts a significant drop in domain performance (94% back down to ~72%) and forces the legal team to manually craft examples rather than leveraging the successful fine-tuned knowledge. It also fails to use the fine-tuned model where it excels.

Key Takeaway: Catastrophic forgetting remedies include mixing general data during fine-tuning, using regularization terms to constrain weight drift, or routing queries to different model versions via an ensemble—reverting entirely discards the domain gains.

Question 9 - Ensemble for Cold Start Recommendation

A streaming media platform has launched in a new region with minimal user interaction data. The recommendation system currently relies on a collaborative filtering matrix factorization model in Amazon SageMaker, which performs poorly for new users with no historical interactions (cold start problem).

The data science team has access to:

  • Basic user demographic information (age, subscription tier, device type)
  • Content metadata (genre, language, runtime, age-rating)
  • No historical rating or watch behavior for new users

To improve recommendations for cold-start users, the team wants to combine an additional model with their existing collaborative filtering approach. The solution must run within the SageMaker ecosystem with minimal custom code.

Which approach should the team implement? (Select ONE)

  • A) Train a content-based recommendation model using the SageMaker built-in Factorization Machines algorithm, then average its scores with the collaborative filtering model for cold-start users.
  • B) Use SageMaker built-in K-Means to cluster users into segments based on demographic features, assign a content preference profile to each cluster, and serve those profiles as fallback recommendations when collaborative filtering predictions lack confidence.
  • C) Extend the collaborative filtering model by adding demographic features as if they were user-item interactions with a constant rating, effectively transforming it into a hybrid model without needing a second algorithm.
  • D) Use Amazon Personalize with the USER_PERSONALIZATION recipe, which automatically incorporates item metadata and user attributes to handle cold-start scenarios without building a custom hybrid ensemble.
Click to reveal Answer & Explanation

Correct Answer: D

Correct Options Analysis

  • D is Correct: Amazon Personalize's USER_PERSONALIZATION recipe natively combines multiple models, including collaborative filtering, content-based filtering, and popularity-based heuristics, through a hierarchical multi-objective optimization. It automatically handles cold-start users by leveraging user/item metadata (demographics, content attributes) when interaction history is sparse. This eliminates custom ensemble engineering and operates as a fully managed AWS service, aligning with "minimal custom code."

Distractor Analysis

  • A is Incorrect: Factorization Machines (FM) is a SageMaker built-in algorithm, but it is a linear model suitable for feature-based regression/classification, not natively designed for content-based recommendation from metadata. More critically, simply averaging FM content scores with CF scores is a naive ensemble that doesn't account for when each model should dominate (e.g., cold start vs. warm start), potentially degrading warm-user performance as well.
  • B is Incorrect: K-Means clustering to create fallback profiles is a plausible cold-start method, but it requires significant manual implementation: feature engineering, choosing cluster count, designing profile assignment logic, and building a custom inference pipeline that switches between the clustering fallback and CF model. This violates the "minimal custom code" constraint. Amazon Personalize automates equivalent strategies internally.
  • C is Incorrect: Filling missing interaction values with constant ratings introduces systematic bias into the collaborative filtering model. It will treat fictional interactions as real signal, distorting learned user embeddings and degrading recommendations for both cold-start and warm users. Proper hybrid approaches keep model types separate and combine their outputs logically.

Key Takeaway: For cold-start problems in SageMaker-native environments, Amazon Personalize's USER_PERSONALIZATION recipe is superior to assembling ad-hoc hybrid ensembles because it automatically blends collaborative filtering with metadata-based content filtering through a managed service.

Question 10 - Multi-Arm Bandit Ensemble for Dynamic Model Selection

An online travel booking platform uses multiple recommendation models in production to rank hotel listings for users:

  • Model A: A price-sensitivity model that excels when users filter by budget.
  • Model B: A location-preference model that excels when users search by map view.
  • Model C: A family-friendly model that excels for bookings with children.

Currently, the platform serves recommendations with a static weighted average of all three models' scores (weights: 0.4, 0.3, 0.3). The team notices that user context (search filters, map usage, booking details) strongly indicates which model would perform best, but these contexts change hourly.

The team wants an ensemble strategy that dynamically adjusts model weights based on real-time user context and automatically optimizes weights over time using user click/booking feedback, all within the SageMaker ecosystem.

Which solution best meets these requirements? (Select ONE)

  • A) Use a multi-armed bandit implemented via Amazon Personalize with the CONTEXTUAL_BANDIT recipe, which dynamically selects among the three models based on user context and learns optimal selection policies from interaction feedback.
  • B) Set up an Amazon SageMaker Automatic Model Tuning job that runs weekly to re-tune the static weights (0.4, 0.3, 0.3) based on accumulated user feedback data.
  • C) Use Amazon SageMaker Inference Recommender to automatically determine the best model for each request based on endpoint latency and throughput metrics.
  • D) Create a stacking ensemble where a SageMaker XGBoost meta-model takes user context features as input and predicts which of the three models should receive the highest weight for that request.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: Amazon Personalize's CONTEXTUAL_BANDIT recipe is specifically designed for dynamic model selection with exploration/exploitation tradeoff. It treats each recommendation model as an "action" and user context (search filters, map usage, booking details) as the "context." The bandit algorithm learns which action (model) maximizes reward (clicks/bookings) for different contexts over time, automatically balancing exploration of untried models with exploitation of known winners. It updates continuously from interaction data without manual intervention.

Distractor Analysis

  • B is Incorrect: Weekly AMT jobs to re-tune static weights are a batch solution to a real-time problem. User context changes hourly, but weights only update weekly—missing fast-moving patterns. AMT also optimizes based on a single objective metric from training jobs, not from online inference feedback (clicks/bookings) in production. This is not a true online learning bandit solution.
  • C is Incorrect: SageMaker Inference Recommender helps select optimal instance types and configurations for machine learning models (e.g., choosing between ml.c5.xlarge and ml.m5.xlarge) based on latency/throughput benchmarks. It has nothing to do with choosing which model's predictions to serve at runtime. It is an infrastructure optimization tool, not a model-selection bandit.
  • D is Incorrect: A stacking meta-model is a valid dynamic weighting method, but it is a batch-supervised approach. It trains once on historical data and does not continuously update based on new user interactions. Without an online learning loop, it fails to adapt to evolving user behavior to the same extent that a contextual bandit can. Personalize's bandit recipe handles the full explore-exploit cycle natively.

Key Takeaway: When ensemble weights need to adapt to real-time context and improve from user feedback over time, Amazon Personalize's CONTEXTUAL_BANDIT recipe is the AWS-native answer, not static weight tuning or batch meta-learners.

This set of 10 questions covers the key subtopics of Skill 2.2.6: multi-model endpoints for cost optimization, ensemble methods (bagging, stacking, routing), catastrophic forgetting mitigation, model distillation, and dynamic model combination. Each question maps to realistic enterprise scenarios and AWS native services aligned with the MLA-C02 exam.