2.2.5

Question	Answer & Explanation
### Question 1 - Identifying and Mitigating Overfitting in SageMaker Built-in Algorithms

A machine learning team is training a predictive model using the Amazon SageMaker XGBoost built-in algorithm on a large tabular dataset. During the training job, they observe that the training accuracy reaches 98%, but the validation accuracy plateaus at 75%.

The team must mitigate this overfitting issue with the least operational overhead while staying within the bounds of the SageMaker XGBoost hyperparameters.

Which TWO actions should the team take to fulfill these requirements? (Select TWO)

- A) Increase the max_depth hyperparameter to allow the model to capture more complex patterns.

- B) Decrease the min_child_weight hyperparameter to make the model more conservative.

- C) Increase the alpha or lambda hyperparameters to apply L1 or L2 regularization.

- D) Increase the gamma hyperparameter to enforce a higher minimum loss reduction for tree splits.

- E) Rewrite the training logic in PyTorch using Script Mode to implement custom dropout layers. | >! Correct Answers: C and D !

Correct Options Analysis:

- C is Correct: In XGBoost, alpha corresponds to L1 regularization and lambda corresponds to L2 regularization on leaf weights. Increasing these values penalizes overly complex models, which directly mitigates overfitting using native hyperparameters.

- D is Correct: The gamma hyperparameter specifies the minimum loss reduction required to make a further partition on a leaf node. Increasing gamma prunes the tree and prevents it from memorizing noise, effectively reducing overfitting.

Distractor Analysis:

- A is Incorrect: Increasing max_depth makes the trees deeper and more complex, which exacerbates overfitting rather than mitigating it.

- B is Incorrect: min_child_weight controls the minimum sum of instance weight needed in a child. Decreasing it makes the algorithm more aggressive in splitting nodes, leading to higher complexity and overfitting. It should be increased to prevent overfitting.

- E is Incorrect: Rewriting the training logic in PyTorch introduces significant operational overhead. Furthermore, dropout is a neural network regularization technique, not applicable to tree-based XGBoost models.

> Key Takeaway: When using SageMaker built-in algorithms like XGBoost, mitigate overfitting by tuning regularization parameters (alpha, lambda, gamma, max_depth, min_child_weight) rather than switching frameworks. | | ### Question 2 - Preventing Catastrophic Forgetting in Bedrock Fine-tuning

An AI engineering team is using Amazon Bedrock to fine-tune a foundation model (FM) on a highly specific, proprietary dataset of internal IT support tickets. The customized model excels at resolving IT issues but has lost its general conversational capabilities and ability to summarize text, which it previously performed well.

The team needs to prevent this catastrophic forgetting while maintaining high performance on the new IT domain. The solution must utilize native Bedrock capabilities.

Which approach should the team take to fulfill these requirements?

- A) Increase the number of epochs during the fine-tuning job to force better convergence on the IT support tickets.

- B) Blend a portion of general-purpose conversational and summarization data into the fine-tuning dataset.

- C) Use Amazon Bedrock Knowledge Bases to retrieve IT documentation at runtime instead of fine-tuning the model.

- D) Lower the learning rate hyperparameter to the absolute minimum allowed by Bedrock customization APIs. | >! Correct Answer: B !

Correct Option Analysis:

- B is Correct: Catastrophic forgetting occurs when a model is customized on a narrow domain and overwrites its general capabilities. The standard defense is to mix broader, general-domain data into the training mixture so the model retains its previous skills while learning the new domain.

Distractor Analysis:

- A is Incorrect: Increasing epochs will force the model to memorize the IT tickets even more aggressively, worsening catastrophic forgetting of general capabilities.

- C is Incorrect: While using Knowledge Bases (RAG) is an excellent pattern to avoid fine-tuning entirely, the scenario specifically states the team needs to fine-tune the model and prevent forgetting during that process. RAG solves the domain knowledge issue but doesn't address the question of preserving FM capabilities during fine-tuning.

- D is Incorrect: While a lower learning rate can slightly reduce the severity of weight updates, it is not a primary or reliable defense against catastrophic forgetting. Data blending is the standard mitigation technique.

> Key Takeaway: To prevent catastrophic forgetting during foundation model fine-tuning, blend general-domain data with your domain-specific training data to preserve the model's broad capabilities. | | ### Question 3 - Resolving Underfitting in Script Mode

A data science team is running a custom PyTorch training script using Amazon SageMaker Script Mode. The team monitors the training job via Amazon CloudWatch and notices that both the training loss and the validation loss remain high and plateau early, never dropping below an acceptable threshold.

The team has confirmed there are no data quality issues. They must adjust their training configuration to resolve this underfitting with the least operational overhead.

Which TWO actions should the team take to fix the underfitting problem? (Select TWO)

- A) Increase the number of epochs to allow the model more time to learn the underlying signal.

- B) Increase the L2 regularization penalty applied to the optimizer to simplify the model.

- C) Decrease the model's capacity by reducing the number of layers and hidden units.

- D) Increase the learning rate to allow the optimizer to take larger steps toward the loss minimum.

- E) Enable SageMaker Distributed Data Parallel (SMDDP) to spread the training across multiple instances. | >! Correct Answers: A and D !

Correct Options Analysis:

- A is Correct: Underfitting means the model has not learned the signal. If the training job was stopped early or hasn't run long enough, increasing the number of epochs allows the optimizer more passes to minimize the loss and fit the data better.

- D is Correct: If the learning rate is too low, the optimizer may get stuck in a local minimum or plateau. Increasing the learning rate can help the optimizer escape this plateau and find a better minimum, reducing both training and validation loss.

Distractor Analysis:

- B is Incorrect: Regularization is used to prevent overfitting. Increasing L2 regularization on an underfitting model will penalize the weights further, making the model even simpler and worsening the underfitting.

- C is Incorrect: Decreasing the model's capacity (fewer layers/units) simplifies the model, which exacerbates underfitting. You would typically increase capacity to fix underfitting.

- E is Incorrect: Distributed training reduces the elapsed wall-clock time of training but does not change the model's fundamental ability to learn the signal. It does not resolve underfitting.

> Key Takeaway: Underfitting is characterized by high loss on both training and validation sets. Fix it by increasing model complexity, training for more epochs, or raising the learning rate. | | ### Question 4 - Mitigating Overfitting with SageMaker Automatic Model Tuning

A company is using SageMaker Automatic Model Tuning (AMT) to optimize a deep learning model. The data scientists notice that during the tuning job, many training jobs run for the full duration of their allocated steps, computing expensive gradients long after the validation loss has stopped improving and started to degrade.

The team must configure AMT to minimize wasted compute resources and prevent these specific training jobs from overfitting.

Which configuration change meets these requirements?

- A) Increase the MaxNumberOfTrainingJobs parameter in the tuning job configuration.

- B) Enable EarlyStopping and set it to Auto in the HyperParameterTuningJobConfig.

- C) Switch the tuning strategy from Bayesian to Random.

- D) Enable SageMaker Checkpointing to save intermediate model states to Amazon S3. | >! Correct Answer: B !

Correct Option Analysis:

- B is Correct: SageMaker AMT supports automatic early stopping. When set to Auto, SageMaker monitors the objective metric during a training job. If the metric stops improving (indicating the model is starting to overfit or plateau), it automatically terminates the training job early, saving compute resources and preventing overfitting.

Distractor Analysis:

- A is Incorrect: Increasing MaxNumberOfTrainingJobs runs more tuning experiments, but it does nothing to stop an individual training job from running too long and overfitting.

- C is Incorrect: Changing the search strategy alters how AMT explores the hyperparameter space, but it does not implement early stopping for individual jobs.

- D is Incorrect: Checkpointing is useful for resuming failed training jobs or capturing model snapshots, but it does not proactively terminate a job that is overfitting.

> Key Takeaway: Enabling EarlyStopping (Auto) in SageMaker Automatic Model Tuning stops individual training jobs when the objective metric stops improving, reducing compute costs and mitigating overfitting. | | ### Question 5 - Catastrophic Forgetting vs. Overfitting in JumpStart

A generative AI team fine-tuned a pre-trained text generation model from SageMaker JumpStart on a dataset of medical transcripts. Post-deployment, the team observes that the model now outputs highly accurate medical summaries. However, when prompted to perform general text translation (which it excelled at prior to fine-tuning), it produces irrelevant, hallucinated medical jargon.

The team must diagnose this issue accurately to determine the correct remediation strategy.

Which diagnosis correctly identifies the model's failure mode?

- A) The model is overfitting to the medical transcripts.

- B) The model is underfitting on the general translation task.

- C) The model is experiencing catastrophic forgetting.

- D) The model is suffering from high bias in the translation task. | >! Correct Answer: C !

Correct Option Analysis:

- C is Correct: Catastrophic forgetting occurs when a model is fine-tuned on a narrow domain (medical transcripts) and overwrites its previously learned general capabilities (text translation). The model retains its new domain knowledge but loses its general pre-training capabilities.

Distractor Analysis:

- A is Incorrect: Overfitting would manifest as the model memorizing the medical transcripts and failing to generalize to unseen medical data. The scenario states the model performs well on the medical task but fails on an entirely different general task.

- B is Incorrect: Underfitting implies the model failed to learn the underlying patterns of the training data. The model successfully learned the medical task, so it is not underfitting.

- D is Incorrect: High bias is synonymous with underfitting. This does not describe the loss of a previously held general capability.

> Key Takeaway: Catastrophic forgetting is the loss of previously learned general capabilities when a model is fine-tuned on a narrow, specific domain. | | ### Question 6 - Mitigating Overfitting in Ensemble Models

An enterprise is building a fraud detection system. The data science team trained several individual decision tree models using SageMaker Script Mode. Each individual model performs poorly and overfits the training data. The team decides to combine these models to improve overall performance and reduce overfitting.

They must implement this solution with the lowest operational overhead for deployment.

Which approach should the team use?

- A) Deploy all models to a single SageMaker endpoint using SageMaker Inference Recommender to load balance requests between them.

- B) Train a Random Forest or Gradient Boosted Tree ensemble using the SageMaker XGBoost built-in algorithm instead of combining individual models.

- C) Use AWS Step Functions to orchestrate real-time inference calls to each individual model endpoint and average the results in an AWS Lambda function.

- D) Increase the depth of each individual decision tree and deploy them behind an Application Load Balancer. | >! Correct Answer: B !

Correct Option Analysis:

- B is Correct: Combining multiple models (ensembles) is a standard technique to offset individual model errors and reduce overfitting. Using the SageMaker XGBoost built-in algorithm natively implements Gradient Boosted Trees (an ensemble method) which aggregates weak learners into a strong learner, directly addressing overfitting while minimizing operational overhead since the algorithm is managed.

Distractor Analysis:

- A is Incorrect: Hosting multiple independent overfitting models on one endpoint does not create a mathematical ensemble. Inference Recommender helps with instance selection and deployment configurations, not aggregating predictions to reduce variance.

- C is Incorrect: Orchestrating individual endpoint calls via Step Functions and Lambda introduces massive operational overhead and latency compared to using a natively trained ensemble algorithm.

- D is Incorrect: Increasing the depth of already overfitting decision trees will make the overfitting significantly worse, not better.

> Key Takeaway: Ensemble methods (like Random Forests or Gradient Boosting via XGBoost) combine multiple weak, overfitting learners to create a robust model that generalizes better and reduces variance. | | ### Question 7 - Underfitting in SageMaker Linear Learner

A retail company is predicting monthly sales volume using the SageMaker Linear Learner built-in algorithm. The team has run multiple training jobs, but the training and validation Mean Squared Error (MSE) remain unacceptably high. The model is clearly underfitting the complex, non-linear relationships in the sales data.

The team must adjust the model configuration to address the underfitting.

Which action will help resolve this issue?

- A) Increase the L1 regularization hyperparameter to force the model to learn sparser representations.

- B) Change the loss hyperparameter to logistic instead of mse.

- C) Decrease the learning_rate hyperparameter so the model takes smaller, more precise steps.

- D) Switch to a non-linear algorithm like SageMaker KNN or XGBoost that can capture the complex data distribution. | >! Correct Answer: D !

Correct Option Analysis:

- D is Correct: Linear Learner is a linear algorithm; it can only draw linear decision boundaries or fit linear regression planes. If the data has complex, non-linear relationships, a linear model will fundamentally underfit. Switching to a non-linear algorithm like KNN or XGBoost is the correct architectural fix.

Distractor Analysis:

- A is Incorrect: Increasing L1 regularization simplifies the model further (increasing bias), which worsens underfitting.

- B is Incorrect: Logistic loss is for classification problems, not regression. Changing to logistic loss would cause the job to fail or produce nonsensical results for a sales volume prediction task.

- C is Incorrect: Decreasing the learning rate might help if the model was bouncing around the minimum, but it does not change the fact that a linear model cannot fit non-linear data. It will just take longer to arrive at the same underfitting state.

> Key Takeaway: If a model underfits because the data is non-linear, no amount of hyperparameter tuning on a linear algorithm (like Linear Learner) will fix it. You must switch to a non-linear algorithm. | | ### Question 8 - Catastrophic Forgetting and Parameter-Efficient Fine-Tuning (PEFT)

A startup is fine-tuning a large language model using Amazon SageMaker JumpStart on a narrow domain of legal contracts. To reduce training costs and prevent catastrophic forgetting, they want to use a technique that updates only a small subset of weights or adds adapters, rather than updating the entire base model.

Which technique should they apply to fulfill these requirements?

- A) Full-parameter supervised fine-tuning with a very low learning rate.

- B) QLoRA (Quantized Low-Rank Adaptation) using the SageMaker distributed training library.

- C) Fine-tuning all layers but freezing the embedding layer.

- D) Reinforcement fine-tuning using a reward model based on legal accuracy. | >! Correct Answer: B !

Correct Option Analysis:

- B is Correct: QLoRA (and LoRA in general) is a Parameter-Efficient Fine-Tuning (PEFT) technique. It freezes the pre-trained model weights and injects trainable low-rank decomposition matrices (adapters) into the architecture. Because the base weights are frozen, the model is much less likely to suffer from catastrophic forgetting of general capabilities. It also requires significantly less memory (especially when quantized), reducing costs.

Distractor Analysis:

- A is Incorrect: Full-parameter fine-tuning updates every weight in the network, making it highly susceptible to catastrophic forgetting and requiring massive compute resources, violating both constraints.

- C is Incorrect: Freezing only the embedding layer still updates the vast majority of the transformer's weights (attention blocks, feed-forward networks), which still leads to catastrophic forgetting and high compute costs.

- D is Incorrect: Reinforcement fine-tuning optimizes the model against a reward function. It typically adjusts the entire model's behavior based on the reward, which can still induce catastrophic forgetting and is computationally expensive.

> Key Takeaway: PEFT techniques like LoRA freeze base model weights and train only small adapter modules, drastically reducing the risk of catastrophic forgetting while lowering training compute costs. | | ### Question 9 - Diagnosing Training Metrics in CloudWatch

A machine learning engineer is monitoring a SageMaker training job in Amazon CloudWatch. The engineer observes the following metrics over 50 epochs:

- Training Loss: Decreased steadily from 2.0 to 0.15.

- Validation Loss: Decreased from 2.0 to 0.8 by epoch 20, but then steadily increased back to 1.5 by epoch 50.

Given these specific CloudWatch metric trends, what is the correct diagnosis of the model's state at epoch 50?

- A) The model is experiencing high bias (underfitting).

- B) The model is experiencing high variance (overfitting).

- C) The model is experiencing catastrophic forgetting.

- D) The model has achieved optimal generalization and should be deployed. | >! Correct Answer: B !

Correct Option Analysis:

- B is Correct: The training loss continues to decrease while the validation loss begins to increase after a certain point (epoch 20). This divergence is the classic signature of high variance, also known as overfitting. The model is starting to memorize the noise in the training data rather than generalizing.

Distractor Analysis:

- A is Incorrect: High bias (underfitting) would show both training and validation loss remaining high and failing to converge. Here, training loss is very low (0.15).

- C is Incorrect: Catastrophic forgetting cannot be diagnosed purely from a single training and validation loss curve for one task. It requires evaluating the model on a separate, previously known task (e.g., general conversational ability) after domain-specific fine-tuning.

- D is Incorrect: Deploying a model at epoch 50 would mean deploying a model with degraded validation performance (1.5) compared to its best state at epoch 20 (0.8).

> Key Takeaway: In CloudWatch, if training loss decreases but validation loss increases, the model is overfitting. The best model is often the checkpoint saved at the lowest validation loss, not the end of the final epoch. | | ### Question 10 - Blending Data to Prevent Forgetting in Bedrock

A global e-commerce company is using Amazon Bedrock to fine-tune a foundation model for customer support chatbot applications. They want the model to learn the specific tone and policies of their brand without losing its broad reasoning capabilities.

The team has prepared 10,000 high-quality examples of brand-specific customer support interactions. They must structure the fine-tuning job to minimize catastrophic forgetting.

What is the recommended best practice for structuring the training data?

- A) Fine-tune exclusively on the 10,000 brand-specific examples to ensure maximum domain adaptation.

- B) Deduplicate the 10,000 examples and remove any examples that contain general conversational text.

- C) Mix the 10,000 brand-specific examples with a smaller subset of general-purpose instruction-following data.

- D) Use the 10,000 examples to train a reward model for Bedrock Reinforcement Fine-Tuning instead of supervised fine-tuning. | >! Correct Answer: C !

Correct Option Analysis:

- C is Correct: To prevent catastrophic forgetting during supervised fine-tuning, it is an AWS best practice to blend domain-specific data with a subset of general-purpose data. This ensures the model updates its weights to accommodate the new domain while continuously reinforcing its foundational reasoning and conversational capabilities.

Distractor Analysis:

- A is Incorrect: Fine-tuning exclusively on brand-specific data maximizes the risk of catastrophic forgetting, as the model's weights will shift entirely to the narrow distribution of the support interactions.

- B is Incorrect: Removing general conversational text from the brand-specific dataset does not add general-domain capabilities to the training mixture. It actually further isolates the training data to the narrow domain.

- D is Incorrect: While reinforcement fine-tuning is a valid customization technique, it optimizes the model against a reward function. To prevent forgetting of general reasoning during RLHF, you still need to blend data or use KL-divergence penalties against the base model. Supervised fine-tuning with blended data is the most direct and standard approach here.

> Key Takeaway: When fine-tuning in Bedrock, blend your domain-specific data with general-purpose data to minimize catastrophic forgetting and preserve the model's foundational reasoning skills. |