Skip to content

Skill 2.2.5: Prevent model overfitting, underfitting, and catastrophic forgetting

Question 1 - Diagnosing Overfitting in SageMaker Training

A financial services company is training a fraud detection model using Amazon SageMaker with a XGBoost built-in algorithm. The data science team observes that the model achieves an AUC of 0.98 on the training dataset but only an AUC of 0.71 on the held-out validation dataset. The team must improve generalization while minimizing additional engineering effort.

Which ONE action should the team take to address this issue?

  • A) Increase the number of boosting rounds (num_round) to allow the model more capacity to learn complex fraud patterns.
  • B) Reduce the maximum depth of trees (max_depth) and increase the regularization parameters (lambda and alpha).
  • C) Switch the objective metric from AUC to accuracy and retrain the model.
  • D) Increase the learning rate to converge faster on the training data.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: The significant gap between training AUC (0.98) and validation AUC (0.71) is a classic symptom of overfitting. Regularization techniques for tree-based models include: reducing tree depth (max_depth), increasing L1 regularization (alpha), and increasing L2 regularization (lambda). These prevent the model from memorizing noise specific to the training set.

Distractor Analysis

  • A is Incorrect: Increasing boosting rounds adds more trees, giving the model additional capacity to memorize training data patterns, which would worsen overfitting.
  • C is Incorrect: Changing the evaluation metric from AUC to accuracy does not alter the model's behavior or fix overfitting—it merely changes how performance is reported.
  • D is Incorrect: A higher learning rate accelerates convergence but can lead to unstable optimization and may further memorize training data, exacerbating the overfitting problem.

Key Takeaway: Overfitting manifests as strong training metrics paired with weak validation metrics; increase regularization and reduce model capacity rather than adding complexity.

Question 2 - Adjusting Hyperparameters to Fix Underfitting

A retail company built a product recommendation model using Amazon SageMaker with a custom PyTorch neural network in Script Mode. After training, the model achieves an AUC of 0.62 on the training set and an AUC of 0.60 on the validation set. The data science team has confirmed that the data preprocessing pipeline is correct and the dataset size is adequate. They want minimal changes to the existing script.

Which TWO hyperparameter adjustments should the team apply to fix the model's performance? (Select TWO)

  • A) Increase the number of hidden layers and units per layer in the neural network.
  • B) Add dropout layers with a high dropout rate (e.g., 0.7) after every layer.
  • C) Decrease the batch size from 1024 to 32.
  • D) Increase the learning rate and remove any existing weight decay.
  • E) Freeze all the layers of the network and only train the final output layer.
Click to reveal Answer & Explanation

Correct Answers: A and D

Correct Answers Analysis

  • A is Correct: Both training and validation AUC are low (0.62 and 0.60), indicating the model is underfitting—it lacks the capacity to learn the underlying patterns. Increasing model complexity (adding layers/units) increases representational capacity.
  • D is Correct: Underfitting may also result from the model learning too slowly or being over-constrained. Increasing learning rate accelerates learning updates, and removing weight decay removes an implicit regularization constraint that may be limiting the model's fit.

Distractor Analysis

  • B is Incorrect: Adding dropout layers with a high dropout rate introduces aggressive regularization, which is a remedy for overfitting, not underfitting. It would further reduce the model's capacity to learn.
  • C is Incorrect: Decreasing batch size changes the noise in gradient estimation (smaller batches introduce more stochasticity which can act as implicit regularization); this is not a direct fix for underfitting and may introduce unnecessary complexity without increasing model capacity.
  • E is Incorrect: Freezing layers reduces trainable parameters and capacity, which would further constrain the model and make underfitting worse.

Key Takeaway: Underfitting is characterized by poor performance on both training and validation data; increase model capacity (depth/width) or remove excessive regularization constraints.

Question 3 - Early Stopping to Prevent Overfitting on SageMaker

An AI startup is training a large Transformer-based language model using Amazon SageMaker with PyTorch. The team has configured n_epochs = 50 and observes that training loss steadily decreases while validation loss plateaus around epoch 15 and begins to increase slightly after epoch 20. The team wants to implement a solution that automatically stops training when the validation metric stops improving, without requiring manual monitoring of CloudWatch logs.

Which TWO approaches should the team implement? (Select TWO)

  • A) Use the SageMaker SDK's estimator with epochs=50 and rely on built-in early stopping in the PyTorch framework container.
  • B) Add a stop_training() handler in the training script that checks the validation loss at each epoch and breaks out of the training loop when it has not improved for N epochs, then use SageMaker Debugger to visualize the results.
  • C) Use SageMaker Automatic Model Tuning (AMT) with early_stopping_type set to "Auto" and an objective metric of validation loss.
  • D) Manually watch CloudWatch logs and terminate the job when validation loss increases for 5 consecutive epochs.
  • E) Increase the train_batch_size to 2048 to reduce the number of epochs required.
Click to reveal Answer & Explanation

Correct Answers: B and C

Correct Answers Analysis

  • B is Correct: Implementing early stopping logic inside the training script (checking validation loss and invoking stop_training() from the SageMaker Python SDK or simply breaking the loop) is a standard technique that directly prevents overfitting by stopping the run when generalization ceases to improve.
  • C is Correct: SageMaker Automatic Model Tuning (AMT) includes built-in support for early stopping via its early_stopping_type="Auto" setting, which automatically stops under-performing job variants early, making it an efficient automated solution.

Distractor Analysis

  • A is Incorrect: SageMaker's prebuilt framework containers do not include automatic early stopping for custom training scripts. Early stopping logic must be implemented explicitly inside the script.
  • D is Incorrect: Manual monitoring of CloudWatch logs introduces unnecessary operational overhead and contradicts the requirement for an automated solution.
  • E is Incorrect: A larger batch size reduces the number of weight updates per epoch (fewer steps), which can change the learning dynamics but does not automatically prevent overfitting. It does not address the need for an early stopping mechanism.

Key Takeaway: Early stopping is a primary defense against overfitting; implement it inside the training loop or leverage AMT's Auto early stopping feature rather than relying on manual log monitoring.

Question 4 - Data Augmentation to Combat Overfitting in Computer Vision

A medical imaging company is training a Convolutional Neural Network (CNN) using Amazon SageMaker for detecting lung nodules in chest X-ray images. The team has a relatively small labeled dataset (3,000 images) and observes that the model achieves 99% accuracy on training data but only 72% accuracy on the validation set. The team is concerned about compliance with data privacy and cannot acquire additional patient data easily.

Which ONE approach should the team implement to MOST effectively reduce overfitting while maintaining the existing dataset size?

  • A) Use SageMaker's built-in image classification algorithm (which already applies data augmentation internally) and declare the data source via DataSource.
  • B) Apply image augmentation techniques (random rotations, horizontal flips, brightness/contrast jittering, and random cropping) within the training script to generate synthetic variations of the training images.
  • C) Add more convolutional layers with higher filter counts to the network architecture.
  • D) Increase the number of epochs to allow the model more opportunities to see the training data.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: Data augmentation synthesizes new training examples through transformations (rotation, flipping, scaling, color jittering), effectively expanding the training dataset without collecting new data. This reduces overfitting by forcing the model to learn invariant, generalizable features rather than memorizing exact training images.

Distractor Analysis

  • A is Incorrect: SageMaker's built-in Image Classification algorithm does not inherently apply data augmentation to the input dataset by default. Data augmentation must be implemented in the training pipeline or via a custom script-mode container.
  • C is Incorrect: Adding more convolutional layers increases model capacity, making it even easier for the model to memorize the training images and worsen overfitting, particularly with a small dataset.
  • D is Incorrect: Increasing epochs means more passes over the same small dataset, which gives the model more opportunities to memorize specific training examples, deepening the overfitting problem.

Key Takeaway: When data is limited, data augmentation is the most effective defense against overfitting because it increases the diversity of the training distribution without requiring new labeled data.

Question 5 - Preventing Catastrophic Forgetting in Fine-Tuning

A customer service AI company is fine-tuning a pre-trained GPT-3.5 foundation model using Amazon Bedrock to handle a niche domain (mortgage loan applications). The fine-tuning job uses a curated dataset of 10,000 mortgage-specific dialogs. After deployment, the AI team discovers that while the model excels at mortgage topics, it has completely lost its ability to handle general customer service inquiries (e.g., password resets, order tracking) that it previously handled effectively.

Which TWO strategies should the team implement to prevent this from happening? (Select TWO)

  • A) Use catastrophic forgetting mitigation by including a subset of the original, broader training data in the fine-tuning mixture.
  • B) Fine-tune with a lower learning rate and a smaller number of epochs to reduce the magnitude of weight updates.
  • C) Add more mortgage-specific data to the fine-tuning dataset only, so the model becomes even more specialized.
  • D) Remove the general customer service capability from the model by training only on the mortgage domain.
  • E) Use parameter-efficient fine-tuning (PEFT) techniques such as Low-Rank Adaptation (LoRA) instead of full fine-tuning.
Click to reveal Answer & Explanation

Correct Answers: A and E

Correct Answers Analysis

  • A is Correct: Blending broader representative data into the fine-tuning set is the standard defense against catastrophic forgetting. This maintains the model's original span of knowledge while specializing it in the new domain.
  • E is Correct: Parameter-efficient fine-tuning (e.g., LoRA) freezes most of the pre-trained weights and only updates a few additional low-rank matrices injected into each layer. This drastically reduces the risk of catastrophic forgetting because the original pre-trained weights are preserved.

Distractor Analysis

  • B is Incorrect: While lower learning rates and fewer epochs help reduce catastrophic forgetting, they are only partial mitigations and are not as effective alone as data blending or PEFT. (This option is plausible but not the primary strategy in the question context.)
  • C is Incorrect: Adding more of the narrow-domain data further biases the model toward that domain and increases catastrophic forgetting of general capabilities.
  • D is Incorrect: Deliberately removing general capabilities by training only on one narrow domain is counterproductive; the goal is to specialize without losing general abilities.

Key Takeaway: Catastrophic forgetting occurs when fine-tuning a foundation model on a narrow domain degrades its original, broader capabilities; prevent it by combining PEFT methods with mixed-domain training data.

Question 6 - Using SageMaker Debugger to Detect Overfitting Early

A streaming service company is training a movie recommendation model using Amazon SageMaker with a custom PyTorch training script. The team wants to automatically detect and alert data scientists when the training run starts to overfit, without manually polling CloudWatch logs after every epoch. They also want the ability to run custom logic (e.g., save a checkpoint) when overfitting is detected.

Which ONE AWS service or feature should the team use?

  • A) SageMaker Automatic Model Tuning (AMT) with early_stopping_type="Auto"
  • B) Amazon SageMaker Debugger with built-in overfitting detection rules and a custom Rule action.
  • C) Amazon CloudWatch Alarms set on the training loss metric.
  • D) Amazon Bedrock Guardrails with PII redaction enabled.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: SageMaker Debugger monitors training jobs and evaluates built-in rules (e.g., Overfit, Overtraining) against incoming tensors and metrics. Debugger can emit alerts and trigger custom actions (such as writing to S3 or sending an SNS notification) when a rule fires, meeting the requirement for automated overfitting detection with custom logic.

Distractor Analysis

  • A is Incorrect: SageMaker AMT is a hyperparameter tuning service that performs search across configurations. While it offers early stopping per trial, it does not provide custom rule-based detection or trigger custom actions in the same way Debugger does for a single training job.
  • C is Incorrect: CloudWatch Alarms alone can only fire on a metric (e.g., training loss) crossing a static threshold. They cannot detect the relationship between training and validation loss that indicates overfitting, nor run arbitrary Python actions when the pattern is detected.
  • D is Incorrect: Amazon Bedrock Guardrails performs content filtering and PII redaction at inference time for generative AI models; it is not involved with training job monitoring.

Key Takeaway: Use SageMaker Debugger's built-in overfitting detection rules for automated, event-driven training run monitoring; CloudWatch Alarms alone cannot capture the training-vs-validation divergence that signals overfitting.

Question 7 - Addressing Underfitting via Learning Rate and Epoch Tuning

A logistics company is training a delivery time prediction model using Amazon SageMaker with a Linear Learner algorithm. The team sets epochs=10 and learning_rate=0.0001. The resulting model has a mean squared error (MSE) of 45.2 on the training set and MSE of 44.8 on the validation set—both significantly higher than the business threshold. The team has confirmed the input features are correct (including distance, driver count, and weather data).

Which ONE adjustment should the team make to improve the model's fit?

  • A) Increase the learning rate to 0.01 and increase the number of epochs to 100.
  • B) Add L2 regularization (weight_decay) to the Linear Learner.
  • C) Decrease the batch size to 8.
  • D) Switch from Linear Learner to a simpler algorithm such as K-Means.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Answer Analysis

  • A is Correct: High error on both training and validation sets is a signature of underfitting. The model is either learning too slowly (small learning rate) or has not been given enough iterations to converge (low epoch count). Increasing both the learning rate and the number of epochs gives the optimizer a greater chance to reach a suitable minimum.

Distractor Analysis

  • B is Incorrect: Adding L2 regularization further constrains the model, making it even harder to fit the training data—worsening underfitting.
  • C is Incorrect: Decreasing batch size increases stochasticity in gradient estimation, which can help with generalization (overfitting) but does not directly address underfitting. It may also slow down training and add overhead.
  • D is Incorrect: K-Means is a clustering algorithm, not a supervised regression algorithm; switching to it does not solve delivery time prediction.

Key Takeaway: Underfitting can be corrected by giving the optimizer more opportunity to learn (higher learning rate, more epochs) rather than shrinking the model further.

Question 8 - SageMaker Multi-Model Endpoint for Serving Cost Optimization

A large e-commerce platform has 200 lightweight product categorization models (all using the same Scikit-learn container). These models were trained independently for different product verticals (electronics, clothing, groceries, etc.). Due to low traffic for each individual model, the company wants to host them in a cost-effective manner that maximizes resource utilization while accepting occasional cold-start latency spikes.

Which ONE AWS solution should the team use?

  • A) Create one SageMaker endpoint for each of the 200 models, each with one ml.t2.medium instance.
  • B) Use SageMaker Multi-Model Endpoint (MME) with a shared serving container that loads models on-demand from S3 when they are invoked.
  • C) Use Amazon Bedrock with a prompt engineering approach, and do not use SageMaker endpoints.
  • D) Use AWS Step Functions to spin up and spin down endpoints on a schedule to reduce running time.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: SageMaker Multi-Model Endpoint (MME) allows multiple models to be hosted behind a single endpoint using a shared serving container. Models are dynamically loaded from S3 into memory when a request arrives, then evicted after inactivity. This dramatically reduces cost by sharing instance resources across many models, with the trade-off of occasional cold-start latency when a model is not already loaded.

Distractor Analysis

  • A is Incorrect: One endpoint per model requires 200 separate instances (or at least 200 separate endpoints, often each with minimum one instance each), which is highly costly compared to sharing infrastructure via MME.
  • C is Incorrect: Amazon Bedrock is a managed foundation model service for generative AI tasks, not a general-purpose hosting platform for custom lightweight Scikit-learn models.
  • D is Incorrect: AWS Step Functions could schedule endpoint start/stop actions, but each model would still need its own endpoint resource. This adds operational complexity, while MME already handles resource sharing and dynamic loading.

Key Takeaway: SageMaker Multi-Model Endpoint (MME) is the go-to cost optimization pattern for hosting many similar models on shared infrastructure, trading cold-start latency for lower cost.

Question 9 - Monitoring and Mitigating Catastrophic Forgetting in Bedrock Fine-Tuning

A legal technology company is fine-tuning a base Anthropic Claude model via Amazon Bedrock to draft legal contracts in a specific practice area. After a successful fine-tuning job, the team validates the model on a holdout set of contract-drafting prompts. The model scores high on legal contract generation (BLEU 0.81). However, when evaluated on general English writing tasks (summarization, translation, general question answering), the BLEU drops from 0.78 (pre-fine-tune) to 0.42 (post-fine-tune).

Which TWO actions should the team take? (Select TWO)

  • A) Re-run fine-tuning with a mixture of general English tasks and legal contract drafting tasks in the training set.
  • B) Increase the size of the fine-tuning dataset to include 500,000 legal contracts for stronger specialization.
  • C) Use Amazon Bedrock Prompt Management to create varied prompts to mitigate duplication in the training data.
  • D) Fine-tune with lower learning rate and use a LoRA-based parameter-efficient method if available.
  • E) Delete the fine-tuned model and go back to the base model for all use cases.
Click to reveal Answer & Explanation

Correct Answers: A and D

Correct Answers Analysis

  • A is Correct: Catastrophic forgetting occurs when the fine-tuning task distribution is too narrow; blending general English examples during fine-tuning preserves the model's pre-existing broad capabilities.
  • D is Correct: Lower learning rates reduce the magnitude of parameter updates, and PEFT methods (such as LoRA) freeze most base parameters while updating only a small set of adapter weights. Both help prevent catastrophic forgetting.

Distractor Analysis

  • B is Incorrect: Increasing the size of the fine-tuning dataset with only narrow-domain examples will further push the model toward the legal domain and worsen forgetting of general English capabilities.
  • C is Incorrect: Prompt Management is about organizing, tracking, and versioning prompts—not about mitigating catastrophic forgetting in the training data.
  • E is Incorrect: Completely discarding the fine-tuned model ignores the business requirement for legal specialization; the right approach is to preserve both capabilities during customization.

Key Takeaway: When fine-tuning foundation models on narrow domains, always mix in broader representative data and consider lower learning rates or PEFT to prevent catastrophic forgetting.

Question 10 - Early Stopping in SageMaker AMT vs. Script Mode

A cybersecurity company is training an anomaly detection model using Amazon SageMaker and a custom TensorFlow script. The team wants to reduce training time while ensuring the model does not overfit. They are considering two approaches: (1) implementing early stopping inside the training script, and (2) using SageMaker Automatic Model Tuning (AMT) with Bayesian search.

Which TWO statements about these approaches are TRUE? (Select TWO)

  • A) Early stopping inside the training script stops the individual training run when validation performance does not improve for N consecutive evaluations.
  • B) AMT early stopping (early_stopping_type="Auto") stops the entire hyperparameter search when one trial overfits.
  • C) Early stopping inside the script and AMT early stopping serve the same purpose and are interchangeable.
  • D) Implementing early stopping inside the training script reduces the runtime of a single trial by halting it once generalization stops improving.
  • E) AMT with Bayesian search will never overfit the objective metric because it uses prior knowledge.
Click to reveal Answer & Explanation

Correct Answers: A and D

Correct Answers Analysis

  • A is Correct: Early stopping inside the script (via TensorFlow's EarlyStopping callback or custom logic) monitors a validation metric and halts training when improvement plateaus, protecting against overfitting.
  • D is Correct: The primary benefit of early stopping is reducing the number of epochs a training job actually runs, thereby shortening the runtime of that trial while still achieving good generalization.

Distractor Analysis

  • B is Incorrect: AMT's early_stopping_type="Auto" stops individual trials (training jobs) that under-perform relative to the best result after a certain number of epochs; it does not stop the entire search.
  • C is Incorrect: Early stopping inside a script directly shortens the training loop of a single job, while AMT early stopping is a higher-level mechanism that decides whether to stop a tuning trial early. They are related but not interchangeable.
  • E is Incorrect: Bayesian optimization improves search efficiency by using prior trial information to suggest the next hyperparameter combination but does not prevent the model itself from overfitting to the training data.

Key Takeaway: Early stopping inside a script saves runtime and improves generalization for a single training run, whereas AMT early stopping operates at the tuning search level.