Skip to content

Skill 2.3.5: Debug model convergence issues

Question 1 - Diagnosing Oscillating Loss During Training

A financial services company is training a fraud detection model using Amazon SageMaker AI. The data science team observes that the training loss oscillates wildly between epochs on the binary cross-entropy curve, sometimes decreasing, then spiking back up. They are using the Adam optimizer with a learning rate of 0.1 and training on a GPU instance. The validation loss also fluctuates, making it impossible to determine whether the model is learning.

The team must diagnose and fix the convergence issue before spending additional compute on the training run.

Which ONE change is the FIRST step the team should take to stabilize training convergence?

A) Increase the batch size from 64 to 512 to reduce variance in gradient estimates. B) Reduce the learning rate to a smaller value and apply a learning rate warm-up schedule. C) Add L2 regularization to the loss function to prevent overfitting. D) Replace Adam with the SGD optimizer, which is always more stable for binary classification. E) Increase the number of training epochs from 50 to 200 to allow the loss more time to settle.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: A learning rate of 0.1 is extremely high for Adam, which typically uses default values between 1e-3 and 1e-4. When the learning rate is too high, the optimizer repeatedly overshoots local minima, causing oscillating loss curves. Reducing the learning rate and using a warm-up schedule (gradually increasing from a small initial value) is the standard first fix for convergence instability.

Distractor Analysis

  • A is Incorrect: Increasing batch size reduces noise in gradient estimates and can help in some cases, but it does not directly address an oversized learning rate. It also changes effective learning dynamics and may require scaling the learning rate proportionally.
  • C is Incorrect: L2 regularization controls model complexity and overfitting, not training instability. The oscillation described occurs during training and affects validation similarly, so overfitting is not the primary issue here.
  • D is Incorrect: While SGD is sometimes used for binary classification, replacing the optimizer without first adjusting the learning rate could make the problem worse because SGD is more sensitive to learning rate selection than Adam.
  • E is Incorrect: Extending training epochs on an unstable loss curve wastes compute resources; the loss will continue oscillating regardless of additional epochs.

Key Takeaway: Learning rates that are too high cause overshooting and oscillating loss curves; for Adam, values above 1e-2 are almost always too aggressive—reduce and use warm-up before investigating other causes.

Question 2 - Distinguishing Overfitting from Underfitting

A retail analytics startup is training a customer churn prediction model (XGBoost) on Amazon SageMaker AI. After 100 training rounds, the team plots the log loss curves and observes:

  • Training log loss: 0.08 (very low, still decreasing)
  • Validation log loss: 0.62 (increasing since epoch 40)

The team plans to address the issue by increasing model depth and adding more trees to improve the validation score.

An experienced ML engineer corrects this approach. What does the evidence suggest the team should do instead?

Which ONE action best addresses the root cause of the validation loss behavior?

A) Increase the max_depth from 6 to 12 and raise num_round to 300 to give the model more capacity. B) Implement early stopping based on validation loss and add regularization parameters (e.g., gamma, lambda, min_child_weight). C) Switch the loss function from logistic to hinge loss, which generalizes better on validation sets. D) Remove the validation split and train on all data to maximize signal. E) Reduce the number of features by using only the top 5 most important ones.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: The gap between training loss (0.08, very low) and validation loss (0.62, increasing) is the classic signature of overfitting. The model memorizes training patterns but fails to generalize. Early stopping halts training when validation loss stops improving, while XGBoost regularization parameters (gamma for minimum split improvement, lambda for L2 penalty, min_child_weight for minimum leaf weight) restrict model complexity.

Distractor Analysis

  • A is Incorrect: Increasing depth and rounds exacerbates overfitting by giving the model more capacity to memorize training noise, widening the train-validation gap even further.
  • C is Incorrect: The loss function choice does not address the fundamental overfitting issue. Additionally, hinge loss is primarily used for SVM classification, not standard XGBoost churn modeling.
  • D is Incorrect: Removing validation removes the only signal of generalization performance. The model will still overfit—you will simply not be able to detect or measure it.
  • E is Incorrect: Dropping to only 5 features is extreme and may discard valuable predictive signal, likely causing underfitting. Feature selection should be based on importance and validation impact, not blanket reduction.

Key Takeaway: Plots with low training loss and high, increasing validation loss indicate overfitting—reduce model capacity or add regularization, do not increase it.

Question 3 - Debugging Vanishing Gradients in Deep Networks

A healthcare imaging startup is fine-tuning a deep convolutional neural network (e.g., ResNet-50) for medical X-ray classification using Amazon SageMaker AI. The team loads pre-trained ImageNet weights, freezes most layers, and replaces the final classification head.

After 3 epochs, the loss has barely changed from its initial value, and the model accuracy is only slightly better than random. The team suspects vanishing gradients.

Which TWO symptoms are consistent with vanishing gradients, and what should the team do? (Select TWO)

A) Gradient norms for early layers are near zero, while later layers show some updates. B) The model predicts the majority class for every input. C) The learning rate is too high, causing the model to overshoot every minimum. D) Add batch normalization layers and consider using skip connections to improve gradient flow. E) Reduce the batch size because large batches make the gradient vanish.

Click to reveal Answer & Explanation

Correct Answer: A and D

Correct Options Analysis

  • A is Correct: Vanishing gradients occur when backpropagated error signals become progressively smaller in earlier layers. Near-zero gradients in initial layers while later layers still update is the hallmark of this phenomenon.
  • D is Correct: Batch normalization stabilizes layer activations (reducing internal covariate shift) and skip connections (like ResNet's identity mappings) allow gradients to flow directly through the network without passing through multiple multiplicative gates.

Distractor Analysis

  • B is Incorrect: Predicting the majority class for all inputs is more indicative of severe underfitting or a collapsed model head that has not learned any decision boundary, but it is not a direct symptom of vanishing gradients (though it can occur when gradients are too small to train the classification head properly).
  • C is Incorrect: Vanishing gradients are an issue of too-small updates, the opposite of the overshooting caused by a high learning rate. A high LR would cause oscillation, not stagnation.
  • E is Incorrect: Batch size does not cause vanishing gradients directly. The problem stems from activation function choices (e.g., sigmoid saturation), network depth without skip connections, or poor initialization.

Key Takeaway: Customizing a pre-trained model by building from SageMaker's available framework containers while adjusting normalization and layer connectivity addresses gradient flow problems without starting from scratch—leverage ReLU/skip connections to prevent vanishing gradients at the source.

Question 4 - Exploding Gradients and Gradient Clipping

A natural language processing (NLP) engineering team is training a Transformer-based sequence-to-sequence model for a document translation service on Amazon SageMaker AI. During training, the team observes that the training loss suddenly spikes to NaN after the first few hundred steps.

The team investigates and discovers that some gradient values exceed 10^6.

What is the immediate cause of this issue, and which ONE fix should the team apply first?

A) The model is underfitting because the training dataset is small; the team should increase the data size. B) The gradients are exploding due to unstable backpropagation through deep layers; apply gradient clipping to limit the global norm. C) The optimizer is Adam, and Adam cannot handle Transformer models; replace with Adagrad. D) The learning rate schedule is decaying too quickly; use a constant learning rate instead. E) The data contains duplicate sentences that cause the loss to diverge; remove duplicates before training.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: Gradient values exceeding 10^6 and loss hitting NaN are definitive symptoms of exploding gradients. This is common in deep recurrent or Transformer architectures without proper initialization or normalization. Gradient clipping (e.g., clamping the global gradient norm to a maximum value such as 1.0 or 5.0) is the standard first-aid fix and is built into most deep learning frameworks.

Distractor Analysis

  • A is Incorrect: Underfitting and small datasets do not cause loss to spike to NaN or gradients to reach extreme magnitudes. Data insufficiency typically results in poor but finite loss values.
  • C is Incorrect: Adam is widely used and well-suited for Transformer training. The optimizer is not causing gradient explosion—unbounded gradient propagation is.
  • D is Incorrect: A decaying learning rate prevents instability rather than causing it. Using a constant learning rate would likely worsen the problem.
  • E is Incorrect: Duplicate data can cause memorization and poor generalization, but not gradient values in the millions or NaN loss.

Key Takeaway: NaN loss and extremely large gradient norms indicate exploding gradients—apply global norm gradient clipping (e.g., clipnorm=1.0) before investigating other causes.

Question 5 - Slow Convergence from Poor Learning Rate Schedule

An e-commerce company is training a deep learning recommendation model on Amazon SageMaker AI using Cosine Annealing with warm restarts as the learning rate schedule.

The team notices that the training loss decreases extremely slowly after the first epoch and the validation score barely improves over 50 epochs. The model's final accuracy is only slightly better than a simple baseline. There are no signs of oscillation or overfitting.

Which ONE adjustment to the learning rate schedule is most likely to improve convergence speed without causing instability?

A) Switch to a step decay schedule that drops the learning rate by 50% every 5 epochs. B) Set the learning rate to a fixed, constant value throughout training. C) Increase the warm-up period to cover the entire training run to avoid sudden changes. D) Decrease the initial learning rate by a factor of 100 to allow more fine-grained updates. E) Remove the learning rate schedule entirely and manually tune a single learning rate using grid search.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: A slow decrease in training loss with no oscillation suggests the learning rate might be too conservative or the schedule is spending too much time at low learning rates. Step decay provides a structured reduction that begins with a reasonably large LR for faster progress, then reduces it to avoid overshooting later in training.

Distractor Analysis

  • B is Incorrect: A fixed learning rate can work but usually requires extensive trial and error. Without a schedule, the model may either oscillate or converge too slowly; schedules exist specifically to balance these trade-offs.
  • C is Incorrect: Extending warm-up to the entire training run defeats the purpose of a schedule—it would make learning unrealistically slow and likely lead to underfitting.
  • D is Incorrect: Reducing the learning rate by 100x would worsen the already-slow convergence. A lower LR yields smaller updates, requiring far more epochs to make meaningful progress.
  • E is Incorrect: Removing the schedule and grid-searching a single value does not address the need for dynamic adjustment during training and incurs significant compute cost.

Key Takeaway: Slow, steady loss descent without oscillation indicates the learning rate may be too low or the schedule too conservative—consider a schedule that starts moderate and decays, such as step decay or linear warm-up with cosine decay.

Question 6 - Convergence Failure from Data Pipeline Issues

An autonomous vehicle startup is training a semantic segmentation model for road scene understanding using Amazon SageMaker AI with Amazon S3 as the data source. The team preprocesses thousands of images into shards using AWS Glue.

During training, the model converges quickly in the first epoch, but then plateaus at a loss of 1.35 and never improves for 40 epochs. Upon inspection, the last mini-batch's loss is identical across 10 consecutive steps.

Which of the following is the most likely cause of the stalled convergence?

A) The learning rate is set too high and the model is overshooting the optimum in every step. B) The data pipeline has a bug causing the same mini-batch of images to be repeatedly fed to the model; the model has effectively memorized one batch. C) The model architecture is too simple and lacks capacity. D) The validation set contains images that are too similar to the training set. E) The optimizer is stuck in a saddle point, which cannot be escaped without increasing momentum.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: A loss that plateaus and becomes identical across consecutive steps is a strong signal of a data pipeline bug. If the same mini-batch is fed repeatedly (e.g., due to improper sharding or a buggy iterator), the model quickly overfits that small dataset and the gradient updates approach zero, producing the repetitive loss pattern.

Distractor Analysis

  • A is Incorrect: A high learning rate causes oscillating loss, not a flat plateau. Oscillation would display as loss jumping up and down between steps, not remaining constant.
  • C is Incorrect: A model lacking capacity would show slow improvement and settling at a mediocre loss (e.g., hovering around a certain value), but the loss would not become identical across steps because the optimizer would still produce varied gradients.
  • D is Incorrect: A validation set too similar to training would produce a deceptively high validation score, not a plateau at a high loss.
  • E is Incorrect: Saddle points can stall convergence in high-dimensional optimization, but the loss rarely becomes literally identical across 10 steps. Saddle point dynamics produce small but non-zero variation.

Key Takeaway: Before tuning hyperparameters, verify data pipeline correctness—identical loss across steps or batches strongly indicates repeated data or a faulty data loader, not an optimization problem.

Question 7 - Troubleshooting Weight Initialization Issues

A startup is building a multi-layer perceptron (MLP) for loan default prediction and trains it from scratch (no pre-training) in Amazon SageMaker AI.

After the first epoch, the training loss is NaN, the validation loss is NaN, and all predictions are 0.5 (sigmoid output). The team checks the data pipeline and finds no duplicates or missing values. The learning rate is set to 0.001, which is reasonable for the optimizer.

Which TWO actions are most likely to fix the NaN loss? (Select TWO)

A) Reduce the batch size from 256 to 32 to increase gradient accuracy. B) Initialize all weights to zero to ensure uniform gradient flow. C) Replace sigmoid activations with ReLU in all hidden layers to prevent saturation. D) Reinitialize weights using Xavier/Glorot uniform initialization, which scales weights based on input/output dimensions. E) Increase the learning rate from 0.001 to 0.1 to overcome the NaN and push through the plateau.

Click to reveal Answer & Explanation

Correct Answer: C and D

Correct Options Analysis

  • C is Correct: Sigmoid activations saturate for large positive or negative inputs (gradient ≈ 0 or numerical instability). In deep networks, this can lead to NaN loss if activations output extreme values that propagate into numerically unstable operations. ReLU mitigates this by preventing saturation on the positive side, though it does not solve all initialization issues alone.
  • D is Correct: Xavier/Glorot initialization scales weight variance based on the fan-in and fan-out of each layer, keeping activations and gradients within reasonable numerical ranges. Poor random initialization (e.g., weights too large) can cause activations to explode through layers, producing NaN or vanishing to exactly zero.

Distractor Analysis

  • A is Incorrect: Batch size reduction increases gradient variance but does not fundamentally address numerical instability caused by poor initialization or activation saturation.
  • B is Incorrect: Initializing all weights to zero breaks symmetry and prevents learning entirely. Every neuron receives identical gradients, and the model fails to update properly.
  • E is Incorrect: Increasing the learning rate when NaN loss occurs will worsen the problem by amplifying already-unstable gradients.

Key Takeaway: NaN loss with sigmoid outputs and fully-connected architectures often indicates weight initialization or activation saturation issues—use Xavier/He initialization and ReLU/LeakyReLU for hidden layers.

Question 8 - Diagnosing Non-Convergence from Loss Function Mismatch

A logistics company is training a predictive model for delivery time estimation (continuous outcome in hours, ranging from 0.5 to 120 hours) on Amazon SageMaker AI.

The team configures the model with a softmax output layer and the categorical cross-entropy loss function. Training loss decreases and converges to 0.02, but inference predictions are nonsensical (e.g., all outputs are 0.0, 1.0, or extreme values outside the expected range). Validation loss is also very low (0.03), suggesting "successful" training.

What is the root cause of the model's incorrect outputs despite low loss?

A) The learning rate was too low, causing the model to underfit the distribution. B) The loss function and output layer are mismatched with the prediction task (regression) and produce meaningless outputs for continuous targets. C) The model suffers from severe overfitting due to insufficient regularization. D) The dataset was not normalized, so the model learned incorrect scaling. E) The optimizer is stuck in a local minimum and needs to be changed to AdamW.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: A softmax output layer paired with categorical cross-entropy is designed for multi-class classification, not regression. For continuous quantity prediction (delivery time in hours), the model needs a linear output layer (or another suitable regression head) with a loss function like Mean Squared Error (MSE) or Huber loss. The model "converged" to a low loss on a mathematically incompatible objective, resulting in meaningless predictions.

Distractor Analysis

  • A is Incorrect: A low training and validation loss of 0.02 indicates the model learned the (wrong) objective well, not underfitting.
  • C is Incorrect: Overfitting would show low training loss and high validation loss. Here both are low, and the predictions are structurally wrong rather than simply poor generalization.
  • D is Incorrect: Missing normalization may degrade regression quality, but it would not cause the loss to be low while outputs are entirely outside the expected range. The fundamental issue is the output/loss mismatch.
  • E is Incorrect: The optimizer minimizing the wrong loss function is not a local minimum problem; it is an objective specification problem. Changing optimizer will not correct a task mismatch.

Key Takeaway: Always validate that the output layer and loss function match the task type—regression tasks require regression heads (linear output + MSE/Huber), and classification tasks require softmax/sigmoid + cross-entropy.

Question 9 - Training Stability with Unnormalized Features

A machine learning team is training a logistic regression model for ad click-through-rate (CTR) prediction using Amazon SageMaker AI. The features include raw counts (impressions, clicks), time-based variables (seconds since last interaction), and user engagement metrics.

During training, the loss initially decreases, but then fluctuates and sometimes rises. Gradient values remain within normal ranges, and the learning rate is set to 0.01 with SGD, which is standard.

What is the most likely cause of unstable convergence in this scenario?

A) The model architecture is too deep and suffers from vanishing gradients. B) The features are on wildly different scales (e.g., raw counts up to millions vs. dwell time in seconds), causing gradients to pull unevenly during optimization. C) The optimizer should be switched from SGD to Adam, since SGD cannot handle sparse features. D) The loss function is incorrect for CTR prediction; it should use mean absolute error. E) The batch size is too large, causing gradient accumulation to become unstable.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: Features with dramatically different scales cause ill-conditioned optimization problems. A feature ranging from 0 to 10^8 can dominate the gradient contribution, while a feature from 0 to 1 barely affects the update. The optimizer oscillates because it cannot find a balanced step size across dimensions. Standardizing or normalizing features (e.g., z-score, min-max) to comparable ranges stabilizes convergence for linear models and neural networks.

Distractor Analysis

  • A is Incorrect: Logistic regression is a shallow model (single layer + sigmoid). Vanishing gradients are not a significant issue for single-layer models.
  • C is Incorrect: SGD handles sparse features fine, though Adam provides adaptive learning rates that help with differing gradient scales. The root cause is the unnormalized input features, not the choice between optimizers.
  • D is Incorrect: CTR prediction is a binary classification problem; the appropriate loss is binary cross-entropy, not mean absolute error.
  • E is Incorrect: Large batch sizes typically produce smoother (more stable) gradient estimates, not instability. Small batch sizes create higher variance in gradients.

Key Takeaway: Unnormalized features across orders of magnitude cause optimization instability; standardize inputs (z-score or min-max) before training linear models and use batch normalization in deeper architectures.

Question 10 - Saddle Points and Plateau Escape Strategies

A research team is training a Graph Neural Network (GNN) for molecular property prediction on Amazon SageMaker AI. The training loss curve displays a long plateau at 0.45, where the loss decreases extremely slowly over 30 epochs. The gradient norms are non-zero but very small (around 1e-6), and the model has not converged to the desired accuracy.

The team considers switching to a different optimizer or increasing the learning rate. However, an ML engineer suggests investigating whether the model is stuck in a saddle point.

Which TWO strategies would help the model escape a saddle point? (Select TWO)

A) Introduce a small amount of noise in the parameter updates (e.g., using stochastic gradient noise) to perturb the trajectory away from the saddle. B) Use an optimizer with momentum (e.g., Adam or SGD with momentum), which builds velocity and can "roll through" flat regions more effectively than vanilla SGD. C) Set the learning rate to zero for 10 epochs to allow the model to settle out of the saddle. D) Increase the batch size to include all data points, which increases the gradient precision and helps identify the saddle. E) Replace the loss function with a custom function that ignores saddle points.

Click to reveal Answer & Explanation

Correct Answer: A and B

Correct Options Analysis

  • A is Correct: In a saddle point, gradients are close to zero but the point is not a local optimum. Adding a small stochastic noise (e.g., gradient noise from mini-batch sampling or explicit noise injection) perturbs the parameter trajectory enough to escape the flat region and continue descending.
  • B is Correct: Optimizers with momentum (e.g., Adam, RMSprop, SGD with momentum) accumulate velocity over steps. Even when gradients are tiny at a saddle point, momentum carries the optimizer through flat regions, preventing it from getting stuck.

Distractor Analysis

  • C is Incorrect: Setting the learning rate to zero means no parameter updates occur, which is the opposite of escaping a saddle point. The model will remain stuck indefinitely.
  • D is Incorrect: Larger batch sizes reduce gradient variance, but in a saddle point the issue is small gradient magnitudes, not noisy estimates. Full-batch gradient descent is actually more likely to get stuck in saddle points because there is no stochasticity to perturb the trajectory.
  • E is Incorrect: Loss functions define the optimization landscape; they cannot be arbitrarily replaced to "ignore" saddle points. Saddle points are a property of the loss surface geometry, not the loss choice.

Key Takeaway: Saddle points are characterized by small, non-zero gradients and extremely slow progress—use momentum-based optimizers and stochastic gradient noise (small batch mini-batches) to escape.