Skill 2.2.5: Prevent model overfitting, underfitting, and catastrophic forgetting
Task 2.2: ML Model and Foundation Model (FM) Development - Train, fine-tune, and customize models for ML and AI solutions
Core objective: Maximize generalization on unseen data. Three failure modes: underfitting (high bias), overfitting (high variance), catastrophic forgetting (fine-tuning erases prior knowledge).
Failure Modes: Signatures & Remedies
| Failure | Training vs Validation | Root Cause | Primary Fixes |
|---|---|---|---|
| Underfitting | Both poor | Model too simple, weak features, over-regularized, too few epochs, LR too low | Increase capacity, add features, reduce regularization, train longer, tune LR, better architecture |
| Overfitting | Train strong, validation weak | Memorizes noise, too complex, too many epochs, small data, leakage | More data, augmentation, L1/L2/dropout, early stopping, reduce complexity, cross-validation, ensemble |
| Catastrophic Forgetting | General benchmark drops after fine-tune | Narrow fine-tune overwrites base weights | PEFT (LoRA/adapters), mix general data, lower LR, fewer epochs, rehearsal, regularization |
Detection & Metrics
- Loss curves: Overfit = gap widens; underfit = both high; ideal = both converge.
- Validation set: Use for tuning/early stopping. Test set only for final evaluation.
- Objective metric direction: Minimize (MSE, log loss) vs maximize (AUC, F1, accuracy). Wrong direction optimizes a worse model.
- Catastrophic forgetting: Evaluate on general benchmark before/after fine-tuning. Drop = forgetting.
- AWS monitoring: SageMaker Debugger, CloudWatch Logs/Metrics, Model Monitor for drift, Bedrock model evaluation.
Hyperparameter Tuning & Training Time
| Technique | What it does | When to use | Gotcha |
|---|---|---|---|
| AMT | Searches ranges across many training jobs | HPO when no closed-form solution | Needs objective metric + direction; not within one run |
| Early stopping | Ends run when validation metric stops improving | Prevent overfitting, save cost | Needs validation metric; can stop too early if noisy |
| Warm start | Reuses prior tuning knowledge | New search related to old | Can bias search |
| Data parallelism | Splits batch across model replicas | Model fits one device but training slow | Communication overhead can negate gains |
| Model parallelism | Splits model across devices | Model too large for one device | Complex; communication heavy |
Regularization & Generalization
| Technique | Fights | AWS Context |
|---|---|---|
| L1/L2, weight decay | Overfitting | Built-in algorithm hyperparameters; script mode |
| Dropout | Overfitting | Framework-level; script mode |
| Data augmentation | Overfitting | Built-in algorithms; custom scripts |
| Cross-validation | Overfitting/underfitting | SageMaker AMT; training jobs |
| Ensembling | Both | Multi-model endpoints for cost; performance gain |
| Batch normalization | Underfitting/overfitting | Framework-level |
| More data / better features | Both | S3 data sources; Feature Store |
| Reduce model complexity | Overfitting | Smaller model; PEFT |
Catastrophic Forgetting in FM Customization
- Definition: Fine-tuning on narrow domain degrades general capabilities.
- Causes: Full fine-tuning, high LR, many epochs, small narrow dataset, no rehearsal.
- Prevention:
- PEFT: LoRA, adapters, prompt tuning; freeze base weights, train small params. AWS: SageMaker JumpStart, Bedrock.
- Data mixing: Blend general/instruction data with domain data.
- Hyperparameters: Lower LR, fewer epochs, early stopping on general eval.
- Rehearsal: Retain some original data.
- Regularization: Elastic weight consolidation (concept), L2 to base weights.
- Detection: Pre/post fine-tune general benchmark; regression on held-out general tasks.
- AWS: Bedrock custom models (supervised fine-tuning, reinforcement fine-tuning, distillation), JumpStart fine-tuning, SageMaker training jobs.
AWS Service Mapping
| Service | Role in Failure Prevention |
|---|---|
| SageMaker AI Training Jobs | Built-in algorithms, script mode, custom containers; run training |
| SageMaker AI AMT | HPO; finds hyperparameters that generalize |
| SageMaker AI Early Stopping | Stops overfitting runs; reduces cost |
| SageMaker AI Debugger | Monitors loss curves, tensors; detects overfitting/underfitting |
| SageMaker AI Model Monitor | Detects data/model drift in production |
| SageMaker JumpStart | Pre-trained models; PEFT/fine-tuning |
| Amazon Bedrock | FM customization: fine-tuning, continued pre-training, distillation; evaluate forgetting |
| Bedrock Knowledge Bases | Retrieval optimization: embedding model, chunking, ranking |
| Amazon CloudWatch | Logs/metrics from training jobs |
Security & Compliance
- IAM roles: SageMaker training jobs assume execution role with least privilege to S3, ECR, KMS, CloudWatch.
- Encryption at rest: KMS for S3 training data, model artifacts, EBS volumes, Bedrock custom models.
- Encryption in transit: Inter-container traffic encryption; TLS for API calls.
- Network isolation: SageMaker training jobs can run without internet; use VPC endpoints (PrivateLink) for S3, ECR, CloudWatch, Bedrock.
- Data privacy: Fine-tuning data may be sensitive; use VPC, KMS, IAM, CloudTrail; avoid public S3 buckets.
- Gotcha: Network isolation disables outbound internet; must use VPC endpoints for dependencies.
Anti-Patterns & Exam Gotchas
Hyperparameters are not computed. They are searched via AMT across many training jobs.
Early stopping vs AMT: Early stopping ends one run; AMT searches across runs. Do not confuse.
Test set leakage: Never use test set for early stopping or tuning; use validation set.
Metric direction: Minimize MSE vs maximize AUC. Wrong direction optimizes a worse model.
Catastrophic forgetting ≠ overfitting. Overfitting is train/validation gap; forgetting is loss of prior general capability after fine-tuning.
More epochs ≠ better. Can cause overfitting.
More complex model ≠ better. Can cause overfitting; underfitting needs more capacity.
Multi-model endpoints: Share memory/container; first request slower; not for strict latency.
Fine-tuning on narrow data without general data: High risk of catastrophic forgetting.
No monitoring: Use CloudWatch/Debugger/Model Monitor to detect drift and degradation.
Decision Matrix
| Symptom | Likely Issue | First Action |
|---|---|---|
| Train and validation both poor | Underfitting | Increase capacity/features, reduce regularization, train longer |
| Train strong, validation weak | Overfitting | More data, augmentation, regularization, early stopping, reduce complexity |
| General benchmark drops after fine-tune | Catastrophic forgetting | PEFT, mix general data, lower LR/epochs, rehearsal |
| Validation metric plateaus, train improves | Overfitting | Early stopping |
| Training too slow, model fits one device | Compute bottleneck | Data parallelism |
| Model too large for one device | Memory bottleneck | Model parallelism |