Skill 2.2.7: Adjust fundamental hyperparameters (for example, epoch, steps, batch size)
Task 2.2: ML Model and Foundation Model (FM) Development - Train, fine-tune, and customize models for ML and AI solutions.
Core objective: Adjust hyperparameters to maximize model generalization while minimizing training time and cost. Hyperparameters are not computed; they are set by heuristic, searched by AMT, or tuned by early stopping/warm start.
Fundamental Hyperparameters: Definitions & Effects
| Hyperparameter | Definition | Increase effect | Decrease effect | Exam trap |
|---|---|---|---|---|
| Epoch | One full pass over training data | More learning, risk overfit, more compute | Underfit, faster | FM fine-tuning: 1–3 typical; more risks catastrophic forgetting |
| Batch size | Samples per weight update | Stable gradients, higher throughput, more memory, may need higher LR | Noisy gradients, regularization effect, less memory, slower, may need lower LR | Changing batch size changes steps/epoch; retune LR |
| Steps / iterations | Weight updates. Steps/epoch = ceil(samples ÷ batch size). Total steps = steps/epoch × epochs | More updates, more learning | Fewer updates, underfit | Distributed data parallel: global batch = per-device × instances; steps/epoch drops |
| Learning rate | Step size per update | Faster, risk divergence | Slower, risk local minima | Scale with batch size: linear or sqrt scaling |
| Max epochs | Cap on passes | Longer training | Early stop | Use with early stopping, not alone |
| Regularization (dropout, weight decay) | Penalizes complexity | Less overfit, risk underfit | More overfit | Tune after LR/batch/epochs |
| LoRA rank / alpha / dropout | FM parameter-efficient tuning | More capacity, more memory | Less forgetting, less capacity | High rank can overfit and increase cost |
Gotcha: Epochs, steps, and batch size are mathematically linked. Changing batch size or distributed instance count changes global batch size and steps/epoch. Always recalculate steps and retune learning rate.
Hyperparameter Tuning (SageMaker AMT)
| Concept | Detail | Exam trap |
|---|---|---|
| Objective metric | Single metric to optimize | Must match direction: minimize for error (MSE, log loss), maximize for score (accuracy, F1, AUC). Wrong direction = worse model |
| Search strategies | Bayesian (sample-efficient), Random (parallel, high-dim), Hyperband (early stopping, many configs), Grid (small discrete) | Hyperband needs early stopping; Random is embarrassingly parallel |
| Ranges | Continuous, Integer, Categorical | Define bounds from known good values; too wide wastes budget |
| Completion criteria | max_jobs, max_parallel_jobs, max_runtime | Controls cost and time. Too high parallel = budget exhaustion |
| Early stopping | Stops underperforming trials | Use for overfitting/long training. Requires validation metric |
| Warm start | Reuses previous tuning | Types: IdenticalDataAndAlgorithm, TransferLearning, SameObjectiveFunction |
Gotcha: AMT extracts the objective metric from CloudWatch Logs via regex. If the training script does not log it correctly, tuning fails.
Training Time Reduction & Distributed Training
| Technique | Use when | Hyperparameter impact | Avoid when |
|---|---|---|---|
| Early stopping | Validation metric plateaus | Reduces effective epochs | Metric is noisy |
| Data parallel | Model fits on one device, large data | Global batch = per-device × instances; scale LR | Small data; communication overhead > savings |
| Model parallel | Model too large for one device | Batch size unchanged | Model fits on one device |
| Mixed precision | Speed/memory needed | May need LR scaling | Numerical instability |
| Gradient accumulation | Simulate large batch without memory | Effective batch = micro-batch × accumulation steps | Exact batch-norm statistics required |
| Checkpointing | Spot instances, long training | None | Restart cost is low |
Gotcha: In data parallelism, increasing instances increases global batch size. If you do not scale learning rate, convergence can degrade. Steps/epoch = ceil(samples ÷ global_batch).
Generalization: Overfitting, Underfitting, Catastrophic Forgetting
| Symptom | Diagnosis | Hyperparameter adjustments |
|---|---|---|
| Training loss low, validation loss high | Overfitting | ↓ epochs, ↑ dropout/weight decay, early stopping, ↑ data, ↓ model capacity |
| Both training and validation loss high | Underfitting | ↑ epochs, ↑ learning rate, ↑ model capacity, ↓ regularization |
| Fine-tuned FM loses general ability | Catastrophic forgetting | ↓ epochs, ↓ learning rate, LoRA/PEFT, mix general data, freeze layers |
| Training loss unstable / NaN | LR too high | ↓ LR, ↑ batch size, gradient clipping, warmup |
| Training too slow | Batch size too small, no distributed | ↑ batch size, data parallel, mixed precision |
Gotcha: For FM fine-tuning, more epochs is not better. Catastrophic forgetting occurs when the model over-specializes to narrow domain data. Prefer 1–3 epochs, low LR, and parameter-efficient fine-tuning.
FM Customization Hyperparameters
| Technique | Key hyperparameters | When to use | Anti-pattern |
|---|---|---|---|
| Prompt engineering | temperature, top-p, max tokens | First attempt, no training | Hardcoding prompts without versioning |
| Supervised fine-tuning | epoch count, batch size, learning rate, warmup | Labeled input-output pairs | Too many epochs, high LR → forgetting |
| Reinforcement fine-tuning | reward function, epochs, LR | Reward signal available | Sparse or noisy rewards |
| Distillation | teacher model, prompt set, student size | Smaller/faster student | No teacher quality check |
| LoRA / PEFT | rank, alpha, dropout | Limited compute, avoid forgetting | Rank too high → overfit/memory |
| Retrieval optimization | chunk size, overlap, embedding model, top-k | RAG relevance issues | Tuning generator when retrieval fails |
Gotcha: If retrieved passages are irrelevant, fix chunking and embedding model first. If passages are correct but answer is poor, fix prompt, then fine-tune. Do not fine-tune to fix retrieval.
Decision Matrix: Manual vs AMT vs Early Stopping
| Scenario | Recommended approach | Why |
|---|---|---|
| Few hyperparameters, strong intuition | Manual tuning | Fast, low cost |
| Large search space, expensive trials | AMT Bayesian | Sample-efficient |
| Many configs, parallel compute | AMT Random | Embarrassingly parallel |
| Many configs, early stopping | AMT Hyperband | Stops poor trials early |
| Overfitting during training | Early stopping | Saves compute, prevents overfit |
| Similar problem tuned before | Warm start | Reuses knowledge |
| Model too large for one GPU | Model parallel | Fits model across devices |
| Large dataset, model fits | Data parallel | Scales throughput |
Security, Compliance & Operations (Relevant Guardrails)
- IAM role: Training/tuning jobs assume an IAM role. Grant least privilege to S3 input/output, ECR if custom container, CloudWatch Logs.
- Encryption: Use KMS for data at rest (S3, EBS volumes), TLS for data in transit. Use VPC endpoints for private access.
- Logging: Training metrics go to CloudWatch. Objective metric regex parses logs for AMT.
- Cost controls: Set max jobs, max parallel jobs, max runtime. Use spot instances with checkpointing. Right-size instances.
Gotcha: AMT objective metric is extracted from CloudWatch Logs. Ensure the training script emits the metric in the expected format.
Quick Reference: Hyperparameter Adjustment Cheat Sheet
| Goal | Adjust |
|---|---|
| Reduce overfitting | ↓ epochs, ↑ dropout, ↑ weight decay, early stopping, ↑ data |
| Reduce underfitting | ↑ epochs, ↑ learning rate, ↑ model capacity, ↓ regularization |
| Speed up training | ↑ batch size, data parallel, mixed precision, ↑ instance count |
| Stabilize training | ↓ learning rate, ↑ batch size, gradient clipping, warmup |
| Reduce memory | ↓ batch size, gradient accumulation, mixed precision, model parallel |
| Prevent catastrophic forgetting | ↓ epochs, ↓ learning rate, LoRA, mix general data |
| Improve RAG relevance | Adjust chunk size/overlap, better embedding model, tune top-k |
| Optimize AMT | Set objective metric direction, use Bayesian/Hyperband, early stopping, warm start |
Final Gotcha: Hyperparameters are not independent. Always consider interactions: learning rate ↔ batch size, epochs ↔ steps, distributed instances ↔ global batch size. Tune the most impactful first: learning rate, batch size, epochs, then regularization.