Skip to content

2.2

Task 2.2: Train and Refine Models – MLA-C01 Study Notes

Key Concepts Summary

Training process elements: An epoch is one full pass over the training dataset. Steps (or iterations) = number of weight updates per epoch. Batch size is the number of samples processed before a weight update. Larger batches improve stability/gradient estimates but increase memory use and can hurt generalization; smaller batches add noise that can help escape local minima.

Reducing training time: Early stopping monitors a validation metric and halts when it stops improving (saves compute + prevents overfitting). Distributed training (data/model parallelism) via SageMaker distributes work across instances. Faster data access (FSx for Lustre > EFS) reduces I/O wait. Transfer learning/fine-tuning a pre-trained model (JumpStart or Bedrock) avoids training from scratch.

Model size factors & reduction: Number of layers/parameters, precision (FP32 → FP16/INT8), feature count, embedding dimensions. Techniques: pruning, quantization, knowledge distillation, feature selection, altering data types, compression.

Improving performance & preventing overfitting/underfitting/catastrophic forgetting:

  • Regularization: L1 (Lasso – sparsity/feature selection), L2 (Ridge – weight decay), dropout, early stopping.
  • Hyperparameter effects: learning rate (too high = diverge; too low = slow), number of trees/layers/nodes, optimizer (SGD, Adam, etc.).
  • Ensembles: bagging, boosting (XGBoost, CatBoost), stacking – combine models for better accuracy/robustness (common in fraud, medical, CV/NLP).

Hyperparameter tuning (SageMaker AMT):

  • Grid search: exhaustive categorical combinations.
  • Random search: random samples from ranges.
  • Bayesian optimization: sequential regression to pick promising next points.
  • Hyperband: adaptive resource allocation + early stopping of poor configs (median/final metrics). Works with built-in algorithms, script mode, or custom containers. Integrate automated HPO easily.

SageMaker-specific:

  • Built-in algorithms + common libraries (scikit-learn, XGBoost, etc.).
  • Script mode: bring your own TensorFlow/PyTorch/etc. training script.
  • JumpStart / Bedrock: fine-tune pre-trained models on custom datasets.
  • Model Registry + ML Lineage Tracking: version models, attach metadata, approval workflows, audit/reproducibility, CI/CD.
  • Studio environment (managed, EFS-backed) vs classic Notebook instances.
  • Bring external models into SageMaker (script mode or custom containers) for AMT, Registry, deployment.

Exam traps from official review:

  • Parameters (learned from data: weights/biases) vs hyperparameters (set by you: epochs, lr, batch size, #layers/trees).
  • Early stopping stops on plateauing validation metric to cut cost + overfitting.
  • FSx for Lustre accelerates S3 → SageMaker data serving (high throughput); EFS works but slower; PCA reduces size/performance but does not speed job startup/I/O.
  • Redshift ML for in-database training/inference when data already lives in Redshift (avoids S3 export + cheaper/faster local preds). Streaming ingestion cannot feed SageMaker inference.
  • Hyperband/AMT auto-stops poor jobs.
  • Ensembles via single training/tuning job + single endpoint reduce cost/ops.
  • Transfer learning / JumpStart for “train without starting from scratch”.
  • Always clean up (endpoints, training jobs, FSx, Studio resources) – lingering resources are a common cost gotcha.

Low-Cost Hands-on Exercises

All exercises use the smallest viable instances (ml.t3.medium or ml.m5.large only when required), Spot training where possible, tiny public/sample datasets, short max runtimes, and aggressive cleanup. Total expected cost per full run-through: well under $5 if you delete everything immediately (monitor Billing console). Prefer SageMaker Studio free-tier-eligible or very short sessions if available in your account; otherwise classic Notebook + local mode simulation where feasible. Never leave endpoints or FSx running.

Exercise 1: Built-in Algorithm Training + Early Stopping + Basic Hyperparameters (Core training loop, epochs/batch/early stop, regularization concepts)

Goal: Internalize epoch/step/batch, early stopping to cut cost/overfit, L1/L2 effects. Uses SageMaker XGBoost built-in (fast, cheap).

Steps:

  1. Create a SageMaker Notebook instance (ml.t3.medium, 5 GB volume) or open Studio. In a notebook cell: python import sagemaker, boto3, os from sagemaker.inputs import TrainingInput from sagemaker.estimator import Estimator sess = sagemaker.Session() role = sagemaker.get_execution_role() bucket = sess.default_bucket() prefix = "mla-c01-ex1"

  2. Download a tiny dataset (e.g., UCI Iris or a 1k-row subset of abalone/Boston housing CSV). Upload to s3://{bucket}/{prefix}/train/ and /validation/. Split 80/20.

  3. Create XGBoost estimator (built-in): python xgb = Estimator( image_uri=sagemaker.image_uris.retrieve("xgboost", sess.boto_region_name, "1.5-1"), role=role, instance_count=1, instance_type="ml.m5.large", # smallest practical output_path=f"s3://{bucket}/{prefix}/output", sagemaker_session=sess, hyperparameters={ "objective": "reg:squarederror", # or binary:logistic "num_round": 50, # epochs equivalent "max_depth": 3, "eta": 0.2, # learning rate "subsample": 0.8, "colsample_bytree": 0.8, "alpha": 0.1, # L1 "lambda": 1.0, # L2 / weight decay "early_stopping_rounds": 10 } )

  4. Fit with early stopping enabled (validation metric monitored): python train_input = TrainingInput(f"s3://{bucket}/{prefix}/train", content_type="text/csv") val_input = TrainingInput(f"s3://{bucket}/{prefix}/validation", content_type="text/csv") xgb.fit({"train": train_input, "validation": val_input}, wait=True)

  5. Observe CloudWatch logs / job metrics: note when early stopping triggers (validation loss plateaus or rises after ~epoch 15–20 equivalent). Compare a run without early_stopping_rounds (higher cost, potential overfit).

  6. Experiment: change batch-equivalent (subsample), add stronger L1 (alpha=1.0) → sparser model; stronger L2. Record final validation metric and job duration/billable seconds.
  7. Cleanup (critical): xgb.delete_endpoint() if any; delete Notebook/Studio resources; empty S3 prefix; confirm no training jobs left in console.

Expected results: Training job completes in <5–10 min. Logs show early stopping cutting rounds (e.g., stops at round 18/50). Stronger regularization increases training loss slightly but stabilizes/improves validation loss (prevents overfit). Job billable time visibly lower with early stop. Model artifact appears in S3 output.

Exam traps/tips linked: Exactly matches the “validation loss rises from 15th epoch → use early stopping” question. Distinguishes L1 (sparsity) vs L2. Shows hyperparameters you set vs learned parameters. Demonstrates cost control via early stop – common exam theme. PCA or other size reduction would not have helped startup time here.

Exercise 2: SageMaker Automatic Model Tuning (AMT) – Random vs Bayesian vs Hyperband + Script Mode Intro

Goal: Master AMT strategies, script mode (bring external framework code), hyperparameter effects, automated HPO integration. Tiny PyTorch or scikit-learn script.

Steps:

  1. Reuse or create another short-lived ml.t3.medium Notebook/Studio.
  2. Prepare the same tiny CSV dataset in S3.
  3. Write a minimal training script train.py (script mode) that accepts hyperparameters via argparse or SM env vars (e.g., learning rate, batch size, epochs, dropout rate). Use PyTorch or even scikit-learn RandomForest for extreme cheapness. Include simple L2/weight decay or dropout. Save model to /opt/ml/model.

  4. Create a PyTorch (or SKLearn) Estimator pointing to your script: python from sagemaker.pytorch import PyTorch # or SKLearn estimator = PyTorch( entry_point="train.py", role=role, instance_count=1, instance_type="ml.m5.large", framework_version="1.12", # or latest small py_version="py38", hyperparameters={"epochs": 10, "batch-size": 32, "lr": 0.01, "dropout": 0.2}, output_path=f"s3://{bucket}/{prefix}/amt-output", # enable spot for extra savings: use_spot_instances=True, max_wait=3600, max_run=1800 )

  5. Define HyperparameterTuner: python from sagemaker.tuner import ContinuousParameter, IntegerParameter, HyperparameterTuner hyperparameter_ranges = { "lr": ContinuousParameter(1e-4, 1e-1), "batch-size": IntegerParameter(16, 64), "dropout": ContinuousParameter(0.1, 0.5), "epochs": IntegerParameter(5, 20) } objective_metric_name = "validation:loss" # or accuracy; ensure script emits it via print or SM metrics tuner = HyperparameterTuner( estimator, objective_metric_name, hyperparameter_ranges, max_jobs=6, max_parallel_jobs=2, # keep tiny! strategy="Bayesian", # also try "Random", "Hyperband" early_stopping_type="Auto" # Hyperband benefits ) tuner.fit({"train": train_input, "validation": val_input})

  6. After completion, inspect tuner.best_estimator(), best hyperparameters, and analytics dataframe (which configs were stopped early by Hyperband). Compare wall-clock and cost of Bayesian vs a quick Random run.

  7. Optional cheap extension: register the best model in Model Registry: python model_package = tuner.best_estimator().register( content_types=["text/csv"], response_types=["text/csv"], inference_instances=["ml.t2.medium"], transform_instances=["ml.m5.large"], model_package_group_name="mla-c01-ex2-group" )

  8. Cleanup: Delete tuner jobs, any endpoints, Model Package Group if desired, Notebook, S3 data. Confirm Billing.

Expected results: 4–6 short training jobs finish. Bayesian finds better configs faster than pure random. Hyperband stops under-performers early (logs show this). Best model has lower validation loss; Registry shows versioned package with metadata. Total runtime <20–30 min billable.

Exam traps/tips linked: Directly practices grid/random/Bayesian/Hyperband differences and AMT with script mode/custom algorithms. Shows automated HPO integration and early-stop inside tuner (compute savings). Model Registry for “manage versions for repeatability and audits”. Hyperparameters (lr, batch, dropout, #epochs) vs learned weights. Transfer/fine-tune mindset: you could swap the script for a JumpStart pre-trained backbone later.

Exercise 3: JumpStart Fine-Tuning + Model Size Awareness + Simple Ensemble Idea (Transfer learning, pre-trained, size reduction, ensembles)

Goal: Fine-tune pre-trained model without scratch training, observe size factors, touch ensemble concept, Registry. Uses JumpStart (very low effort/cost for small models).

Steps:

  1. In SageMaker Studio (preferred, managed) or Notebook, open JumpStart.
  2. Pick a tiny pre-trained model (e.g., small Hugging Face text classifier, or tabular JumpStart model, or lightweight CV model). Avoid large LLMs.
  3. Fine-tune on your tiny custom dataset (upload CSV/JSON to S3 or use JumpStart sample). Set few epochs (3–5), small batch, enable early stopping if exposed. Use ml.m5.large or Spot.
  4. After fine-tune, note model artifact size in S3. Experiment mentally or via code: change precision (if supported), prune features, or apply simple quantization post-training (TorchScript/ONNX conversion in a processing job – keep tiny).
  5. Deploy briefly to a serverless inference endpoint or ml.t2.medium for 2–3 test predictions, then immediately delete endpoint.
  6. Register the fine-tuned model in Model Registry (as in Ex 2). Use Lineage Tracking view in Studio to see data → train → model links.
  7. Cheap ensemble simulation: Train two weak models (different hyperparameters or one XGBoost + one linear) via separate short Estimators, then average predictions in a simple inference script (or note that SageMaker supports custom ensemble containers / multi-model endpoints). Observe improved robustness on a hold-out set.
  8. Cleanup: Delete JumpStart endpoints, fine-tuning jobs, Studio apps/kernels, S3 artifacts, Model Registry entries if temporary. Empty buckets.

Expected results: Fine-tuning finishes in minutes (pre-trained weights already know general patterns; only head/adaptation layers update). Model size smaller than from-scratch equivalent. Registry + lineage show versioning/audit trail. Simple average ensemble yields modest metric lift. Serverless or tiny endpoint incurs almost zero ongoing cost once deleted.

Exam traps/tips linked: Exactly the “fine-tune pre-trained / transfer learning instead of scratch” solution. JumpStart + Studio resources (projects, experiments, registry). Model size reduction techniques. Ensemble use-cases (fraud etc.) and single-job/single-endpoint deployment for cost. Catastrophic forgetting prevention via careful fine-tune + regularization. Ties back to Redshift ML alternative when data is already in warehouse (you could have exported tiny sample instead of full movement).

Additional Low-Cost Tips & General Exam Readiness

  • Always prefer Spot (use_spot_instances=True) + short max_run + early stopping + Hyperband.
  • Monitor via SageMaker Experiments / Trials (free metadata) and CloudWatch.
  • For distributed: only demo conceptually or with 2× tiny instances if needed; cost multiplies.
  • Outside SageMaker models: package as script mode or Docker (but Docker build is free locally; push only if testing).
  • After every session: check Cost Explorer filtered to SageMaker + S3 + FSx (if used) and set a billing alarm at $5.
  • Practice the Redshift ML path conceptually (or with a free-tier Redshift Serverless trial if available) – “data already in Redshift → in-DB ML” beats export-to-S3-to-SageMaker for cost/latency.

These three exercises cover virtually every knowledge/skill bullet and the video transcript traps while keeping spend minimal. Run them in order (Ex1 → Ex2 → Ex3) over 1–2 days, document metrics/hyperparams/logs, then delete everything. You will be exam-ready for Task 2.2. Good luck!