Skip to content

Domain 2 Review

Introduction

Original video transcript

OK, so most engineers have trained a model in a notebook, so model development feels familiar, right? And then a job asks a harder question, not whether you can train a model, whether you can choose the right one and you can prove it works. This domain, ML model and foundation model development covers how you choose, train, and evaluate models from traditional ML to foundation models, and it breaks into three tasks. Let's walk through them. The first is choosing appropriate modeling approaches. That means weighing traditional ML models against foundation models and custom builds against managed options. It also means selecting rag when a use case needs them. The second is training, fine tuning, and customizing models. This is where you can figure training, tune hyperparametters, prevent overfitting, and apply customization techniques like prompt engineering and fine tuning. And the third is analyzing and evaluating performance. You set baselines, detect drift, run reproducible experiments, and evaluate traditional ML models and generative AI outputs with the right metrics. 3 tasks, and they build on each other. The modeling choice shapes the training, and the evaluation tells you whether both actually worked. Skip that last step and you're deploying a guess. So what does this matter for the machine learning engineer? Because these decisions are what separate an engineer from someone who just runs scripts. Exactly. Anyone can start a training job. The engineer knows why this model, why these trade-offs, and how to back the result with evidence instead of a feeling. Make those choices deliberately and you ship models the business can trust. Make them casually and you ship expensive guesses. See, here's how the review works. Each task gets a light pass over the AWS services that relate to it, and then a deeper review of the skills underneath. Some tasks also get a review together video where we work a scenario out loud and the knowledge texts along the way tell you where to slow down. So that's what's covered in this domain choosing, training, and proving models in three connected pieces. Let's get started.

Core Principle: Anyone can launch a training script. An engineer knows why a specific model architecture was chosen, understands the architectural trade-offs, and backs model performance with quantitative evidence rather than guesswork.

The 3 Core Tasks

  1. Choosing Appropriate Modeling Approaches

    • Traditional ML vs. Foundation Models (FMs): Selecting tabular/classical algorithms vs. deep learning or GenAI foundation models based on data and task requirements.
    • Managed Services vs. Custom Builds: Weighing fully managed options (e.g., Amazon Bedrock, SageMaker JumpStart) against custom training on SageMaker instances.
    • Architectural Selection: Identifying when a scenario requires Retrieval-Augmented Generation (RAG) vs. direct inference or fine-tuning.

  2. Training, Customizing, & Fine-Tuning Models

    • Training Operations: Configuring instance types, distributed training, hyperparameter tuning (HPO), and regularizing to prevent overfitting.
    • GenAI Customization: Applying prompt engineering techniques first, followed by Parameter-Efficient Fine-Tuning (PEFT/LoRA) or Supervised Fine-Tuning (SFT) when prompting alone is insufficient.

  3. Analyzing & Evaluating Performance

    • Baselines & Tracking: Establishing performance baselines, logging metrics, and running reproducible experiments (e.g., SageMaker Experiments).
    • Traditional Metrics: Evaluating tabular models using Precision, Recall, F1, ROC-AUC, RMSE, etc.
    • GenAI & FM Metrics: Evaluating foundation model outputs using specialized benchmarks, human-in-the-loop (SageMaker Clarify/Ground Truth), and automated LLM-as-a-judge metrics (e.g., ROUGE, BLEU, semantic similarity).

Key Pipeline Architectural Takeaway

  • Sequential Dependency Loop: Domain 2 decisions build directly on one another:
\[\text{Model Choice} \longrightarrow \text{Training / Customization} \longrightarrow \text{Performance Evaluation}\]
  • Skipping or rushing evaluation means deploying an unverified guess.
  • Evaluation metrics determine whether the modeling approach and training configuration actually succeeded before pushing to deployment.