Domain 1 overview

Introduction

Original video transcript

All right, this is the first domain, and it's the one people rush past. Data work looks unglamorous, and everyone wants to get to the models, right? And then a model underperforms in ways no amount of tuning can fix. The problem started earlier in the data. That's what this domain covers. So, let's look at what this domain actually is. Data preparation for ML and AI is everything that turns raw data into something a model can learn from. And it breaks into three tasks. Let's walk through them. Sure, the first is collecting and storing data. That means pulling data from the sources where it lives and choosing storage that fits cost and access patterns. It also covers streams and diverse data types. The second is transformation, feature engineering, and preprocessing. So this is where you reshape raw data into features and turn text and images into embeddings. You also prepare documents for retrieval augmented generation or RAG applications. And the third is validating data quality and managing bias. You check that data is complete and clean, label it, and resolve class imbalance, and catch bias before it reaches a model. Three tasks and they're connected. You validate raw data as it arrives. You validate the transformation outputs, and you keep checking quality as the data moves downstream. Skip the checking and the problem surfaces later where it costs more. So why does this matter for the machine learning engineer? Because models are won or lost here before training ever starts. Exactly. Algorithm choice rarely makes up for features built on broken data. The engineer who can trace a model problem back to a data problem is the one who actually fixes it. Get the data right and everything downstream gets easier. Get it wrong and the failures are expensive and hard to trace. So, here's how the review works. Each task area gets a light pass over the AWS services that relate to it. And then a deeper review of the skills underneath. Some tasks also get a review together video where we work a scenario out loud, and the knowledge checks along the way tell you where to slow down. So that's what's covered in this domain raw data to model ready data in 3 connected pieces. Let's get started.

Core Principle: Models are won or lost in data preparation before training ever starts. Algorithm selection and hyperparameter tuning cannot compensate for features built on broken, biased, or unvalidated data.

The 3 Core Tasks

1. Collecting & Storing Data

   • Ingesting raw data from diverse sources (batch and streaming).
   • Selecting AWS storage solutions that align with access patterns, throughput requirements, and cost constraints.

2. Transformation, Feature Engineering, & Preprocessing

   • Structured Data: Reshaping raw tables into numerical features (encoding, scaling, handling outliers).
   • Unstructured Data: Converting text and images into dense vector embeddings.
   • RAG Workflows: Document chunking, metadata attachment, and vector indexing.

3. Validating Data Quality & Managing Bias

   • Checking data completeness, schema consistency, and cleanliness.
   • Managing dataset labeling and mitigating class imbalance/underrepresentation.
   • Detecting and resolving statistical bias before it reaches model training.

Key Pipeline Architectural Takeaway

• Continuous Validation Loop: Data validation is not a one-off step. Quality checks must occur at three distinct stages:
  1. At Ingestion: Validating raw data as it arrives.
  2. Post-Transformation: Validating features and embeddings before training or vector store ingestion.
  3. Downstream: Monitoring data quality continuously as pipelines move toward production.