Task 1.2 - Perform data transformation, feature engineering, and pre-processing

Overview

Task 1.2, Perform data transformation, feature engineering, and pre-processing, is where raw data becomes model input.

For traditional models that means numeric features on comparable scales. For generative AI it means text split into retrievable pieces, converted into embeddings, or arranged into the paired examples that customizing a foundation model requires. The same toolset serves both, which is why keeping the goal clear matters more than knowing any single transform.

As you review these lessons for Task 1.2, check that you understand how to do the following.

Transform data by using AWS tools (for example, AWS Glue, AWS Glue DataBrew, Spark on Amazon EMR, SageMaker Data Wrangler).
Create and manage features by using AWS tools (for example, SageMaker Feature Store).
Transform streaming data (for example, by using AWS Lambda, Spark).
Perform feature engineering (for example, scaling, standardization, feature splitting, binning, log transformation, normalization).
Configure and use embedding models to transform text and image data into numerical representations.
Apply advanced text pre-processing techniques (for example, tokenization, domain-specific augmentation).
Prepare documents for Retrieval Augmented Generation (RAG) applications (for example, chunking strategies, metadata extraction).
Mask, redact, and anonymize data.
Prepare data for FM fine-tuning, continuous pre-training, and model distillation.

AWS services overview

The services here divide by how the transformation runs: on a schedule over a batch, interactively while you inspect the data, or continuously as records arrive. Review each service to learn more.

AWS Service	Description
Amazon Bedrock	Access to embedding models that turn text and images into numerical representations, and to the customization paths fine-tuning and distillation use.
Amazon Bedrock Knowledge Bases	Manages the ingestion path for retrieval: splits documents into chunks, converts them to embeddings, and writes them to a vector store.
AWS Glue	Serverless extract, transform, and load jobs, most commonly on Apache Spark, running on a schedule, on demand, or on an event.
AWS Glue DataBrew	Visual data preparation. Steps are assembled into a recipe without code, and a profile job measures the dataset and writes out what it found.
Amazon EMR	Managed clusters running Apache Spark, for transformations needing cluster scale or a framework Glue does not host.
Amazon SageMaker Data Wrangler	An interactive, low-code surface in Amazon SageMaker Canvas for bringing in a dataset, exploring it, and reshaping it, including transforms that rebalance it.
Amazon SageMaker Feature Store	Stores and serves engineered features, so the definition used in training is the definition served at inference.
Amazon Comprehend	Detects personally identifiable information in English or Spanish text so it can be masked or redacted.
AWS Lambda	Event-driven compute that transforms records as they arrive, per batch by default, or across a fixed time window when the event source is configured for one.
Amazon Macie	Finds and reports sensitive data across an Amazon S3 estate, which is how the places needing masking are located before any transformation runs.

Knowledge Check

A team needs to inspect an unfamiliar dataset, see how complete and consistent each column is, and decide what preparation it needs before writing any transformation code. Which option fits that step? (Choose 1)

A. Convert the dataset into embeddings with an embedding model
B. Run an AWS Glue DataBrew profile job over the dataset
C. Load the dataset into Amazon SageMaker Feature Store
D. Deploy an AWS Lambda function to transform each record

Answer & Explanation

Correct Answer: B. Run an AWS Glue DataBrew profile job over the dataset

Explanation:
AWS Glue DataBrew is a visual data preparation tool that includes a data profiling feature. Running a DataBrew profile job automatically analyzes a dataset to generate column-level statistical reports, identifying missing values, data distributions, completeness, and consistency—all without writing any custom code.

Why the other options are incorrect: * A. Convert the dataset into embeddings with an embedding model: Embedding models transform unstructured text or images into vector representations for retrieval/semantic search, not for evaluating column consistency or profiling tabular data. * C. Load the dataset into Amazon SageMaker Feature Store: SageMaker Feature Store is designed to store, manage, and serve clean, pre-engineered features for model training and inference rather than exploring raw, unfamiliar datasets. * D. Deploy an AWS Lambda function to transform each record: Lambda executes custom event-driven transformation code. Writing and deploying transformation code is premature when the team has not yet inspected the dataset to determine what transformations are required.

AWS skills overview

In this lesson, you review the skills behind Task 1.2 as a light refresher. Each task groups a handful of related skills into a few topics. Read through them and notice which ones already feel familiar, and which ones feel shaky enough that you want to come back to them before exam day.

Transforming batch and streaming data

In practice, this group of skills lets the machine learning engineer choose a transformation tool that matches how the data arrives, and keep the batch path and the streaming path consistent with each other.

Be comfortable doing the following:

Transforming data with the managed AWS tools, and recognizing what each one is for: serverless scheduled jobs, visual recipe building, cluster-scale processing, and interactive preparation.
Transforming streaming data, and matching the tool to the window the requirement needs: event-driven compute for per-record work and for fixed non-overlapping windows, a stream processing framework for sliding windows and ordering across them.
Recognizing when a transformation needs to see more than one record at a time, because that requirement decides the tool.

That last point is the one worth holding onto. A transformation that treats each record independently, such as parsing a field or masking a value, runs comfortably in event-driven compute. A transformation that spans records needs somewhere to hold state, and the shape of the window decides how much tool you need. Event-driven compute can carry state across a fixed, non-overlapping time window when the event source is configured for one. A continuously sliding window, an ordering guarantee across that window, or state that has to outlive a fixed interval is what a dedicated stream processor is built for. The question to ask is not whether the tool can be stateful at all, but whether the window shape the requirement describes is one that tool can hold.

Knowledge Check

A streaming transformation must maintain a continuously sliding average over the last several minutes of records. Which requirement does that place on the tool? (Choose 1)

A. It must run on a fixed schedule
B. It must hold state across a continuously sliding window
C. It must be invoked by an event
D. It must offer a visual interface for building steps

Answer & Explanation

Correct Answer: B. It must hold state across a continuously sliding window

Explanation:
Calculating rolling aggregates (such as a moving average, sum, or count) over a sliding time frame requires stateful stream processing. The processing engine (e.g., Apache Flink or Amazon Managed Service for Apache Flink) must maintain intermediate state in memory or storage across incoming events. This allows it to continuously incorporate new records into the calculation while evicting records that pass outside the sliding time window boundary.

Why the other options are incorrect: * A. It must run on a fixed schedule: Fixed schedules are characteristic of batch processing (e.g., hourly Glue jobs), not real-time streaming window transformations. * C. It must be invoked by an event: Event-driven compute engines (such as AWS Lambda) are typically stateless and process records independently. Event invocation alone does not provide the state management mechanism needed to keep track of historical records in a sliding window. * D. It must offer a visual interface for building steps: A visual GUI (like AWS Glue DataBrew) is a developer interface feature, not a technical runtime capability required for streaming state management.

Engineering and managing features

In practice, this group of skills lets the machine learning engineer turn columns into features a model can learn from, then manage those features so they stay consistent over time.

Be comfortable doing the following:

Applying the common feature engineering transforms, and knowing what each one is for: scaling and normalization to put features on comparable ranges, standardization to center and rescale by spread, binning to group a continuous value into ranges, feature splitting to break a compound field into parts, and log transformation to compress a heavily skewed distribution.
Creating and managing features in a feature store, so a definition is written once and reused.
Recognizing that a transform fitted on training data must be applied, not refitted, on data seen later.

The refitting point causes real defects. A scaler learns its parameters from the data it is fitted on. If you fit it again on a later dataset, the same input value maps to a different feature value, and the model receives inputs it was never trained on. The transform is part of the model's contract, which is why the feature definition belongs somewhere both training and inference read from rather than being reimplemented in two places.

Knowledge Check

A scaling transform fitted during training is refitted on a later batch of data before inference. What is the consequence? (Choose 1)

A. The model automatically adjusts to the new scale
B. Nothing changes, because scaling is deterministic regardless of the data
C. The same input value now maps to a different feature value than the model was trained on
D. The feature becomes categorical rather than numeric

Answer & Explanation

Correct Answer: C. The same input value now maps to a different feature value than the model was trained on

Explanation:
Preprocessing transformers (such as StandardScaler or MinMaxScaler) compute parameters (like mean \(\mu\), standard deviation \(\sigma\), minimum, or maximum) during the .fit() phase on the training dataset. During inference, you must call .transform() using those exact saved parameters.

If you refit the scaler on a new inference batch, it recalculates those parameters based on the new batch's statistics. Consequently, an identical raw input value will map to a different normalized output value than it did during training. This introduces training-serving skew (or transformation skew), where the feature values passed to the model no longer align with the weights learned during training.

Why the other options are incorrect: * A. The model automatically adjusts to the new scale: Trained machine learning models have static, fixed mathematical weights and cannot automatically detect or adjust to shifts in feature scaling at inference time. * B. Nothing changes, because scaling is deterministic regardless of the data: Although the normalization formula itself is deterministic, the output depends entirely on dataset-level parameters calculated during .fit(). Changing the dataset used to fit the scaler changes those parameters. * D. The feature becomes categorical rather than numeric: Refitting a continuous numerical scaler changes the feature's numerical scale and distribution, but it remains a numeric data type.

Preparing text and documents for generative AI

In practice, this group of skills lets the machine learning engineer turn a body of documents into something a generative AI application can retrieve from.

Be comfortable doing the following:

Configuring and using embedding models to convert text and image data into numerical representations, so similarity can be measured.
Applying text pre-processing, including tokenization and domain-specific augmentation, so specialized vocabulary survives into the model input.
Preparing documents for Retrieval Augmented Generation, which means choosing a chunking strategy and extracting the metadata that makes a chunk filterable and attributable.

Chunking is the decision with the most leverage here. The managed ingestion path offers a small set of approaches: a fixed size, with a default size available if none is chosen; no chunking at all, when a file is already one logical unit; a hierarchical split that keeps a parent and child relationship; and a semantic split that breaks on meaning rather than on length alone. The tradeoff runs in both directions. Chunks that are too large retrieve a lot of irrelevant text along with the relevant sentence, and chunks that are too small lose the surrounding context that made the sentence meaningful. Metadata extraction is what lets you narrow retrieval to the right subset before similarity is even considered, and it is what lets the application tell the user where an answer came from.

Knowledge Check

Retrieval returns passages that contain the right sentence buried in a large amount of unrelated text. Which chunking adjustment addresses that directly? (Choose 1)

A. Store the chunks in a relational database instead
B. Reduce the chunk size, or split on meaning rather than on length
C. Increase the chunk size so more context is retrieved
D. Remove metadata from the chunks

Answer & Explanation

Correct Answer: B. Reduce the chunk size, or split on meaning rather than on length

Explanation:
When retrieved passages contain the target information buried within excessive unrelated text, the chunk size is too large or arbitrary. This dilutes the vector embedding's semantic signal and introduces context-window noise during generation (the "lost in the middle" effect). Reducing the chunk size or using semantic/hierarchical chunking (splitting based on topic shifts, natural paragraph breaks, or document structure) ensures each chunk is precise, focused, and coherent.

Why the other options are incorrect: * A. Store the chunks in a relational database instead: The underlying database type (e.g., PostgreSQL with pgvector vs. OpenSearch) governs storage and query execution, not the content granularity of individual text chunks. * C. Increase the chunk size so more context is retrieved: Increasing chunk size worsens the problem by pulling in even more surrounding, non-relevant text alongside the target sentence. * D. Remove metadata from the chunks: Metadata (such as document title, page number, or section heading) assists with filtering and citations; removing it does not alter the size or text content of the chunk.

Protecting data and preparing it for customization

In practice, this group of skills lets the machine learning engineer remove sensitive values from a dataset and assemble the examples that customizing a foundation model requires.

Be comfortable doing the following:

Masking, redacting, and anonymizing data, and recognizing the difference: masking hides a value while keeping its shape, redaction removes it, and anonymization changes the data so an individual can no longer be identified from it.
Locating sensitive data before you transform, using automated discovery over object storage and entity detection over text.
Preparing data for foundation model fine-tuning, continued pre-training, and model distillation, and recognizing that these three want different artifacts.

The three customization paths are the distinction to carry away, and each wants a different artifact. Fine-tuning wants labeled examples: prompts paired with the response the model should produce, commonly supplied as JSONL in the schema the model expects. Continued pre-training, which the exam guide calls continuous pre-training, wants a large body of unlabeled domain text, so the model absorbs the vocabulary and patterns of a field. Distillation wants a set of prompts that cover the use case; the managed path generates the larger model's responses to those prompts and fine-tunes the smaller model on them, so the artifact prepared is the prompt set rather than a collected set of answers. Labeling a set of prompt and response pairs and then pointing it at continued pre-training wastes the labeling effort, because that path does not use the responses as targets. Redaction runs before any of them, not after, since anything left in the training data can surface in a model output.

Knowledge Check

A team wants a foundation model to absorb the vocabulary and writing patterns of a specialized field, and it holds a large body of unlabeled domain documents. Which preparation approach matches that goal? (Choose 1)

A. Assemble a prompt set for distillation
B. Assemble the unlabeled domain text for continued pre-training
C. Build paired prompts and target responses for fine-tuning
D. Convert the documents to embeddings and store them in a vector index

Answer & Explanation

Correct Answer: B. Assemble the unlabeled domain text for continued pre-training

Explanation:
Continued pre-training (also known as domain adaptation) uses self-supervised learning on large volumes of unlabeled text to adapt a base foundation model to a specific domain (such as medical, legal, or financial fields). This process updates the model's token vocabulary, internal weight representations, and probability distributions so it naturally adopts the terminology, jargon, and stylistic conventions of the target domain.

Why the other options are incorrect: * A. Assemble a prompt set for distillation: Knowledge distillation transfers capabilities from a larger "teacher" model to a smaller "student" model using paired outputs; it is a model compression technique rather than a domain adaptation strategy for unlabeled text. * C. Build paired prompts and target responses for fine-tuning: Supervised Fine-Tuning (SFT) or instruction tuning requires labeled pairs (prompt + target response) to train a model on specific tasks or instruction-following behaviors, not raw, unlabeled domain documents. * D. Convert the documents to embeddings and store them in a vector index: This sets up a Retrieval-Augmented Generation (RAG) architecture. RAG provides factual ground-truth context at query time, but it does not alter or update the underlying foundation model's weights, vocabulary, or internal writing style.

Review together - Transforming data for ML and AI

Data preparation spans two distinct operational paths depending on whether the workload uses traditional tabular ML or Generative AI (RAG / Fine-Tuning).

Original video transcript

So data transformation is a skill most people think they already have. They can encode a column and scale a feature, so they mark it as done, right? And the problem is that preparation now covers two very different jobs. The same team that encodes tables for a churn model also prepares documents for a chatbot, and the steps barely overlap. Exactly. So let's test that with a real scenario. I'll describe a situation. You reason through the preparation and we'll find where the choices get harder.

You're an ML engineer at a retail company building a customer churn model. The data is structured purchase history, subscription tier, region, and support contexts. You need categorical columns encoded, outliers handled, engineered features like tenure and recency, and scaling that suits the algorithm you pick. So where do you start?

Well, I'd start where the raw data lives because these tables come from more than one system. AWS Glue joins the purchase history with the subscription data and runs the heavy transformation work. OK. And the cleanup, like those outliers. Well, for visual low code cleaning, AWS Glue Data Brew handles the outliers without anyone writing code. And then for interactive feature work, I do Sagemaker Data Wrangler capabilities and Amazon Sagemkaker Canvas.

OK, so take me through the actual feature engineering. All right. Well, the categorical columns like region and tier get encoded into numeric columns. The skewed spending values get a log transformation, and features that need scaling get it consistently for the selected algorithm. Every step here is doing one job, which is turning data into numbers that the model can learn from. Hold on to that phrase, turning data into numbers because it comes back.

Now, what happens to those engineered features after this project? Yeah, they go into Amazon Sagemaker feature store, so training and live inference consume the same shared features. That reduces the risk of training serving skew. Nice. That's a complete traditional pipeline, and it's where people get comfortable.

So let's change the project. The same retail company now wants a product FAQ chatbot built on retrieval augmented generation, or RAG for short. It needs the product documents split for retrieval, an embedding model, a vector store, and metadata so every answer cites the right product. Start me at the documents.

Well, here's where your phrase from earlier kind of comes back. We're still turning data into numbers, but almost nothing else survives. There's no table to encode. The documents get split into chunks size for retrieval with metadata like the product name attached to each chunk.

OK. And this is where a lot of people stall. Encoding turned categories into numbers a minute ago. Why can't encoding handle this text too? Because the two techniques answer different needs. Categorical encoding turns categories into numbers that stand in for labels, and those numbers carry no learned meaning. For semantic retrieval over text, an embedding model through Amazon Bedrock creates vectors that capture relationships and meaning. Similar meaning, similar vectors. So that's the rule. Categories get encoded and meaning gets embedded, right? And the vectors then go into a vector store like Amazon Open Search Service. Retrievalla finds the chunks closest in meaning to a customer's question.

OK, one more decision, because teams genuinely argue about this one. Someone wants the chatbot to also match the company's tone, and they say the fix is fine tuning. Someone else says rag data already covers it. Who's right? Well, neither really, until they separate the goals. Rag preparation makes knowledge retrievable, so it produces chunks, embeddings and metadata. Supervised fine tuning preparation teaches the model a behavior, so it produces model-specific labeled examples, commonly Jason L, in the schema that the model expects.

So knowledge that changes often belongs in the vector store, and behavior that stays constant belongs in the tuning data. Exactly. The product catalog changes all the time, so you retrieve that. And for stable behavior like company tone, start with prompting. And then consider fine tuning if your evaluations show prompting alone isn't enough.

So let's review the two scenarios together. All right, well, the goal never moved. Both pipelines turn data into numbers that a model can use, and both share ingestion, cleaning, validation, and metadata. The structured pipeline ran on Glue Data Brew and Data Wrangler with encoding and scaling. The rag pipeline ran on chunking, an embedding model, and a vector store. And every swap had a reason. Encoding represents categories. Embedding captures meaning, and retrieval data is not tuning data. That's right.

So here's a self check. If every one of those swaps made sense as we went, you're well prepared for this kind of work. And even if one of them surprised you, that swap is your gap. Go back, work out why the preparation changed, and then keep studying.

1. Traditional Tabular ML Pipeline (e.g., Customer Churn Model)

Objective: Convert structured relational tables into numerical matrices optimized for statistical learning.
AWS Service Architecture:
- AWS Glue: Handles heavy ETL and joins tabular data from multiple source systems (e.g., purchase history + subscription tiers).
- AWS Glue DataBrew: Provides low-code visual cleaning and automated data profiling to handle outliers without code.
- Amazon SageMaker Data Wrangler & Canvas: Interactive feature engineering for transformations, balancing, and encoding.
- Amazon SageMaker Feature Store: Stores output features to serve both offline batch training and low-latency online inference, preventing training-serving skew.
Core Transformations:
- Categorical Encoding: Converts discrete categories (e.g., region, tier) into numerical representations that act as label placeholders (carrying no learned semantic relationships).
- Log Transformation: Normalizes highly skewed continuous variables (e.g., spend amounts).
- Feature Scaling: Normalizes numerical features consistently across training and test splits for the chosen algorithm.

2. Generative AI / RAG Pipeline (e.g., Product FAQ Chatbot)

Objective: Prepare unstructured text documents for semantic retrieval and context injection.
AWS Service Architecture:
- Chunking & Metadata Ingestion: Documents are split into retrieval-sized text chunks and tagged with metadata (e.g., product name, document ID).
- Amazon Bedrock: Hosts embedding models to convert text chunks into high-dimensional numerical vectors.
- Amazon OpenSearch Service: Serves as the vector database for indexing embeddings and performing vector similarity search.
Core Transformation:
- Semantic Embedding: Maps words and context into dense vector spaces where mathematical closeness corresponds to semantic similarity.

3. Decision Matrix & Architectural Rules

Rule 1: Encoding vs. Embedding

Characteristic	Categorical Encoding	Text Embedding
Input Data	Tabular categories (e.g., `Region`, `Tier`)	Unstructured text passages / documents
Output	Discrete numerical values / One-Hot vectors	High-dimensional dense vectors
Learned Meaning	None (acts strictly as a label substitute)	Captures semantic relationships & context
Primary AWS Tool	SageMaker Data Wrangler / AWS Glue	Amazon Bedrock (Embedding Models)

Rule 2: Knowledge vs. Behavior (RAG vs. Fine-Tuning)

                    Is the goal Knowledge or Behavior?
                                   │
         ┌─────────────────────────┴─────────────────────────┐
         ▼                                                   ▼
Frequently Changing Knowledge                            Static Behavior & Tone
 (e.g., product catalogs)                                (e.g., company voice)
         │                                                   │
         ▼                                                   ▼
Retrieval-Augmented Generation (RAG)                    Start with Prompting
 • Chunking + Metadata                                       │
 • Embeddings + Vector Store                                 ▼
                                                        Evaluations sufficient?
                                                         ├── Yes ➔ Production
                                                         └── No  ➔ Supervised Fine-Tuning
                                                                   (JSONL prompt-response pairs)

RAG (Dynamic Knowledge): Use when data changes frequently. Updating knowledge requires re-indexing chunks into the vector store without retraining model weights.
Prompt Engineering (Initial Behavior): Use first to control tone and instruction-following without compute overhead.
Fine-Tuning / SFT (Static Behavior): Use when prompt evaluation proves insufficient. Requires building labeled JSONL prompt-response pairs to update model weights.

Shared vs. Divergent Steps

Shared Pipeline Elements: Data ingestion, raw storage, data validation, and metadata management apply to both paradigms.
Divergent Pipeline Elements: Tabular workflows rely on encoding, log scaling, and feature store caching, whereas RAG workflows rely on document chunking, embedding generation, vector indexing, and semantic retrieval.

Quick recall

Without rewatching, write the two rules the conversation lands on: categories get encoded, and meaning gets embedded. Then name what retrieval preparation produces, and what fine-tuning preparation produces instead, which the video gives as chunks against JSONL.

Find your gap

The video ends on a self-check. If every swap between the two scenarios made sense as it went, you are well prepared for this work. If even one surprised you, that swap is your gap, so return to the skills lesson above and work out why the preparation changed.

Knowledge Check

AnyCompany wants a foundation model to answer questions using its internal policy documents, and the policies are revised frequently. Which preparation approach fits that requirement with the LEAST rework as policies change? (Choose 1)

A. Split and embed the policy documents into a vector store the application retrieves from
B. Build prompt and response pairs from the policies and fine-tune the model
C. Assemble the policies as unlabeled text for continued pre-training
D. Assemble a prompt set covering the policies and distill a smaller model from a larger one

Answer & Explanation

Correct Answer: A. Split and embed the policy documents into a vector store the application retrieves from

Explanation:
This scenario describes a classic Retrieval-Augmented Generation (RAG) architecture. When source documents (like internal company policies) change frequently, storing chunked document embeddings in a vector database allows for dynamic knowledge updates. Updating a vector index requires minimal computational overhead—simply deleting or re-embedding the updated document chunks—without modifying or re-training the underlying foundation model weights.

Why the other options are incorrect: * B. Build prompt and response pairs from the policies and fine-tune the model: Supervised fine-tuning bakes factual knowledge directly into model weights. Every time a policy changes, the team would have to recreate prompt-response pairs and re-run expensive training pipelines to avoid presenting stale or conflicting information. * C. Assemble the policies as unlabeled text for continued pre-training: Continued pre-training requires significant GPU compute resources and time. Re-running pre-training for every policy revision is cost-prohibitive and impractical for frequently updated documents. * D. Assemble a prompt set covering the policies and distill a smaller model from a larger one: Model distillation compresses reasoning capabilities from a teacher model into a student model, but it still bakes static data into weights and requires re-distilling whenever source knowledge updates.