Skip to content

Skill 1.2.1: Transform data by using AWS tools

Task 1.2: Data Preparation for ML and AI – Perform data transformation, feature engineering, and pre-processing

Core objective: Convert raw data into model-ready features using the right AWS tool for the execution model: serverless batch, visual no-code, cluster-scale Spark, or interactive ML-native preparation.

Tool Selection Matrix

Tool Execution Model Best For Key Features Not Ideal For
AWS Glue Serverless Apache Spark / Ray / Python Shell Scheduled or event-driven ETL, catalog integration, large-scale batch Crawlers, Data Catalog, job bookmarks, DPU/worker types, Glue Studio, partitioning Interactive exploration, tiny ad-hoc jobs, custom non-Spark frameworks
AWS Glue DataBrew Visual, no-code, serverless Data profiling, cleaning, standardization, outlier handling Projects, recipes, profile jobs, data quality rules, 250+ transforms Complex code-heavy transforms, custom algorithms
Amazon EMR Managed Hadoop/Spark cluster (EC2, EKS, Serverless) Petabyte-scale, custom Spark/Hive/Presto, fine-grained cluster control EMRFS, instance fleets, spot, EMR Studio, notebooks, transient clusters Small jobs where serverless Glue is cheaper/simpler, no-code users
SageMaker Data Wrangler Interactive, low-code in SageMaker Studio/Canvas ML feature engineering, data exploration, quick model, export to pipelines/Feature Store 300+ transforms, data quality report, bias detection, custom transforms Production ETL at massive scale without exporting to scalable pipeline

Decision Rules: When to Use X vs Y

  • AWS Glue vs. Amazon EMR: Glue = serverless Spark, pay per DPU-hour, minimal ops. EMR = more control, custom frameworks, cluster tuning, spot, EMR Serverless for serverless Spark. Choose Glue for standard ETL; EMR for scale/custom.
  • DataBrew vs. Data Wrangler: DataBrew = visual, no-code, profiling, recipes, scheduled jobs. Data Wrangler = ML-native, interactive feature engineering, export to SageMaker Pipelines/Feature Store.
  • Glue vs. DataBrew: Glue for code-based ETL and complex Spark. DataBrew for no-code cleaning and profiling.
  • EMR vs. Data Wrangler: EMR for large-scale distributed processing. Data Wrangler for interactive single-node or sampled ML preparation.
  • Streaming: Lambda for per-record or fixed non-overlapping windows. Spark on EMR/Glue Streaming for sliding windows, stateful processing, or ordering across windows.

Key Configurations & Concepts

  • AWS Glue:
  • Job types: Spark, Spark Streaming, Ray, Python Shell.
  • Workers: G.1X, G.2X, G.4X, G.8X; auto scaling; DPU capacity.
  • Job bookmarks: Track processed data to avoid reprocessing; critical for incremental ETL.
  • Crawlers & Data Catalog: Discover schemas, create table definitions, partitions.
  • Partitioning: Partition S3 data by date/category to reduce scan and cost.
  • AWS Glue DataBrew:
  • Dataset: S3, Data Catalog, JDBC.
  • Project: Interactive session for exploring and building recipe.
  • Recipe: Ordered steps of transforms; can be versioned.
  • Profile job: Computes column statistics, missing values, distributions, data quality rules.
  • Job: Runs recipe on full dataset; can be scheduled.
  • Transforms: Filter, join, aggregate, pivot, unpivot, scale, bin, one-hot.
  • Amazon EMR:
  • Cluster modes: EC2 (long-running/transient), EKS, Serverless.
  • Instance fleets: Mix on-demand and spot for cost.
  • EMRFS: S3 as storage layer; consistent view.
  • EMR Studio: Notebooks and collaboration.
  • Security configurations: Encryption at rest/transit, Kerberos, IAM roles.
  • SageMaker Data Wrangler:
  • Data flow: Import, transform, analyze, export.
  • Transforms: 300+ built-in; custom Pandas/PySpark transforms.
  • Analysis: Data quality and insights report, target leakage, bias detection, quick model.
  • Export: To SageMaker Pipelines, Feature Store, S3, Python code.
  • Integration: SageMaker Canvas for no-code ML.

Security & Compliance

  • IAM roles: Glue, EMR, SageMaker need roles with least privilege to S3, Glue Data Catalog, KMS, Secrets Manager.
  • Encryption at rest: S3 SSE-KMS; Glue Data Catalog encryption; EMR EBS/EMRFS encryption; DataBrew/SageMaker KMS.
  • Encryption in transit: TLS for JDBC, S3, Glue, SageMaker endpoints.
  • VPC endpoints: Use Gateway endpoint for S3; Interface endpoints for Glue, KMS, SageMaker, EMR.
  • Lake Formation: Fine-grained access control on Data Catalog tables for Glue/EMR/Athena.
  • PII: Use Macie to discover sensitive data in S3; Comprehend for text PII; DataBrew can profile and apply masking transforms.
  • Audit: CloudTrail, CloudWatch Logs, Glue job metrics, EMR logs.

Anti-Patterns & Exam Gotchas

Gotcha: AWS Glue is serverless Spark; EMR is managed cluster. If the requirement says "no infrastructure management," choose Glue or EMR Serverless, not EMR on EC2.

Gotcha: DataBrew is for visual/no-code data preparation; Data Wrangler is for ML feature engineering inside SageMaker.

Gotcha: Job bookmarks must be enabled for incremental Glue jobs; without them, reprocessing occurs.

Gotcha: Never refit scalers/encoders at inference; fit on training, transform on inference. Feature Store helps prevent training-serving skew.

Anti-pattern: Using Lambda for large-scale batch transformations (timeout, memory limits). Use Glue/EMR.

Anti-pattern: Using EMR for small ad-hoc jobs where Glue or DataBrew is cheaper/simpler.

Anti-pattern: Using Data Wrangler for petabyte-scale production ETL without exporting to a scalable pipeline (SageMaker Pipelines/Glue/EMR).

Anti-pattern: Ignoring partitioning in S3; leads to full scans and high cost.

Anti-pattern: Not encrypting data at rest/in transit; fails compliance.

Feature Engineering Transforms (Supported by Tools)

  • Scaling/Normalization: Min-max, z-score standardization.
  • Binning: Group continuous values into ranges.
  • Log transformation: Compress skewed distributions.
  • Encoding: One-hot, label encoding for categorical.
  • Feature splitting: Break compound fields (e.g., address).
  • Aggregation/Join: Combine datasets.
  • Balancing: SMOTE, undersampling (Data Wrangler).
  • Text: Tokenization, embeddings (Bedrock) for generative AI.

Generative AI Data Prep (RAG / Fine-Tuning / Distillation)

  • Embeddings: Amazon Bedrock embedding models convert text/images to vectors.
  • Chunking: Fixed size, hierarchical, semantic; metadata extraction for RAG.
  • Fine-tuning: JSONL prompt-response pairs.
  • Continued pre-training: Unlabeled domain text.
  • Distillation: Prompt set; teacher generates responses.
  • Rule: Knowledge that changes often → RAG/vector store. Stable behavior → prompting, then fine-tuning.