Skip to content

Skill 1.2.1: Transform data by using AWS tools (for example, AWS Glue, AWS Glue DataBrew, Spark on Amazon EMR, SageMaker Data Wrangler)

Question 1 - AWS Glue DataBrew vs. SageMaker Data Wrangler Selection

A financial services company needs to perform data preparation on a large customer transaction dataset stored in Amazon S3. The data engineering team consists primarily of business analysts with limited coding experience. They need to visually inspect data quality issues, identify outliers, and create a reusable transformation recipe that can be scheduled to run weekly as new transaction data arrives. Operational overhead must be minimized.

Which AWS service or combination of services should the team use to meet these requirements? (Select ONE)

  • A) Use SageMaker Data Wrangler in SageMaker Canvas to create the transformation flow, then export the flow as a Jupyter notebook for scheduled execution via AWS Glue.
  • B) Use AWS Glue DataBrew to create a visual recipe with data profiling, then schedule the recipe job to run weekly.
  • C) Use Amazon EMR with Apache Spark to write custom transformation scripts that read from S3 and process data on a fixed weekly schedule.
  • D) Use AWS Lambda functions with pandas to transform the data individually on a CloudWatch Events (Amazon EventBridge) weekly trigger.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: AWS Glue DataBrew is a visual data preparation tool specifically designed for non-coders. It provides built-in data profiling to identify data quality issues, outliers, and inconsistencies without writing code. Users can visually build transformation workflows (recipes) by adding steps like filtering, handling outliers, and normalizing data. These recipes can then be saved and scheduled as recurring jobs to run on new data arriving in S3, minimizing operational overhead.

Distractor Analysis

  • A is Incorrect: While SageMaker Data Wrangler is an interactive, low-code option, it is designed for ML-focused workflows within Canvas and Studio. Exporting flows to Jupyter notebooks introduces additional overhead and requires custom coding to schedule and run the job weekly. The combination is more complex than a single DataBrew scheduled job.
  • C is Incorrect: Amazon EMR with Apache Spark is a heavy, cluster-based big data processing solution that requires coding expertise and cluster management overhead. It is over-engineered for a business analyst team needing visual no-code tools, even though it supports scheduled jobs.
  • D is Incorrect: Using AWS Lambda with custom Python pandas code goes beyond low-code visual transformation needs and requires specialized development skills. Lambda also has memory/timeout limits that may not suit large dataset transformations.

Key Takeaway: AWS Glue DataBrew is the go-to serverless, low-code tool for visual data profiling and recipe-based transformations, ideal for teams without deep coding expertise and for scheduled batch jobs.

Question 2 - Choosing the Right Tool for Stateful Streaming Transformation

A logistics company streams GPS tracking data from IoT devices into Amazon Kinesis Data Streams. Their ML pipeline requires continuous calculation of a rolling z-score over the last 15 minutes of speed and acceleration data to detect real-time driving anomalies. The system must maintain state across a continuously sliding window, and ordering guarantees are essential. The ML engineer must choose a processing framework that minimizes operational overhead while meeting these stateful processing requirements.

Which solution BEST meets these requirements? (Select ONE)

  • A) Deploy AWS Lambda functions triggered by Kinesis Data Streams with a batch window of 15 minutes to compute the rolling z-score.
  • B) Use Amazon EMR with Apache Spark Structured Streaming to read from Kinesis and compute the sliding window aggregates.
  • C) Use Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics) to process the stream with a 15-minute sliding window and built-in state management.
  • D) Use AWS Glue streaming jobs with Spark Structured Streaming to read from Kinesis and compute the rolling z-score continuously.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Answer Analysis

  • C is Correct: Amazon Managed Service for Apache Flink is specifically built for stateful stream processing with continuously sliding windows. It natively maintains intermediate state in memory and checkpoints it for fault tolerance, supporting event-ordering guarantees across the sliding window. This is exactly what a rolling z-score over the last 15 minutes requires, and it does so with a fully managed service, minimizing operational overhead.

Distractor Analysis

  • A is Incorrect: AWS Lambda is an event-driven, stateless compute service by default. While Lambda can be configured with a fixed batch window (non-overlapping tumbling windows) or a time-based fixed window, it does not natively support continuously sliding windows with out-of-order event handling and state management. A continuously sliding window over 15 minutes of high-velocity IoT data would exceed Lambda's capabilities and operational efficiency.
  • B is Incorrect: Amazon EMR with Spark Structured Streaming can perform stateful streaming aggregations, including sliding windows, but it requires the team to provision, manage, and tune the EMR cluster. This introduces significantly more operational overhead compared to the fully managed Apache Flink service, even though both support sliding windows.
  • D is Incorrect: AWS Glue streaming jobs (based on Spark Structured Streaming) do support streaming ETL, but they are optimized for ETL-style transformations (e.g., joining, filtering, mapping) and still run as serverless Spark jobs. Managing continuously sliding windows with complex state and event-time ordering in Glue is less straightforward and not the optimal managed choice for stateful real-time analytics. Amazon Managed Service for Apache Flink is purpose-built for this.

Key Takeaway: For streaming analytics that requires continuously sliding windows with state and ordering guarantees, choose Amazon Managed Service for Apache Flink. For per-record/fixed-window, stateless transformations, AWS Lambda is sufficient.

Question 3 - Selecting the Correct Feature Engineering Transform for Skewed Data

A healthcare analytics company is building a model to predict patient readmission risk. One of the features is "total_claim_amount," which contains billing amounts for patients over the past year. The data scientist observes a heavily right-skewed distribution for this feature (most values are small, but some are extremely large). The model chosen is a linear regression model that assumes features are approximately normally distributed.

Which transformation should the data scientist apply to "total_claim_amount" before training? (Select ONE)

  • A) Min-max scaling (normalization) to map all values to a 0–1 range.
  • B) Standardization (z-score normalization) using the mean and standard deviation.
  • C) Log transformation to compress the skewed distribution into a more Gaussian-like shape.
  • D) One-hot encoding to convert the continuous values into categorical bins.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Answer Analysis

  • C is Correct: Log transformation is specifically designed to compress heavily right-skewed distributions. By applying log(1 + x) or log(x), extreme outliers are pulled inward, and the distribution becomes more symmetric and closer to Gaussian. This is the standard pre-processing step for features like monetary amounts, counts, or durations that follow a long-tail distribution, especially for linear models that assume normality.

Distractor Analysis

  • A is Incorrect: Min-max scaling only maps values linearly to a [0,1] range. It preserves the relative distances and does not address skewness. The outlier values would still dominate the scaled feature space, violating the linear model's assumption of near-normality.
  • B is Incorrect: Standardization centers the data by subtracting the mean and dividing by standard deviation, resulting in zero mean and unit variance. However, it does not fundamentally change the shape of the distribution; the skewness remains after standardization, so the model's normality assumption is still violated.
  • D is Incorrect: One-hot encoding is used for categorical variables, not continuous ones. Converting a continuous feature to one-hot bins would result in a massive number of sparse columns and lose the ordinal nature of the underlying data. Binning is a different approach, but one-hot encoding it is not standard unless the feature becomes a true categorical bucket.

Key Takeaway: Log transformation is the primary technique for normalizing heavily skewed continuous data (e.g., money, time, counts) before training models like linear regression that assume normally distributed features.

Question 4 - Feature Store for Training-Serving Consistency

An e-commerce company is building a real-time fraud detection model. The ML team developed a feature engineering pipeline in Amazon SageMaker that scales numerical features (e.g., transaction_amount) using a StandardScaler fitted on the training dataset, then trains a fraud classifier. When deploying the model to production for real-time inference, the team must ensure that the exact same scaling parameters (mean, standard deviation) are applied to incoming transaction data, preventing any difference between training features and inference features.

Which AWS service should the team use to centrally store and serve the engineered features? (Select ONE)

  • A) Store the scaling parameters in an Amazon S3 bucket and retrieve them via a Lambda function during inference.
  • B) Use Amazon SageMaker Feature Store to store the transformed features and their corresponding scaler metadata, and retrieve them at inference time.
  • C) Write custom inference code that recomputes scaling parameters from the incoming batch of data before feeding them to the model.
  • D) Embed the scaling logic directly in the model artifact and retrain the model monthly.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: Amazon SageMaker Feature Store is designed precisely for this use case. It acts as a central repository where both the transformed features and their associated metadata (including scaler parameters) are stored and retrieved consistently by both training and inference pipelines. By serving features from a single source of truth, the team eliminates training-serving skew and ensures the exact transformation fitted on training data is applied during inference.

Distractor Analysis

  • A is Incorrect: While storing parameters in S3 and retrieving via Lambda is technically possible, it introduces custom operational code for retrieval, versioning, and consistency. This is more operationally complex and less robust than using a dedicated feature store service that inherently manages consistency and low-latency retrieval.
  • C is Incorrect: Recomputing scaling parameters from inference data is the root cause of training-serving skew. The parameters (mean, std dev) must be fixed from the training fit. Re-fitting on new data changes the feature values, causing the model to receive inputs on a different scale than it was trained on.
  • D is Incorrect: Embedding scaling logic in the model artifact does not address consistency across different model versions or data pipelines. It also requires manual retraining to update parameters, which doesn't solve the inference-time consistency issue for real-time scoring.

Key Takeaway: Amazon SageMaker Feature Store provides a single source of truth for engineered features, ensuring the fitted transformation (e.g., StandardScaler) is applied identically during training and inference, preventing training-serving skew.

Question 5 - AWS Glue vs. Amazon EMR for Batch Transformation

A large telecommunications company needs to run a nightly batch job that joins call detail records (CDRs) from multiple S3 buckets (terabytes of data), applies complex transformations (joins, aggregations, filtering), and outputs a consolidated dataset in Parquet format for a downstream ML model. The job must be cost-effective, run on a fixed schedule, and the team wants to avoid managing any underlying infrastructure. The data is partitioned by date, and the job runs once per day.

Which AWS service should the ML engineer choose for this batch transformation? (Select ONE)

  • A) Amazon EMR with a long-running cluster and custom Spark scripts.
  • B) AWS Glue job with Apache Spark, triggered on a daily schedule via AWS Glue Scheduler.
  • C) AWS Lambda functions triggered by EventBridge to process each file as it arrives.
  • D) Amazon SageMaker Data Wrangler flow scheduled via SageMaker Pipelines.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: AWS Glue is a serverless ETL service built on Apache Spark that is ideal for scheduled batch transformations over large datasets. It automatically provisions and manages the compute resources needed for the job, so operational overhead is minimal. Glue can handle joins, aggregations, and filtering across multiple S3 sources and write output in Parquet format. Its built-in scheduler triggers daily runs, meeting the fixed-schedule requirement.

Distractor Analysis

  • A is Incorrect: Amazon EMR requires provisioning and managing a cluster, which violates the constraint of avoiding infrastructure management. While EMR can run Spark jobs on terabytes of data, it introduces operational overhead (cluster setup, tuning, scaling, and shutdown). A long-running cluster is also not cost-effective for a nightly job; a transient one would be better but still more complex than serverless Glue.
  • C is Incorrect: AWS Lambda is designed for event-driven, short-lived compute with limited memory and execution time. Processing terabytes of data with complex joins across multiple datasets is beyond Lambda's capacity (15-minute timeout, 10GB memory limit). It is not the right tool for heavy batch ETL.
  • D is Incorrect: Amazon SageMaker Data Wrangler is an interactive, low-code tool for feature engineering on smaller datasets for ML. Scheduling flows via SageMaker Pipelines is possible, but Data Wrangler is not designed for heavy multi-terabyte joins and aggregations. AWS Glue is the better fit for large-scale ETL.

Key Takeaway: AWS Glue is the serverless, Spark-based service of choice for scheduled batch ETL jobs on large datasets where operational overhead must be minimized.

Question 6 - S3 Event Notification with Lambda for Per-Record Transformation

A manufacturing company streams sensor readings from factory floor machines into an Amazon S3 bucket as individual JSON files, which arrive every few seconds. The company needs to transform each JSON file as soon as it arrives: parse it, extract specific fields, convert timestamp formats, and write the transformed record to an Amazon OpenSearch Service for real-time dashboards. The transformation logic is simple, stateless, and processes one record at a time.

Which architecture is MOST appropriate for this near-real-time transformation? (Select ONE)

  • A) Run an AWS Glue job every 15 minutes to read new files from S3 and transform them.
  • B) Configure an S3 Event Notification to trigger an AWS Lambda function that transforms each file and writes it to OpenSearch.
  • C) Set up an Amazon Kinesis Data Firehose delivery stream to read from S3 and transform records before sending to OpenSearch.
  • D) Use Amazon EMR with Spark Streaming to continuously read new files from S3 and transform them.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: S3 Event Notifications integrate directly with AWS Lambda, which is the ideal service for simple, stateless, per-record transformations. When a new JSON file arrives, S3 triggers Lambda almost immediately. The Lambda function parses the JSON, extracts fields, converts timestamps, and writes the result to OpenSearch. This is a fully serverless, low-latency, event-driven pattern with minimal operational overhead and cost, as Lambda scales automatically with the frequency of file arrivals.

Distractor Analysis

  • A is Incorrect: Running an AWS Glue job every 15 minutes introduces a fixed latency that does not meet the "as soon as it arrives" requirement. Glue is better suited for batch ETL on larger datasets, not near-real-time per-record processing.
  • C is Incorrect: Kinesis Data Firehose is designed to ingest streaming data from producers (like IoT devices writing to Kinesis Data Streams or Firehose directly), not to read from S3 as a source. While Firehose can transform records via Lambda, it does not natively pull from S3. This option misuses Firehose's data source architecture.
  • D is Incorrect: Amazon EMR with Spark Streaming is a heavy, cluster-based framework that introduces significant operational complexity and cost for a simple, stateless transformation. For per-record, low-latency work, Lambda is more appropriate and cost-effective.

Key Takeaway: S3 Event Notifications + AWS Lambda is the canonical serverless pattern for low-latency, stateless, per-record data transformation triggered by object creation events in S3.

Question 7 - AWS Glue DataBrew Features and Limitations

A retail company wants to perform visual data preparation on a dataset stored in Amazon S3. The dataset contains customer purchase history with columns that have missing values, duplicate rows, and inconsistent date formats. The business analyst wants to build a reusable recipe that includes removing duplicates, filling missing values with the median, standardizing date formats, and flagging outliers in a numerical column (marked with an F_OUTLIER flag instead of being removed). The final output must be written to S3 in CSV format.

Which of the following steps are feasible in AWS Glue DataBrew? (Select TWO)

  • A) Add a recipe step to flag outliers in a numeric column using built-in analysis (e.g., z-score) and create a new indicator column F_OUTLIER without removing the original values.
  • B) Use custom Python code within a DataBrew recipe step to fill missing values with the median of each column.
  • C) Define a DataBrew recipe that includes steps to remove duplicate rows based on a primary key, fill missing numeric values with the column median, and standardize date formats using a built-in date parsing formula.
  • D) Schedule the DataBrew recipe to run daily using Amazon EventBridge and write the output directly to an Amazon Aurora MySQL database.
  • E) Use DataBrew's built-in machine learning-based imputation to fill missing values with predicted values before writing to S3.
Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: DataBrew supports statistical functions and profiling-based transformations. You can use built-in formulas or transformations to detect outliers (e.g., using z-score thresholds) and create a new flag column, such as F_OUTLIER, without deleting the original data. This preserves the raw data while flagging for downstream review.
  • C is Correct: DataBrew allows you to build recipes with multiple steps, including deduplication (removing duplicate rows based on a key), fill missing values with a specified statistic (mean, median, mode, or a constant), and date/time transformations (parsing and standardizing date formats using formula functions). Writing output to S3 is a standard job configuration, and CSV is a supported output format.

Distractor Analysis

  • B is Incorrect: AWS Glue DataBrew is a visual, low-code tool. While it provides hundreds of built-in transformations and formulas, it does not allow arbitrary custom Python code within recipe steps. For custom code, you would use AWS Glue (Spark jobs) instead, which allows Python/Scala code.
  • D is Incorrect: While DataBrew jobs can be scheduled (including via EventBridge), the output writing is limited to certain sinks. DataBrew job output sinks include Amazon S3, AWS Glue Data Catalog tables (via S3), and JDBC connections to relational databases. Amazon Aurora is a MySQL/PostgreSQL-compatible relational database; DataBrew does support JDBC targets, but the direct "write to Aurora MySQL" is possible only if you configure a JDBC connection. The phrase "directly to Aurora MySQL" is a nuanced stretch; the typical supported pattern is writing to an S3-backed Glue Catalog table or S3 files. However, more importantly, EventBridge is not the native scheduler for DataBrew—DataBrew has its own scheduling mechanism. The core issue is that writing directly to an RDS database is less standard and often avoided for cost/latency, making this distractor partially correct but not the best practice. (Adjusted: DataBrew schedules jobs internally; EventBridge integration is possible but not the primary method. The JDBC sink is possible but less common and still requires configuration; not a best practice for a daily CSV pipeline.)
  • E is Incorrect: AWS Glue DataBrew does not offer ML-based imputation for missing values. This is a feature more characteristic of SageMaker Data Wrangler (which has data quality and some ML-based analysis) or custom SageMaker processing. DataBrew's imputation uses common statistics (mean, median, mode, constant) or advanced options like interpolation, but not trained ML models.

Key Takeaway: DataBrew is powerful for visual data quality tasks (dedupe, fill missing, date standardization, outlier flagging), but it is not a custom-code environment and does not include ML-based imputation. For custom Python transformations, use AWS Glue jobs.

Question 8 - Real-Time Streaming with AWS Lambda vs. Spark on EMR

A social media analytics company ingests a high-volume stream of user interaction events (likes, shares, comments) from a mobile app into Amazon Kinesis Data Streams. The ML team needs to transform each event in real-time by enriching it with user demographic data from an Amazon DynamoDB table, then write the enriched record to an Amazon Redshift cluster for downstream analytics. The transformation is stateless per record. The system must handle thousands of events per second with minimal latency and low operational overhead.

Which solution should the ML engineer recommend? (Select ONE)

  • A) Use Amazon EMR with Apache Spark Structured Streaming to read from Kinesis, perform the enrichment in-memory using a cached copy of the DynamoDB table, and write to Redshift.
  • B) Configure AWS Lambda with a Kinesis Data Streams event source mapping to process each batch of events, perform the enrichment by reading from DynamoDB, and write to Redshift.
  • C) Use AWS Glue streaming jobs to read from Kinesis, enrich with DynamoDB, and write to Redshift via a JDBC connection.
  • D) Use Amazon Kinesis Data Firehose with a Lambda pre-processing function to enrich events and deliver to Redshift.
Click to reveal Answer & Explanation

Correct Answer: D

Correct Answer Analysis

  • D is Correct: Amazon Kinesis Data Firehose is purpose-built for ingesting streaming data and delivering it to destinations like Amazon Redshift. It supports Lambda pre-processing (transformations) where the Lambda function can read from DynamoDB to enrich each record before Firehose batches and loads data into Redshift. This approach is fully serverless, scales automatically to thousands of events per second, and minimizes operational overhead. Firehose also handles batching, retries, and delivery to Redshift natively.

Distractor Analysis

  • A is Incorrect: Amazon EMR with Spark Structured Streaming can handle this workload, but it requires running and managing a Spark cluster. This introduces significant operational overhead, including provisioning, tuning, and scaling, which violates the "low operational overhead" constraint. It is also more expensive and complex for a stateless per-record enrichment task.
  • B is Incorrect: While Lambda with a Kinesis event source mapping can process events and enrich them, writing directly to Amazon Redshift from Lambda is not a supported or recommended pattern. Redshift expects batch loads (e.g., via COPY from S3 or via Firehose). Writing row-by-row from Lambda to Redshift would be inefficient, prone to throttling, and require custom connection management. Firehose is the managed, optimized path for streaming into Redshift.
  • C is Incorrect: AWS Glue streaming jobs can read from Kinesis, but they are more suited for ETL/ELT tasks, not ultra-low-latency per-record enrichment for thousands of events per second with delivery to Redshift. Glue streaming jobs have higher latency and are not a perfect match for this immediate delivery requirement. They also have limitations with DynamoDB lookups and Redshift writes compared to the Firehose-native approach.

Key Takeaway: For stateless per-record enrichment at high volume with delivery to Amazon Redshift, use Kinesis Data Firehose with a Lambda pre-processing function—it is serverless, low-latency, and avoids cluster management.

Question 9 - SageMaker Data Wrangler Use Case and Integration

A machine learning team is working on a churn prediction model for a subscription-based media streaming company. The team has a dataset of user activity and billing information in Amazon S3. They need to perform interactive feature engineering, including:

  1. Performing one-hot encoding on categorical columns (e.g., subscription_plan, country).
  2. Applying SMOTE to the dataset to address class imbalance.
  3. Joining two datasets (activity and billing) based on user_id.
  4. Dividing a timestamp column into separate features for day_of_week and hour_of_day.

The team wants to use a low-code, interactive tool and eventually export the resulting workflow for production use.

Which AWS service or feature best fits this workflow? (Select ONE)

  • A) Use AWS Glue DataBrew to perform all transformations via a visual recipe and export the recipe to a CloudFormation template for scheduled runs.
  • B) Use SageMaker Data Wrangler in Amazon SageMaker Canvas to perform the joins, one-hot encoding, feature splitting, and SMOTE rebalancing, then export the flow to a reusable pipeline.
  • C) Use Amazon EMR with Spark to write PySpark code for all transformations and save the transformed dataset back to S3.
  • D) Use AWS Glue with a Spark job to apply the transformations using Custom Visual Jobs (low-code interface) and schedule it daily.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: Amazon SageMaker Data Wrangler (now part of SageMaker Canvas) is specifically designed as a low-code, interactive feature engineering tool for ML workflows. It natively supports:
    • Joins between multiple datasets.
    • One-hot encoding for categorical features.
    • Feature splitting (e.g., extracting day_of_week, hour_of_day from timestamps).
    • Data balancing with SMOTE (Synthetic Minority Over-sampling Technique). Data Wrangler flows can be exported to SageMaker Pipelines, Jupyter notebooks, or Python scripts for productionization, making it the ideal choice for this ML-specific preprocessing workflow.

Distractor Analysis

  • A is Incorrect: AWS Glue DataBrew is also a visual tool, but it does not natively support SMOTE for data balancing or some advanced ML-specific transforms like feature splitting from timestamps (though it has some date functions, feature splitting is more specific in Data Wrangler). DataBrew is more oriented toward general data preparation/cleaning rather than ML-specific feature engineering. Also, exporting to CloudFormation is not the primary productionization path for Data Wrangler.
  • C is Incorrect: Amazon EMR with PySpark requires custom coding and cluster management. This exceeds the "low-code, interactive" requirement. While PySpark can perform all the transformations, the overhead is not justified for an interactive ML workflow better served by Data Wrangler.
  • D is Incorrect: AWS Glue (low-code or otherwise) does not have native SMOTE support for imbalanced classification problems. SMOTE is an ML feature available in SageMaker Data Wrangler. AWS Glue is an ETL tool for general data transformations, not ML-specific preprocessing.

Key Takeaway: SageMaker Data Wrangler is the low-code, ML-specific feature engineering tool that supports advanced transformations like SMOTE, feature splitting, and one-hot encoding, with export to production pipelines (SageMaker Pipelines, notebooks, scripts).

Question 10 - Selecting Between AWS Glue, EMR, DataBrew, and Data Wrangler for a Complex Batch Job

A large healthcare company needs to process medical claims data stored in Amazon S3. The dataset consists of 50 terabytes of data split into multiple tables (patients, claims, providers, medications). The team needs to perform complex SQL-like transformations including multiple JOINs, GROUP BY aggregations, window functions, and data quality checks (e.g., detecting NULL values in key columns). The job will run once per day, and the output must be stored in Amazon Redshift for downstream BI reporting. The team includes experienced data engineers comfortable with coding.

Which AWS service or combination best fits this use case, balancing cost, performance, and operational management? (Select ONE)

  • A) Use AWS Glue (Spark ETL) to perform the transformations and write the results to Redshift via the Redshift connector, scheduled daily.
  • B) Use AWS Glue DataBrew to visually build the transformation recipe and schedule it to run daily, writing output to Redshift.
  • C) Use Amazon EMR with Spark to run the transformations on a transient cluster, then write to Redshift via a JDBC connection, orchestrated with Step Functions.
  • D) Use Amazon SageMaker Data Wrangler to perform the multi-table joins and aggregations, then export the flow to a SageMaker Processing job scheduled daily.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Answer Analysis

  • A is Correct: AWS Glue is a serverless Spark-based ETL service that is ideal for large-scale batch transformations involving complex joins, aggregations, and data quality checks across terabytes of data. Experienced data engineers can write Spark code (PySpark/Scala) or use Glue Studio's visual ETL. Glue's Redshift connector allows writing transformed data directly into Amazon Redshift. The job can be scheduled daily using Glue's built-in scheduler. Serverless ETL eliminates cluster management overhead (compared to EMR) while scaling to handle 50TB efficiently and cost-effectively.

Distractor Analysis

  • B is Incorrect: AWS Glue DataBrew is a visual data preparation tool optimized for cleaning and transforming single tables (or simple joins) on smaller datasets. Handling 50TB with complex multi-table joins, window functions, and aggregations is beyond DataBrew's intended use case and would be slow, inefficient, and limited in capability. DataBrew sacrifices raw compute power and code flexibility for simplicity.
  • C is Incorrect: Amazon EMR with Spark on a transient cluster is technically capable, but it requires the team to provision, configure, and manage the cluster (even transient). This adds operational overhead compared to serverless AWS Glue. While Step Functions can orchestrate the cluster lifecycle, the overhead is not justified when Glue already provides the same Spark engine serverless. EMR is preferable only when you need deeper customizations (e.g., specific Spark versions, other frameworks like HBase, or node-level access).
  • D is Incorrect: SageMaker Data Wrangler is designed for interactive, ML-oriented feature engineering on smaller datasets (typically up to tens of GB, not 50TB). It is not an enterprise-scale ETL tool for massive batch jobs with complex SQL-like transformations and writing to a data warehouse. Using it for 50TB would be impractical and very expensive.

Key Takeaway: For large-scale (multi-TB), complex batch ETL with experienced engineers, AWS Glue (serverless Spark) is the most balanced choice—code flexibility of Spark without cluster management overhead, integrated with Redshift for output.