Skip to content

Skill 1.2.3: Transform streaming data (for example, by using AWS Lambda, Spark)

Question Answer & Explanation
### Question 1 - Real-Time Data Masking with AWS Lambda

A healthcare startup is building an ML pipeline to ingest streaming patient records from an Amazon Kinesis Data Stream. Before the data is written to an Amazon S3 data lake, any Personally Identifiable Information (PII), such as SSNs and patient names, must be redacted or masked. The engineering team wants to minimize operational overhead and infrastructure management while ensuring the transformation occurs on a per-record basis with minimal latency.

Which ONE solution meets these requirements with the LEAST operational overhead? (Select ONE)

- A) Deploy an Amazon Kinesis Data Firehose delivery stream with an AWS Lambda transformation function to mask PII before writing to S3.
- B) Deploy an AWS Glue streaming job that reads from the Kinesis Data Stream, masks the PII using PySpark, and writes to S3.
- C) Run an Amazon EMR cluster with Spark Structured Streaming to consume the stream, apply masking transformations, and commit to S3.
- D) Deploy an Amazon Managed Service for Apache Flink application to read the stream, apply a Flink map function to mask PII, and sink to S3.		 Correct Answer: A

Correct Answer Analysis:
- A is Correct: Amazon Kinesis Data Firehose natively supports attaching an AWS Lambda function to transform records as they flow through the stream. This is a fully serverless approach that handles per-record transformation (masking) with minimal operational overhead, eliminating the need to manage clusters or streaming frameworks.

Distractor Analysis:
- B is Incorrect: While AWS Glue streaming jobs are serverless, they use Spark Structured Streaming which is designed for micro-batch processing and introduces more overhead and complexity than necessary for a simple per-record masking task.
- C is Incorrect: Running an Amazon EMR cluster requires significant operational overhead to provision, manage, and scale the compute resources, violating the "least operational overhead" constraint.
- D is Incorrect: Amazon Managed Service for Apache Flink is powerful for stateful stream processing but adds unnecessary complexity and overhead for a stateless, per-record masking operation compared to Firehose + Lambda.

Exam Tip / AWS Best Practice: For stateless, per-record transformations on streaming data bound for S3, Kinesis Data Firehose with Lambda provides the lowest operational overhead.
### Question 2 - Batching Streaming Records for Inference

An e-commerce platform processes user clickstream events using an Amazon Kinesis Data Stream. The ML engineering team has built an AWS Lambda function that sends these events to an external SageMaker endpoint for real-time inference. To optimize costs and reduce the number of API calls to the SageMaker endpoint, the team must process records in batches rather than individually, while still maintaining near-real-time latency. The solution must use native AWS features without requiring custom windowing code.

Which ONE configuration meets these requirements? (Select ONE)

- A) Configure the Kinesis Data Stream to deliver data to an Amazon S3 bucket every 60 seconds, and trigger the Lambda function via an S3 Event Notification.
- B) Increase the BatchSize parameter on the Lambda event source mapping to the maximum allowed value to accumulate records.
- C) Set the MaximumBatchingWindowInSeconds parameter on the Lambda event source mapping to 60 seconds.
- D) Use Amazon Kinesis Data Firehose to buffer the records for 60 seconds before invoking the Lambda function.		 Correct Answer: C

Correct Answer Analysis:
- C is Correct: The MaximumBatchingWindowInSeconds parameter in the Lambda event source mapping allows the service to collect records for a specified duration (up to 300 seconds) before invoking the function. This natively batches records over a fixed, non-overlapping time window without writing custom windowing logic, reducing API calls while maintaining near-real-time processing.

Distractor Analysis:
- A is Incorrect: Writing to S3 and triggering via S3 Event Notification introduces high latency (S3 event consistency can take seconds to minutes) and unnecessary storage costs for transient data. It breaks the near-real-time streaming paradigm.
- B is Incorrect: While increasing BatchSize accumulates records, it relies solely on record count. If the stream has low traffic, the batch may never reach the size threshold, causing high latency. The batching window guarantees time-based invocation.
- D is Incorrect: Kinesis Data Firehose does not natively support invoking a Lambda function for arbitrary external API calls (like SageMaker endpoints) in the way an event source mapping does; it uses Lambda for data transformation before sinking to destinations.

Exam Tip / AWS Best Practice: Use MaximumBatchingWindowInSeconds in Lambda event source mappings to implement fixed, non-overlapping time windows for batching streaming records natively.
### Question 3 - Stateful Sliding Window Aggregation

A financial trading firm ingests high-frequency stock ticker data into an Amazon Kinesis Data Stream. The data engineering team needs to compute a continuously sliding 5-minute moving average of stock prices to feed into a fraud detection ML model. The transformation requires maintaining state across the stream and updating the average with every incoming event. The solution must guarantee sub-second processing latency.

Which ONE AWS service is best suited for this transformation? (Select ONE)

- A) Use an AWS Lambda function triggered by the Kinesis stream and store the previous 5 minutes of data in Amazon ElastiCache.
- B) Use AWS Glue DataBrew to schedule a job every 5 minutes to calculate the moving average.
- C) Use Amazon Managed Service for Apache Flink to process the stream and maintain the sliding window state.
- D) Use an AWS Glue streaming job to calculate the moving average using Spark Structured Streaming.		 Correct Answer: C

Correct Answer Analysis:
- C is Correct: Amazon Managed Service for Apache Flink is purpose-built for stateful stream processing with low latency. It natively supports sliding windows and maintains state in memory/backed by checkpoints, perfectly matching the requirement to update a moving average continuously with every event.

Distractor Analysis:
- A is Incorrect: Using Lambda with ElastiCache for a continuously sliding window is an anti-pattern. It introduces high operational complexity, potential race conditions, and Lambda's invocation overhead violates the sub-second latency constraint for continuous updates.
- B is Incorrect: AWS Glue DataBrew is a visual batch data preparation tool. It cannot process continuous streams or calculate continuously sliding windows in real-time.
- D is Incorrect: AWS Glue streaming jobs use Spark Structured Streaming, which operates on a micro-batch model rather than a true continuous event-by-event processing model. This introduces latency that may violate the sub-second requirement compared to Flink.

Exam Tip / AWS Best Practice: For continuously sliding windows and true event-by-event stateful stream processing, use Amazon Managed Service for Apache Flink rather than Lambda or Spark micro-batch processing.
### Question 4 - Stream-to-Batch Join for Feature Engineering

A streaming data pipeline needs to enrich real-time IoT sensor readings from an Amazon Kinesis Data Stream with static device metadata (e.g., device type, installation date) stored in an Amazon S3 data lake. The enriched data must be written back to another Kinesis Data Stream for downstream consumption. The team wants to leverage a serverless Apache Spark environment and natively integrate with the AWS Glue Data Catalog to look up the S3 metadata.

Which ONE approach satisfies these requirements with the LEAST operational overhead? (Select ONE)

- A) Run an Amazon EMR cluster with Spark Structured Streaming, joining the Kinesis stream with the S3 metadata.
- B) Create an AWS Glue streaming job using PySpark to read from Kinesis, join with the S3 data via the Glue Data Catalog, and write to the target Kinesis stream.
- C) Use an AWS Lambda function to read from Kinesis, query the S3 metadata directly via the S3 API, and put records into the target Kinesis stream.
- D) Use Amazon Kinesis Data Firehose to dynamically partition the data based on device ID and trigger a Lambda function to perform the join.		 Correct Answer: B

Correct Answer Analysis:
- B is Correct: AWS Glue streaming jobs run Apache Spark in a serverless environment, minimizing operational overhead. Spark Structured Streaming can seamlessly join streaming data with static data (S3) and integrates directly with the AWS Glue Data Catalog for schema lookups.

Distractor Analysis:
- A is Incorrect: While EMR with Spark can perform the join, it requires provisioning and managing clusters, introducing unnecessary operational overhead compared to serverless AWS Glue.
- C is Incorrect: A Lambda function querying S3 for every incoming stream record would be highly inefficient, slow, and likely exceed Lambda's execution time limits under high throughput, lacking native Spark join optimization.
- D is Incorrect: Kinesis Data Firehose is designed to deliver data to destinations (like S3, Redshift, or HTTP endpoints), not to perform stream-to-batch joins or write back to another Kinesis Data Stream.

Exam Tip / AWS Best Practice: AWS Glue streaming jobs provide a serverless Spark environment ideal for joining streaming data with static datasets using the Glue Data Catalog.
### Question 5 - Error Handling for Poison Pill Records

A media streaming company uses AWS Lambda to process records from an Amazon Kinesis Data Stream, transforming them before storing them in an Amazon OpenSearch Service domain. Occasionally, malformed records (poison pills) cause the Lambda function to fail. The engineering team must ensure that these bad records do not block the processing of subsequent records and are preserved for debugging. Operational overhead must be kept to a minimum.

Which TWO configurations should the engineer implement to fulfill these requirements? (Select TWO)

- A) Configure an Amazon SQS queue as the DestinationConfig (On-failure destination) for the Lambda event source mapping.
- B) Enable BisectBatchOnFunctionError on the Lambda event source mapping.
- C) Increase the RetryAttempts on the Lambda function to 5, and configure a Dead Letter Queue (DLQ) on the Lambda function itself.
- D) Configure the Kinesis Data Stream to automatically skip poisoned records using the SkipOnFailure parameter.
- E) Use Amazon Kinesis Data Analytics to filter out malformed records before they reach the Lambda function.		 Correct Answers: A and B

Correct Answer Analysis:
- A is Correct: Configuring an on-failure destination (such as an SQS queue or SNS topic) in the Lambda event source mapping allows the service to automatically send the offending batch of records to the queue when processing fails, isolating the bad records.
- B is Correct: Enabling BisectBatchOnFunctionError tells Lambda to split the failed batch in half and retry each half separately. This isolates the exact poisoned record, allowing the valid records in the batch to process successfully without being blocked.

Distractor Analysis:
- C is Incorrect: Configuring a DLQ on the Lambda function itself handles asynchronous invocation failures, not event source mapping failures (like Kinesis). For Kinesis, the destination must be configured on the event source mapping.
- D is Incorrect: There is no native SkipOnFailure parameter for Kinesis Data Streams. Error handling must be managed at the consumer (Lambda) level.
- E is Incorrect: Introducing Kinesis Data Analytics solely to filter malformed records adds massive operational overhead and complexity compared to native Lambda event source mapping error handling features.

Exam Tip / AWS Best Practice: For Kinesis and Lambda, use BisectBatchOnFunctionError to isolate poison pills and DestinationConfig to route failed batches to SQS/SNS.
### Question 6 - Converting Streaming Data to Parquet

A retail company ingests real-time customer purchase events using Amazon Kinesis Data Firehose. The data arrives in JSON format and must be stored in an Amazon S3 data lake in Parquet format for downstream ML training. The team must apply a lightweight transformation to drop a few unnecessary fields before conversion. The solution must minimize operational overhead and utilize Firehose's native data format conversion capabilities.

Which ONE sequence of actions meets these requirements? (Select ONE)

- A) Use an AWS Glue streaming job to read from Firehose, drop the fields, and write Parquet directly to S3.
- B) Configure Firehose to transform records using an AWS Lambda function to drop the fields, then enable Firehose's built-in Parquet conversion using an AWS Glue Data Catalog table schema.
- C) Configure Firehose to write JSON directly to S3, then trigger a Lambda function via S3 Event Notifications to convert the data to Parquet.
- D) Enable Firehose dynamic partitioning and write the data as JSON, relying on Amazon Athena to view the JSON as Parquet at query time.		 Correct Answer: B

Correct Answer Analysis:
- B is Correct: Firehose natively supports transforming data using a Lambda function (to drop fields) and then converting the resulting JSON into Parquet format using a schema defined in the AWS Glue Data Catalog. This is the most efficient, serverless approach with the least operational overhead.

Distractor Analysis:
- A is Incorrect: AWS Glue is not a consumer of Kinesis Data Firehose; it consumes Kinesis Data Streams. Using Glue here introduces unnecessary complexity and overhead.
- C is Incorrect: Writing JSON to S3 and triggering a downstream Lambda function to convert to Parquet is a custom, high-overhead architecture that duplicates the native capabilities of Kinesis Data Firehose.
- D is Incorrect: Athena cannot magically "view" JSON as Parquet; the physical storage format must be Parquet to get the performance benefits. Dynamic partitioning does not handle format conversion.

Exam Tip / AWS Best Practice: Kinesis Data Firehose can natively invoke Lambda for transformations and then use the AWS Glue Data Catalog to convert JSON to Parquet/ORC before writing to S3.
### Question 7 - Dynamic Partitioning for Data Lake Routing

A logistics company streams GPS telemetry data from its delivery trucks using Amazon Kinesis Data Firehose. The data science team needs the data partitioned in Amazon S3 by truck_id and event_date so that Amazon Athena queries can efficiently filter by specific trucks and timeframes. The partition keys are nested inside a JSON payload. The solution must minimize operational overhead and avoid managing custom compute infrastructure.

Which ONE approach satisfies these requirements? (Select ONE)

- A) Deploy an AWS Glue streaming job to parse the JSON, extract the keys, and write to the respective S3 partitions.
- B) Configure an Amazon EMR Serverless application to run a Spark job every hour, parsing the JSON and partitioning the output to S3.
- C) Enable dynamic partitioning on the Firehose delivery stream and use an inline extraction or a Lambda transformation function to parse the JSON and specify the partition keys.
- D) Configure Firehose to write to a single S3 prefix and create an AWS Lambda function triggered by S3 PUT events to move files into the correct partitioned folders.		 Correct Answer: C

Correct Answer Analysis:
- C is Correct: Amazon Kinesis Data Firehose supports dynamic partitioning natively. By enabling this feature and configuring either Firehose's built-in JSON parsing (inline extraction) or a lightweight Lambda transformation, Firehose can automatically route records to specific S3 prefixes based on the truck_id and event_date keys without managing compute clusters.

Distractor Analysis:
- A is Incorrect: AWS Glue streaming introduces unnecessary overhead and complexity for a task that Firehose can handle natively through dynamic partitioning.
- B is Incorrect: Using EMR Serverless for hourly batch processing breaks the real-time streaming ingestion paradigm and adds significant latency compared to Firehose.
- D is Incorrect: Using S3 Event Notifications and Lambda to move files between S3 prefixes is an anti-pattern. It is highly inefficient, error-prone, and incurs extra S3 operation costs compared to writing the files to the correct partitions initially.

Exam Tip / AWS Best Practice: Use Kinesis Data Firehose dynamic partitioning with Lambda or inline JSON extraction to automatically route streaming data into Hive-style partitioned S3 prefixes.
### Question 8 - Streaming Feature Ingestion to Feature Store

A data science team is building a real-time recommendation engine. User interaction events (clicks, views) are ingested via an Amazon Kinesis Data Stream. The team must engineer features from these events and ingest them into an Amazon SageMaker Feature Store in near real-time so that online inference endpoints can retrieve the latest user features with single-digit millisecond latency. The transformation logic requires parsing JSON and applying a simple mathematical formula.

Which ONE solution provides the LEAST operational overhead for this continuous ingestion and transformation? (Select ONE)

- A) Use an AWS Lambda function triggered by the Kinesis Data Stream to parse the events, calculate the features, and call the SageMaker Feature Store PutRecord API.
- B) Use an AWS Glue streaming job to read from the stream, calculate features, and write them to an offline S3 store, then sync to the online Feature Store.
- C) Configure Amazon Kinesis Data Firehose to deliver the raw data directly to SageMaker Feature Store.
- D) Deploy a custom Amazon EC2 consumer application using the Kinesis Client Library (KCL) to process events and write to SageMaker Feature Store.		 Correct Answer: A

Correct Answer Analysis:
- A is Correct: An AWS Lambda function triggered by Kinesis is a serverless, event-driven solution. It can easily parse JSON, apply mathematical formulas, and use the AWS SDK to call the SageMaker Feature Store PutRecord API to ingest data into the online store in real-time, minimizing operational overhead.

Distractor Analysis:
- B is Incorrect: AWS Glue streaming is designed for heavy ETL and writing to data lakes, not for low-latency, record-by-record writes to an online feature store. It cannot directly update the SageMaker Feature Store online component efficiently for real-time inference.
- C is Incorrect: Kinesis Data Firehose does not have Amazon SageMaker Feature Store as a native delivery destination. It supports S3, Redshift, OpenSearch, and HTTP endpoints.
- D is Incorrect: Deploying and managing an EC2 cluster with KCL introduces significant operational overhead compared to a serverless Lambda function.

Exam Tip / AWS Best Practice: For streaming ingestion into SageMaker Feature Store, use a Lambda function triggered by Kinesis to transform and call the PutRecord API.
### Question 9 - Reusing Code for Batch and Stream Processing

An ML engineering team has developed a complex feature engineering pipeline using Apache Spark. They currently run this pipeline as an AWS Glue streaming job to process real-time IoT data from Amazon Kinesis Data Streams. The team now needs to backfill the last 12 months of historical IoT data, which is stored in Amazon S3, using the exact same feature engineering logic. They want to avoid rewriting the transformation code.

Which ONE approach allows the team to reuse their existing Spark code for the historical backfill with the least effort? (Select ONE)

- A) Rewrite the pipeline as an AWS Glue DataBrew recipe to process the S3 data.
- B) Convert the AWS Glue streaming job into a standard AWS Glue batch ETL job by changing the input source from Kinesis to S3 and the output sink accordingly.
- C) Use an Amazon EMR cluster and manually port the Spark streaming code to a Spark batch job.
- D) Use Amazon Kinesis Data Firehose to read the historical S3 data and feed it back into the Kinesis stream so the existing streaming job can process it.		 Correct Answer: B

Correct Answer Analysis:
- B is Correct: Apache Spark Structured Streaming supports a unified API for both batch and streaming workloads. By simply changing the readStream to read (which AWS Glue handles natively by switching the datasource) and changing the sink, the exact same transformation code can be executed as a batch job in AWS Glue.

Distractor Analysis:
- A is Incorrect: AWS Glue DataBrew is a visual, low-code tool. Rewriting complex Spark code into a DataBrew recipe would require significant effort and likely be impossible for highly custom logic.
- C is Incorrect: Manually porting the code to an EMR cluster introduces operational overhead and requires code refactoring, which violates the "least effort" constraint. AWS Glue natively supports running the same Spark code as a batch job.
- D is Incorrect: Reading historical S3 data and pushing it through Kinesis Data Firehose/Streams just to reuse the streaming job is an anti-pattern. It incurs unnecessary costs, limits throughput, and adds complexity.

Exam Tip / AWS Best Practice: Spark's unified API allows the same transformation logic to be reused for both batch (S3) and streaming (Kinesis) workloads by simply swapping the data sources and sinks.
### Question 10 - Processing Large State with Custom ML Libraries

A smart city initiative ingests continuous video feeds from traffic cameras as streaming data. The ML engineering team needs to run a real-time transformation pipeline that applies a custom object detection library (written in C++) to extract vehicle counts and feeds this into an anomaly detection model. The processing requires heavy compute resources (GPUs) and maintains large state across the stream. The solution must allow full control over the underlying compute environment and libraries.

Which ONE AWS service is best suited for this scenario? (Select ONE)

- A) Use an AWS Lambda function with attached GPU support to process the video frames.
- B) Use Amazon Managed Service for Apache Flink to run the application and attach GPU-enabled worker nodes.
- C) Use an Amazon EMR cluster with custom AMIs, GPU instances, and Spark Structured Streaming to apply the C++ library via native JNI calls.
- D) Use Amazon Kinesis Data Firehose to deliver the video frames to an Amazon SageMaker Asynchronous endpoint for processing.		 Correct Answer: C

Correct Answer Analysis:
- C is Correct: Amazon EMR provides full control over the cluster environment, allowing the use of custom AMIs to install C++ libraries and GPU drivers. Spark Structured Streaming on EMR can manage large state efficiently and run heavy compute transformations using custom native code.

Distractor Analysis:
- A is Incorrect: AWS Lambda does not support GPU execution or custom C++ libraries in a manner suitable for continuous video stream processing with large state management.
- B is Incorrect: Amazon Managed Service for Apache Flink (KDA) does not support customizing the underlying worker nodes with GPUs or installing custom C++ libraries onto the runtime environment.
- D is Incorrect: Firehose is a delivery service, not a stream processing engine. Routing every video frame to a SageMaker Asynchronous endpoint would be highly inefficient and lack the stateful windowing capabilities required for stream processing.

Exam Tip / AWS Best Practice: When stream processing requires custom native libraries, specialized hardware (GPUs), or deep OS-level control, use Amazon EMR with Spark Structured Streaming instead of serverless options.