Skill 1.1.4: Use AWS streaming data sources to ingest data
| Question | Answer & Explanation |
|---|---|
| ### Question 1 - Selecting the Right Streaming Service for Greenfield AWS Architecture An AI engineering team is building a new real-time fraud detection pipeline on AWS. The pipeline must ingest continuous transaction events, process them with sub-second latency, and feed the features into a SageMaker endpoint. The team has no existing Kafka infrastructure or Kafka expertise, and the architectural mandate requires minimizing operational overhead. Which AWS service should the team use for the streaming data source? (Select ONE) - A) Amazon Managed Streaming for Apache Kafka (Amazon MSK) - B) Amazon Kinesis Data Streams - C) Amazon Kinesis Data Firehose - D) Amazon EMR with Apache Kafka |
Correct Answer: B Correct Option Analysis: - B is Correct: Amazon Kinesis Data Streams is a fully managed, AWS-native streaming service that requires no Kafka cluster management or infrastructure provisioning, perfectly fitting the greenfield and least-overhead constraints. Distractor Analysis: - A is Incorrect: Amazon MSK requires managing Apache Kafka brokers and underlying infrastructure (or managing KRaft/Zookeeper setups), which introduces significant operational overhead for a team without Kafka expertise. - C is Incorrect: Kinesis Data Firehose is designed for delivering streaming data to destinations like S3, OpenSearch, or Redshift, not for building custom, low-latency consumer applications like a fraud detection model. - D is Incorrect: Running Kafka on Amazon EMR introduces massive operational complexity and is not a managed streaming service. > Key Takeaway: For AWS-native, least-overhead streaming workloads without existing Apache Kafka dependencies, choose Amazon Kinesis Data Streams. |
| ### Question 2 - Processing Streaming Data with Complex Windowing Aggregations A retail company ingests point-of-sale data continuously and needs to calculate 5-minute tumbling window aggregations of sales by store ID before writing the results to an OpenSearch Service index. The processing logic requires stateful computations and exactly-once processing guarantees. The team wants to use an AWS managed service to avoid maintaining processing infrastructure. Which solution meets these requirements with the LEAST operational overhead? (Select ONE) - A) AWS Lambda function triggered by Kinesis Data Streams - B) Amazon Managed Service for Apache Flink - C) Amazon Kinesis Data Firehose with dynamic partitioning - D) Amazon EMR Serverless with Spark Structured Streaming |
Correct Answer: B Correct Option Analysis: - B is Correct: Amazon Managed Service for Apache Flink provides a fully managed environment for running Apache Flink applications. It natively supports stateful stream processing, windowed aggregations, and exactly-once processing semantics without managing compute infrastructure. Distractor Analysis: - A is Incorrect: AWS Lambda is inherently stateless. Implementing exactly-once processing and stateful tumbling window aggregations in Lambda requires writing extensive custom logic and managing state externally (e.g., in DynamoDB), which is complex and error-prone. - C is Incorrect: Firehose is a delivery stream service and does not support custom stateful aggregations or exactly-once windowing logic. - D is Incorrect: While EMR Serverless can run Spark Structured Streaming, it carries more operational overhead and startup latency compared to a continuously running Managed Service for Apache Flink application for an always-on pipeline. > Key Takeaway: Use Amazon Managed Service for Apache Flink for stateful, windowed stream processing with exactly-once semantics to minimize infrastructure management. |
| ### Question 3 - Cost-Effective Capacity Mode for Unpredictable Spiky Traffic A logistics company uses Amazon Kinesis Data Streams to ingest real-time GPS telemetry from its delivery fleet. The traffic experiences highly unpredictable spikes during peak holiday seasons, sometimes reaching 10x the normal throughput, but drops to almost zero on weekends. The company wants to optimize costs without manually provisioning shards in advance. Which configuration is the MOST cost-effective and operationally efficient solution? (Select ONE) - A) Use Provisioned mode with reserved shards and implement an auto-scaling script using the UpdateShardCount API. - B) Use On-Demand mode for the Kinesis Data Stream. - C) Use Provisioned mode with a fixed high number of shards to accommodate peak traffic. - D) Buffer the data in Amazon SQS before sending it to a Provisioned Kinesis Data Stream. |
Correct Answer: B Correct Option Analysis: - B is Correct: On-Demand mode automatically accommodates unpredictable spikes in ingestion throughput (up to 200 MiB/s instantly) without requiring manual shard provisioning or auto-scaling scripts. You only pay for the actual data ingested and retained, making it highly cost-effective for spiky, unpredictable traffic. Distractor Analysis: - A is Incorrect: Auto-scaling Kinesis shards using custom scripts requires operational maintenance and cannot scale as instantaneously as On-Demand mode. It also scales in specific increments, which may not perfectly match traffic. - C is Incorrect: Provisioning a fixed high number of shards ensures capacity but incurs high continuous costs even during weekends when traffic drops to near zero. - D is Incorrect: Adding Amazon SQS as a buffer introduces unnecessary architectural complexity and latency, and doesn't solve the core requirement of simplifying Kinesis capacity management. > Key Takeaway: Use Kinesis Data Streams On-Demand mode for unpredictable, spiky workloads to automatically manage capacity and optimize costs. |
| ### Question 4 - Diagnosing a Streaming Ingestion Bottleneck A data engineering team has an Amazon Kinesis Data Stream feeding an AWS Lambda consumer that writes records to an Amazon DynamoDB table. The team notices that the Lambda consumer is consistently falling behind the incoming record rate, but CloudWatch metrics show that the DynamoDB table is not throttling and has plenty of write capacity. What is the MOST likely cause and the best solution to clear the backlog? (Select ONE) - A) The DynamoDB table is the bottleneck; increase the write capacity units (WCUs). - B) The Lambda function's parallelization factor or batch size is too low; increase them to process more records in parallel. - C) The Kinesis stream has insufficient shards; double the shard count. - D) The stream is using On-Demand mode; switch to Provisioned mode to increase read throughput. |
Correct Answer: B Correct Option Analysis: - B is Correct: Since the downstream write target (DynamoDB) is healthy and not throttling, the bottleneck is the consumer processing speed. Increasing the Lambda batch size and setting a parallelization factor allows multiple Lambda instances to process records from the same shard concurrently, clearing the backlog. Distractor Analysis: - A is Incorrect: CloudWatch explicitly shows that DynamoDB is not throttling, so it is not the bottleneck. - C is Incorrect: While adding shards increases overall stream capacity, the bottleneck here is the consumer's ability to process records, not the stream's ability to ingest them. Adding shards alone won't help if the Lambda concurrency isn't adjusted accordingly. - D is Incorrect: The capacity mode (On-Demand vs Provisioned) governs ingestion throughput limits, not the read throughput limit for consumers (which is 2 MiB/sec per shard, or 2 MiB/sec per consumer per shard with Enhanced Fan-Out). > Key Takeaway: When a consumer falls behind but the downstream target is healthy, increase consumer parallelism (e.g., Lambda batch size or parallelization factor) rather than scaling the stream. |
| ### Question 5 - Migrating On-Premises Kafka Workloads to AWS An enterprise currently runs a large, self-managed Apache Kafka cluster on-premises to stream user interaction data for their recommendation models. They are migrating to AWS and want to lift-and-shift their existing Kafka applications (producers and consumers) with minimal code changes, while offloading the underlying cluster management overhead to AWS. Which AWS service best fits this requirement? (Select ONE) - A) Amazon Kinesis Data Streams - B) Amazon Managed Streaming for Apache Kafka (Amazon MSK) - C) Amazon Managed Service for Apache Flink - D) Amazon Kinesis Data Firehose |
Correct Answer: B Correct Option Analysis: - B is Correct: Amazon MSK is a fully managed service that makes it easy to set up, configure, and run Apache Kafka on AWS. Because it uses native Apache Kafka APIs, existing producers and consumers can be migrated without code changes, satisfying the lift-and-shift requirement. Distractor Analysis: - A is Incorrect: Kinesis Data Streams uses proprietary AWS APIs. Migrating native Kafka applications to Kinesis requires rewriting producer and consumer code to use the Kinesis Producer Library (KPL) and Kinesis Client Library (KCL). - C is Incorrect: Managed Service for Apache Flink is a stream processing framework, not a message broker. It can consume from Kafka, but it does not replace the Kafka broker itself. - D is Incorrect: Firehose is a fully managed delivery service for moving data to storage layers (S3, Redshift, OpenSearch), not a general-purpose, highly scalable Kafka-compatible message broker. > Key Takeaway: Use Amazon MSK to migrate existing Apache Kafka workloads to AWS without rewriting application code while offloading cluster management to AWS. |
| ### Question 6 - Real-time Feature Ingestion into SageMaker Feature Store A machine learning team needs to ingest streaming clickstream events, compute rolling window aggregations (e.g., clicks in the last 5 minutes per user), and store them as features in the Amazon SageMaker Feature Store online endpoint for low-latency inference. Which combination of AWS services provides the MOST operationally efficient pipeline for this requirement? (Select TWO) - A) Amazon Kinesis Data Streams - B) Amazon Managed Service for Apache Flink - C) Amazon EMR Serverless - D) Amazon Kinesis Data Firehose - E) AWS Glue Streaming Jobs |
Correct Answers: A and B Correct Options Analysis: - A is Correct: Amazon Kinesis Data Streams is the ideal service to capture the continuous, high-throughput clickstream events in real-time. - B is Correct: Amazon Managed Service for Apache Flink can read from Kinesis Data Streams, compute the stateful rolling window aggregations, and natively write the computed features directly to SageMaker Feature Store using its built-in connectors. Distractor Analysis: - C is Incorrect: While EMR Serverless can run Spark Structured Streaming, it carries more operational overhead and startup latency compared to a continuously running Flink application for a real-time, always-on pipeline. - D is Incorrect: Kinesis Data Firehose cannot perform custom stateful aggregations (like rolling windows) and does not natively write to SageMaker Feature Store. - E is Incorrect: AWS Glue Streaming Jobs (which also use Spark) have higher overhead and latency compared to Flink for sub-second to near-real-time stateful processing. > Key Takeaway: Managed Service for Apache Flink can natively ingest from Kinesis and write computed features directly to SageMaker Feature Store for real-time inference. |
| ### Question 7 - Near Real-Time Indexing to OpenSearch with Transformation A news aggregator receives a continuous stream of article metadata. Before indexing this data into an Amazon OpenSearch Service domain for a RAG application, the metadata must be transformed (e.g., converting JSON keys and dropping null fields) without writing custom processing infrastructure. The solution must minimize operational overhead. Which AWS service should be used to ingest and deliver the data? (Select ONE) - A) Amazon Kinesis Data Streams with an AWS Lambda transformation function - B) Amazon Kinesis Data Firehose with an AWS Lambda transformation function - C) Amazon Managed Service for Apache Flink - D) Amazon MSK Connect |
Correct Answer: B Correct Option Analysis: - B is Correct: Amazon Kinesis Data Firehose natively supports Amazon OpenSearch Service as a direct delivery destination. It also allows you to attach an AWS Lambda function for lightweight ETL transformations on the fly. This requires zero infrastructure management, fulfilling the least operational overhead constraint. Distractor Analysis: - A is Incorrect: Using Kinesis Data Streams requires writing a custom consumer application (e.g., using Lambda or KCL) to read the data, transform it, and write it to OpenSearch, introducing unnecessary custom code and overhead. - C is Incorrect: Using Managed Service for Apache Flink is overkill for simple key conversions and dropping null fields, as it requires developing and managing a full Flink application. - D is Incorrect: Amazon MSK Connect is used to deploy and manage Kafka Connect connectors, not for simple Kinesis-to-OpenSearch transformations. > Key Takeaway: Use Kinesis Data Firehose for serverless, near-real-time delivery to OpenSearch with optional Lambda transformations to minimize operational overhead. |
| ### Question 8 - Durability and Data Retention for Replay A financial services company is building an anomaly detection system. The streaming data source must guarantee that all incoming records are durably stored and can be replayed for up to 7 days in case a consumer application fails and needs to reprocess the historical events. Which AWS service provides this built-in data retention capability natively? (Select ONE) - A) Amazon Kinesis Data Firehose - B) Amazon Kinesis Data Streams - C) Amazon SNS - D) Amazon EventBridge |
Correct Answer: B Correct Option Analysis: - B is Correct: Amazon Kinesis Data Streams stores records from 24 hours up to 365 days (default is 24h, but easily configurable). A 7-day retention period is well within limits, allowing consumers to rewind the stream and replay missed or historical data. Distractor Analysis: - A is Incorrect: Kinesis Data Firehose is a delivery stream that moves data immediately to destinations (S3, OpenSearch, etc.). It does not retain data for extended replay capabilities. - C is Incorrect: Amazon SNS is a pub/sub messaging service designed for fan-out delivery, not for durable, long-term storage and sequential replay of streams. - D is Incorrect: Amazon EventBridge is an event bus that supports limited archive and replay, but it is not a dedicated high-throughput streaming data source like Kinesis Data Streams. > Key Takeaway: Kinesis Data Streams supports configurable data retention (up to 365 days) for replaying missed data, whereas Firehose is strictly for immediate delivery. |
| ### Question 9 - Optimizing Read Throughput from Kinesis Data Streams An analytics application reads from an Amazon Kinesis Data Stream using the Kinesis Client Library (KCL). The consumer application frequently hits the ReadProvisionedThroughputExceededException error, even though the total data ingress rate is well below the shard limits. The application needs to read data at a higher rate without changing the number of shards.Which feature should the engineer enable to resolve this issue? (Select ONE) - A) Enable Enhanced Fan-Out - B) Switch the stream to On-Demand mode - C) Increase the batch size in the Kinesis Client Library (KCL) - D) Use Kinesis Data Firehose as an intermediary buffer |
Correct Answer: A Correct Option Analysis: - A is Correct: Enhanced Fan-Out provides dedicated read throughput of up to 2 MiB/sec per shard per consumer. This bypasses the standard shared limit of 2 MiB/sec per shard total across all consumers, resolving the ReadProvisionedThroughputExceededException without adding shards.Distractor Analysis: - B is Incorrect: Switching to On-Demand mode scales the ingestion capacity automatically, but it does not increase the read throughput limits per consumer on a shard. - C is Incorrect: Increasing the batch size means the consumer requests more records per GetRecords call, but if the read throughput limit is already exceeded, this will not solve the exception.- D is Incorrect: Introducing Firehose as an intermediary would require Firehose to read from the stream (acting as a consumer), which still hits the same read throughput limits, and adds unnecessary complexity. > Key Takeaway: Use Enhanced Fan-Out to give each Kinesis Data Streams consumer dedicated read throughput (2 MiB/sec per shard) without hitting shared provisioned limits. |
| ### Question 10 - Ingesting Ordered Event Streams for AI Pipelines A media company needs to ingest a continuous stream of user interaction metadata (JSON payloads under 1 MB) to trigger an AI recommendation model. The system must capture the exact sequence of user interactions in real-time, scale automatically to handle unpredictable traffic, and require minimal operational management. Which AWS service is the BEST choice for ingesting this ordered event stream? (Select ONE) - A) Amazon Kinesis Data Firehose - B) Amazon S3 with Event Notifications - C) Amazon Kinesis Data Streams (On-Demand mode) - D) Amazon MQ |
Correct Answer: C Correct Option Analysis: - C is Correct: Kinesis Data Streams in On-Demand mode automatically scales to handle unpredictable traffic. It guarantees strict ordering of records within individual shards, making it ideal for capturing the exact sequence of user interactions while requiring zero shard management. Distractor Analysis: - A is Incorrect: Kinesis Data Firehose is designed for delivering streaming data to storage destinations (S3, OpenSearch) and does not support the low-latency, ordered triggering of custom consumer applications like an AI model. - B is Incorrect: Amazon S3 is object storage, not a high-throughput, low-latency ordered event stream. - D is Incorrect: Amazon MQ is a managed message broker service for ActiveMQ/RabbitMQ. While it supports ordering, it is not as highly scalable or serverless as Kinesis Data Streams for large, spiky event ingestion workloads. > Key Takeaway: For ordered, automatically scalable, AWS-native streaming with minimal operational overhead, choose Amazon Kinesis Data Streams in On-Demand mode. |