Skip to content

Skill 1.1.4: Use AWS Streaming Data Sources to Ingest Data

Task 1.1: Data Preparation for ML and AI — Collect and store data

Objective: Land continuously arriving records into a durable, replayable, queryable store so ML features, training sets, and real-time inference consume fresh data — without reprocessing history or corrupting point-in-time correctness.

1. Service Selection Matrix (highest-yield decision)

Service Role in the pipeline Replay / multiple consumers Ordering Latency Choose when
Kinesis Data Streams (KDS) Durable AWS-native record buffer; consumers pull Yes — retention 24h → 365d, many consumers Per shard, by partition key ~70 ms–200 ms You need replay, fan-out, custom consumers, or KCL/Lambda processing
Kinesis Data Firehose Managed delivery service to S3/Redshift/OpenSearch/Splunk No — not a stream, no replay, no pull Best-effort; at-least-once 60 s+ buffering Straight "stream → storage/lake" with zero consumer code
Amazon MSK (provisioned) Managed Apache Kafka; Kafka API/topic/consumer-group compatible Yes — topic retention, log compaction, replay Per partition Low ms Org standardized on Kafka, existing producers/Connect connectors
MSK Serverless Same Kafka semantics, no broker/partition capacity management Yes Per partition Low ms Kafka compatibility + unpredictable throughput; less tuning control
Managed Service for Apache Flink Stateful processor (not a buffer) — windowing, joins, enrichment Reads from KDS/MSK/S3; sinks to KDS/Firehose/S3/OpenSearch Event-time semantics Sub-second You need windowed aggregates, stream-stream joins, exactly-once state
AWS Glue Streaming ETL Serverless micro-batch Spark over KDS/MSK Checkpointed micro-batch Micro-batch Seconds Reuse existing Glue/Spark jobs, no cluster ops
Lambda + event source mapping Serverless consumer for KDS/MSK Checkpointed by service Per shard/partition Sub-second Light per-record transform, low volume, spiky
MSK Connect Managed Kafka Connect workers Kafka offsets Per partition Low ms Sink/source connectors (S3, OpenSearch, JDBC) without running Connect
Kinesis Video Streams Time-indexed media ingest Yes (fragments) Per fragment Seconds Video/audio for Rekognition Video or SageMaker CV pipelines

Trap: Firehose cannot be read by consumers and cannot be an event source for Lambda. If the scenario says "replay," "reprocess," "second consumer," or "backfill," the answer is KDS or MSK — never Firehose.

Trap: Kinesis Data Analytics was renamed Managed Service for Apache Flink. Same service, new name in distractors.

2. Capacity, Scaling & Throughput

Kinesis Data Streams — per-shard hard limits (memorize):

  • Write: 1 MB/s and 1,000 records/s per shard.
  • Read (shared): 2 MB/s per shard shared across all consumers → 5 consumers polling = 400 KB/s each.
  • Enhanced fan-out: 2 MB/s per registered consumer, dedicated HTTP/2 push (no polling contention).
  • Record size: 1 MB max (includes partition key); batch put up to 500 records / 5 MB per call.
  • Retention: 24 h default → up to 365 days (long retention = replay/backfill without re-ingest).
  • Provisioned mode: manual shard count; scale via split/merge/update-shard-count; cost per shard-hour + PUT payload units.
  • On-demand mode: auto-scales, default 4 MB/s write / 200 MB/s read per stream, doubles on sustained >50% utilization; no shard math but a throughput ceiling.
  • Partition key = shard assignment. Low-cardinality or skewed keys create hot shards while others idle → provision more shards but throughput does not rise.

MSK / Kafka:

  • Partition = unit of parallelism (and ordering). A topic with N partitions caps a consumer group at N effective consumers.
  • Replication factor 3 across AZs for durability; min.insync.replicas governs the durability/latency trade-off (acks=all = strongest).
  • Retention by time or size; log compaction keeps the latest value per key (useful for feature snapshots).
  • Consumer group offsets live in __consumer_offsets — this is what makes replay and backfill possible.
  • Over-partitioning raises broker memory, replication cost, and rebalance time; under-partitioning caps throughput permanently.

Flink:

  • KPU = 1 vCPU + 4 GB; 1 KPU minimum; scales parallelism of the job graph.
  • Checkpoints to S3 = exactly-once state; checkpoint interval trades recovery time vs overhead.
  • Watermarks define how long to wait for late events before closing a window.

Default quotas are soft and region-dependent (e.g., KDS shards per region, on-demand streams per account, Firehose ~5 MB/s or 5,000 records/s per delivery stream). Exam answers usually target the architecture pattern, not the exact number — but per-shard 1 MB/s write / 2 MB/s read is testable.

3. Delivery Semantics, Ordering & Reprocessing

Concern KDS MSK Firehose Flink
Ordering Per shard only Per partition only None guaranteed Per key (keyed state)
Default delivery At-least-once At-least-once (configurable) At-least-once Exactly-once w/ checkpoints
Replay Yes (retention window) Yes (retention/compaction) No Via source replay
Consumer coordination KCL lease table (DynamoDB) + checkpoints Consumer group + offsets N/A Checkpointed state
Scaling the consumer Add consumers up to shard count; KCL rebalances Add consumers up to partition count N/A Raise parallelism/KPUs

Key rules:

  • Global ordering does not exist in KDS or Kafka. Any answer promising total ordering across a stream is wrong.
  • Deduplication is the consumer's job on at-least-once sources (idempotency key, or Flink exactly-once sinks).
  • Lambda event source mapping: tune batch size, batch window, parallelization factor (per-shard concurrency, breaks strict order), and bisect-on-error to isolate poison records. On-failure destinations (SQS/SNS) prevent silent loss.
  • Checkpointing ≠ replay. Checkpoints resume processing; retention enables reprocessing.
  • Tumbling/hopping/sliding windows → rolling aggregates (avg spend last 5 min) written to an online feature store.
  • Event time vs processing time: event time + watermarks handles out-of-order mobile/IoT data; processing time is simpler but wrong for late arrivals.
  • Stream–stream joins (e.g., clicks ⋈ impressions) with a bounded state TTL — unbounded state is the classic Flink OOM.
  • Stream enrichment: join the live stream against a static/dimension source (S3 or DynamoDB) before writing features.
  • Late data: allowed lateness + side output, or route to a dead-letter path rather than dropping silently.

Gotcha: The narrowest bottleneck sets pipeline throughput. If the sink (OpenSearch, DynamoDB, Redshift) is saturated, adding consumers/KPUs only moves the queue. Diagnose upstream limit vs downstream block before scaling.

5. Landing Zone: Formats, Buffering & ML Targets

  • Columnar for the curated layer. Convert streaming JSON to Parquet/ORC (Firehose record-format conversion using a Glue Data Catalog table) so Athena/Spark/SageMaker training read only needed columns.
  • Small-file problem: high-frequency delivery creates millions of tiny S3 objects → slow crawls and slow training. Mitigate with larger buffer size/interval and periodic compaction.
  • Firehose buffering hints (size/interval) are the main latency-vs-efficiency dial; dynamic partitioning writes Hive-style prefixes for partition pruning.
  • Streaming → SageMaker Feature Store: stream processors write the online store for low-latency inference lookups while the same feature group's offline store accumulates history for training — one definition, no training–serving skew.
  • Streaming → vector store: Flink/Glue generate embeddings and upsert to OpenSearch, pgvector, or S3 vector indexes for retrieval in generative AI apps.
  • Near-real-time retraining loop: stream → S3 (Parquet) → Glue Catalog → SageMaker training job; batch backfill and stream must write the same schema and partition layout.
  • Diverse data types: text/images/audio land in S3 as objects with metadata; streams carry the reference + metadata, not the payload, when payloads exceed the 1 MB record limit.

Gotcha: Preserve event timestamps end-to-end. Using ingestion time as the event time silently corrupts point-in-time feature joins and produces leakage in training sets.

6. Security, Networking & Compliance

Layer Controls
KDS IAM identity policies + resource-based policies; KMS SSE at rest; TLS in transit; VPC interface endpoints (PrivateLink) for private producers/consumers; CloudTrail for control-plane and optional data-plane events
MSK KMS at rest; TLS in transit; IAM access control or SASL/SCRAM; security groups + private subnets; PrivateLink; broker logs to CloudWatch/S3/Firehose
Firehose IAM service role to S3/OpenSearch/Redshift; KMS on the destination bucket; Lambda/OpenSearch sinks can run in-VPC; no public endpoint exposure required
Flink / Glue Streaming Runs inside your VPC with an execution IAM role; KMS for state/checkpoints; CloudWatch Logs for job telemetry
Compliance drivers Residency/AZ placement, retention length (KDS retention vs Kafka retention vs S3 lifecycle), and who may decrypt (KMS key policy) are configured at the storage layer, not in training code

Gotcha: Enabling SSE-KMS on a stream/topic requires producers and consumers to hold kms:Decrypt / kms:GenerateDataKey — an "AccessDenied" after enabling encryption is a permissions problem, not a stream problem. Per-request KMS calls also add cost and latency.

7. Troubleshooting Matrix

Symptom Likely cause Fix
Consumer lag grows linearly, never recovers Incoming rate > consumer capacity or blocked sink Measure sink latency first; scale sink, then consumers
WriteProvisionedThroughputExceeded Shard exceeded 1 MB/s or 1,000 rec/s Add shards, aggregate records (KPL-style batching), or switch to on-demand
One shard hot, others idle Low-cardinality / skewed partition key High-cardinality key (add salt/random suffix)
Multiple consumers each see throttling 2 MB/s per shard shared read Enable enhanced fan-out for dedicated 2 MB/s per consumer
Records out of order downstream Parallel consumers or partitioned write target Partition sink by the same key; serialize per shard if order is required
Kafka consumers idle though lag exists Consumer group size > partition count Add partitions (irreversible) or accept the parallelism cap
Job restarts from scratch after failure Checkpointing not enabled / state backend misconfigured Enable checkpointing to S3; raise checkpoint interval tolerance
Duplicate rows in the lake At-least-once delivery (Firehose, non-idempotent sink) Idempotent upsert, dedupe key, or exactly-once sink semantics
Training set has look-ahead leakage Event time replaced by ingestion time Carry and join on the source event timestamp

8. Anti-Patterns & Distractor Traps

  • Firehose as a queue. No replay, no pull consumers, no backfill. Use KDS/MSK.
  • Scaling consumers into a blocked sink. Moves the queue; does not drain it.
  • Assuming stream-wide ordering. Ordering is per shard/partition only.
  • Raw JSON forever. Row-oriented, uncompressed files inflate every downstream query and training read; convert to columnar in the curated layer.
  • Ignoring the small-file problem. Thousands of tiny S3 objects cripple Glue crawls and training throughput.
  • Unbounded Flink state. Joins/keys without TTL exhaust task memory.
  • Self-managed Kafka on EC2 when MSK fits. Operations burden with no functional gain; MSK Serverless if capacity is unpredictable.
  • Under-partitioned topic + "add consumers." Parallelism is capped by partition count — a permanent ceiling.
  • One feature definition per environment. Divergent train/serve logic is the root cause of training–serving skew; centralize in Feature Store.
  • Streaming everything. If the SLA is hourly or daily, micro-batch/Glue/Step Functions is cheaper and simpler than always-on Flink KPUs.

Cost rule of thumb: KDS charges per shard-hour + per PUT payload unit (so batching/aggregation directly cuts cost); MSK charges per broker-hour + storage (so right-sizing brokers and partitions matters); Flink charges per KPU-hour (so idle always-on jobs are the most common cost mistake).