Skip to content

Skill 1.2.3: Transform streaming data (for example, by using AWS Lambda, Spark)

Task 1.2: Data Preparation for ML and AI - Perform data transformation, feature engineering, and pre-processing.

Core Objective & Mental Model

  • Streaming transformation applies the same feature engineering/preprocessing logic as batch, but continuously and with low latency.
  • Primary decision: match the tool to the window shape and state requirement—not just throughput.
  • Stateless vs. stateful:
  • Per-record transforms (parse, mask, encode, scale with fixed training parameters) → event-driven compute.
  • Multi-record transforms (rolling averages, sessionization, joins, dedup) → stateful stream processor.
  • Train/serve consistency: transforms fitted on training data must be applied, not refitted, in the stream. Persist parameters in SageMaker Feature Store, S3, or model artifact.

Service Selection Matrix

Service Best For State / Window Key Config Anti-Pattern
AWS Lambda Per-record or micro-batch stateless transforms; simple tumbling aggregates Stateless; tumbling windows (fixed, non-overlapping) Batch size, batching window, parallelization factor, retries, DLQ, bisect, tumbling window Long-running sliding windows, event-time ordering, state outliving fixed interval
Spark Structured Streaming on Amazon EMR Complex transforms, sliding windows, stream-stream joins, custom ML libs Stateful; event-time watermarks; sliding/tumbling Checkpoints, watermarks, output mode, triggers Overkill for simple per-record parsing
AWS Glue Streaming Serverless Spark micro-batch ETL from Kinesis/MSK/Kafka Stateful; micro-batch Checkpointing, windowing, job bookmarks Sub-second latency requirements
Managed Service for Apache Flink True event-time processing, exactly-once, complex state, sub-second latency Stateful; event-time, watermarks, exactly-once Checkpoints, savepoints, parallelism Simple transforms where Lambda suffices
Kinesis Data Firehose Delivery to S3/Redshift/OpenSearch with light Lambda transform Stateless; no complex aggregation Buffer size/interval, Lambda transform Stateful aggregation, joins
Amazon MSK Kafka-based streaming source for Spark/Flink/Glue/Lambda Depends on consumer Topics, partitions, consumer groups Using Lambda for high-throughput stateful Kafka processing

Lambda Streaming Deep Dive

  • Event sources: Kinesis Data Streams, DynamoDB Streams, MSK. Lambda polls, batches, and invokes.
  • Key event source mapping parameters:
  • BatchSize + MaximumBatchingWindowInSeconds → tune latency vs. efficiency.
  • ParallelizationFactor → multiple batches per shard concurrently; breaks strict ordering.
  • StartingPosition → LATEST, TRIM_HORIZON, AT_TIMESTAMP.
  • DestinationConfig → on-failure DLQ/SNS/SQS.
  • MaximumRetryAttempts, MaximumRecordAgeInSeconds, BisectBatchOnFunctionError → poison record handling.
  • TumblingWindowInSeconds → fixed, non-overlapping window up to 15 minutes.
  • Concurrency: reserved concurrency for throttling control; provisioned concurrency for cold-start latency.
  • Exactly-once: Lambda is at-least-once; duplicates occur on retries. Use idempotent writes and dedup keys.
  • Limits: 15-minute timeout, 10 GB memory, 10 GB /tmp, 6 MB synchronous payload.

Exam Gotcha: Lambda event source mapping for Kinesis/DynamoDB is at-least-once. Duplicates are expected. Design idempotent sinks.

Exam Gotcha: Lambda tumbling windows are fixed and non-overlapping. For continuously sliding windows, event-time ordering across late data, or long-lived state, use Spark Structured Streaming or Managed Service for Apache Flink.

Spark Streaming Deep Dive

  • Spark Structured Streaming:
  • Micro-batch or continuous processing.
  • Checkpointing required for fault tolerance and exactly-once state recovery; store in S3.
  • Watermarking handles late data and bounds state growth. Without watermark, state grows unbounded.
  • Output modes: Append, Update, Complete—choose based on aggregation and sink.
  • Triggers: processing time, once, continuous.
  • Exactly-once achieved with checkpointing + idempotent/replayable sinks (Delta Lake, Iceberg, transactional DB).
  • AWS Glue Streaming:
  • Serverless Spark; auto-scaling; reads Kinesis/MSK/Kafka; writes S3/Redshift/etc.
  • Use for ETL without cluster management; micro-batch latency.
  • Amazon EMR:
  • Managed Spark cluster; full control; custom libraries; long-running.
  • Use for complex ML feature engineering at scale.
  • Managed Service for Apache Flink:
  • True stream processing; event-time, watermarks, exactly-once, stateful.
  • Use when sub-second latency and complex event processing are required.

Feature Engineering in Streams

  • Stateless transforms: parsing, masking, redaction, encoding, scaling with training-fitted parameters, log transform. Safe in Lambda.
  • Stateful transforms: rolling averages, sessionization, counts, joins with reference data, dedup. Need state store: Spark/Flink state or external store (ElastiCache, DynamoDB).
  • Train/serve consistency: do not refit scalers/encoders in stream. Persist parameters from training. Apply same transform.
  • Online feature store: SageMaker Feature Store online store for low-latency feature serving. Streaming jobs can write features to online store.
  • Late/out-of-order data: use event-time processing and watermarks. Lambda is processing-time; Spark/Flink support event-time.
  • PII: mask/redact before landing. Use Amazon Comprehend/Macie for detection. KMS encryption.

Security & Compliance

  • IAM: least privilege execution roles. Lambda role needs Kinesis/MSK read, S3 write, KMS decrypt. EMR role needs S3, Glue Catalog, etc.
  • Encryption: Kinesis/MSK/S3/KMS at rest. TLS in transit. Lambda env vars encrypted with KMS.
  • Network: VPC endpoints for Kinesis, MSK, S3, Secrets Manager. Lambda in VPC for private resources (RDS, ElastiCache). NAT for internet.
  • Audit: CloudTrail, CloudWatch Logs, X-Ray for tracing.
  • Secrets: Secrets Manager/Parameter Store, not env vars.

Anti-Patterns & Traps

  • Lambda for sliding windows: fails or requires external state. Use Spark/Flink.
  • Refitting scaler in stream: causes training-serving skew. Apply saved parameters.
  • Non-idempotent writes: duplicates from at-least-once. Use dedup.
  • Ignoring late data: no watermark leads to incorrect aggregates or unbounded state.
  • Too small batch: high overhead, throttling. Too large: timeouts. Tune batch size + batching window.
  • No DLQ: poison records block shard. Configure on-failure destination, max retries, bisect.
  • Processing PII without masking: compliance risk. Redact before storage.
  • Over-provisioning EMR for simple transforms: Lambda or Glue may be cheaper.

Quick Decision Tree

  • Per-record stateless transform? → Lambda.
  • Micro-batch ETL with serverless Spark? → Glue Streaming.
  • Sliding windows, joins, stateful aggregation? → Spark Structured Streaming (EMR/Glue) or Managed Service for Apache Flink.
  • Sub-second latency, event-time, exactly-once? → Managed Service for Apache Flink.
  • Simple delivery to S3/Redshift/OpenSearch? → Kinesis Data Firehose (with Lambda transform for light transforms).
  • Kafka source? → MSK + Spark/Flink/Glue/Lambda.