Skip to content

Skill 1.1.1: Extract data from data sources (for example, Amazon S3, Amazon EBS, Amazon EFS, Amazon RDS, Amazon DynamoDB, Amazon OpenSearch Service)

Task 1.1: Data Preparation for ML and AI – Collect and store data

Objective: Move data out of the system that produces it and land it where a training job or inference path can read it efficiently. Source choice is driven by structure, access pattern, cost, and compliance — never by "what the app already uses."

Source Selection Matrix (Extraction → Store)

Source Data model Native extraction pattern Best ML role
Amazon S3 Objects Prefix/byte-range GET, multipart, S3 Select Default landing zone + training source
Amazon EBS Block Attached block device, snapshots Local disk for a single instance
Amazon EFS File (NFS) POSIX mount on many nodes Shared namespace for distributed jobs
Amazon FSx for Lustre File (HPC) POSIX mount, S3-linked repo High-throughput training reads
Amazon RDS / Aurora Relational SQL, read replicas, snapshot export Structured/queryable records
Amazon DynamoDB Key-value / document Query, parallel Scan, Export to S3 Single-digit-ms key lookups
Amazon OpenSearch Search index Bulk ingest, PIT + search_after Full-text and vector kNN
Kinesis / MSK / Managed Flink Stream Shards/partitions, consumer groups Continuous ingestion
AWS Glue / Amazon EMR Catalog + compute Crawlers, Spark jobs Merge, transform, catalog
SageMaker Feature Store Features Online + offline store Training/serving consistency

EBS attaches to one instance at a time (limited io1/io2 multi-attach is clustered-FS only). A distributed training job needing one shared file namespace → EFS or FSx for Lustre. This is a top distractor.

Per-Source Extraction Nuances

Amazon S3

  • Parallel read – SageMaker and EMR split by prefix; many medium objects beat one giant object.
  • Small-file anti-pattern – thousands of KB-sized objects inflate per-object request overhead and slow LIST; compact with Glue/EMR (target ~128 MB–1 GB).
  • Server-side filtering – S3 Select pushes projection/filter down so less data crosses the network.
  • Cross-region/global uploads – Transfer Acceleration; per-team isolation via Access Points.

Amazon EBS

  • gp3 decouples IOPS/throughput from capacity (baseline 3,000 IOPS / 125 MB/s, independently scalable); io2 Block Express for extreme IOPS.
  • Snapshots are S3-backed and incremental → fast volume/AMI rebuild for training nodes.
  • Anti-pattern: treating EBS as a shared dataset for multi-node training.

Amazon EFS

  • Throughput modes: Bursting (credit-based — small file systems can starve), Provisioned (fixed), Elastic (auto-scaling, default).
  • Performance modes: General Purpose vs Max I/O (higher aggregate parallelism, higher latency).
  • Storage classes: Standard / IA / Archive with lifecycle policies; metadata-heavy workloads are the classic bottleneck.

Amazon RDS / Aurora

  • Read replicas absorb extraction load. Multi-AZ is failover, not read scaling.
  • Snapshot export to S3 in Parquet is the preferred bulk path for ML — it avoids querying the live writer entirely.
  • Aurora: parallel query, zero-ETL to Redshift, native SageMaker/Bedrock integration.
  • Anti-pattern: running large training extracts against a production writer.

Amazon DynamoDB

  • Query on partition key + sort key; Scan reads the whole table and burns RCUs.
  • Parallel Scan (TotalSegments) for bulk reads; Export to S3 (PITR-based) is better — no RCU consumption, no impact on live traffic.
  • DAX is for microsecond inference lookups, not bulk training export.
  • Fine-grained IAM via dynamodb:LeadingKeys conditions.

Amazon OpenSearch Service

  • Bulk API for ingestion; PIT + search_after for large sequential extracts — not scroll for real-time paging.
  • kNN: HNSW = approximate/fast (recall vs latency trade-off); brute force = exact/slow. Distance metrics: cosine, L2, inner product.

Streaming (Kinesis / MSK / Managed Flink)

  • Kinesis Data Streams: ordering guaranteed per shard only; partition key determines shard. Provisioned vs on-demand mode; enhanced fan-out gives each consumer dedicated read throughput; retention 24 h → 365 d.
  • Amazon MSK: topics/partitions/consumer groups for Kafka-standardized teams.
  • Managed Service for Apache Flink: stateful processing, exactly-once, Kinesis/MSK connectors.
  • Firehose: no-code near-real-time delivery into S3.

File Formats — Access-Pattern Driven

Format Layout Choose when Trap
Parquet Columnar Analytics/ML reads touching few columns across many rows No benefit for full-row or single-record reads
ORC Columnar Hive/EMR-centric stacks Same as above
Avro Row Streaming, schema evolution Poor for column-selective analytics
JSON Row Semi-structured interchange Verbose, no type enforcement
CSV Row Simple interchange No schema, no types, slow

Columnar only pays off when the query is column-selective. It enables column pruning/predicate pushdown, so the engine reads fewer bytes — that's the cost win. Row formats still win for record-at-a-time writes and full-record reads.

Curated layer rule: raw layer may be any format; the curated/serving layer is written columnar with compression (Snappy/ZSTD) even when the raw layer is not.

Merging Multi-Source Data

  • Catalog first – Glue Crawler → Glue Data Catalog so later jobs find the table definition instead of rediscovering the schema.
  • Join engines – Glue ETL (serverless Spark) for moderate scale; Amazon EMR when cluster scale is required; Athena for query-in-place.
  • Reconciliation is the hard half – normalize keys, time zones, nullability, and declare which source is authoritative per field. Survivorship rules are an engineering decision you document, not something a crawler infers.
  • Partition (year/month/day) and compact to keep downstream reads cheap.

Vector Stores vs Feature Stores

Vector store Feature store
Answers "What is similar to this embedding?" "What are this entity's features now, and at event time?"
Options OpenSearch kNN; RDS/Aurora PostgreSQL + pgvector; S3 Vectors SageMaker Feature Store
Selection drivers Scale, latency, hybrid filter+vector, data locality, cost Training-serving skew elimination
Key detail pgvector avoids moving embeddings already sitting beside relational rows; S3 Vectors suits huge, infrequently queried sets Online store = low-latency latest value; Offline store = S3 history with point-in-time (EventTime) correctness

Feature store ≠ vector store. One is identity + time-travel; the other is similarity. Sharing one feature definition across training and inference is what prevents training–serving skew.

Security, Networking & Compliance

  • At rest: SSE-S3 / SSE-KMS (CMK, key policy, grants) / SSE-C on S3; KMS-backed encryption on EBS, EFS, RDS, DynamoDB, OpenSearch.
  • In transit: TLS everywhere; EFS TLS mount option; enforce SSL on RDS.
  • Network isolation: Gateway VPC endpoints for S3 and DynamoDB; Interface endpoints (PrivateLink) for EFS, RDS, OpenSearch, Glue, Kinesis — keeps ingestion off the public internet.
  • Least privilege: execution roles per job, S3 Access Points, Lake Formation for table/column-level grants, DynamoDB leading-key conditions.
  • Compliance constraints land in storage config, not training code: S3 Object Lock (WORM), retention periods, Macie for PII discovery, cross-region replication, data-residency rules that can dictate both format and location.

Troubleshooting: Symptom → Cause → Fix

Symptom Likely cause Correct fix
Batch job slows as dataset grows Reading more data than needed Partition + columnar + predicate pushdown
Slow listings, long job startup Small-file problem Compact objects to 128 MB–1 GB
Consumer lag grows, sink at throughput limit Downstream is the binding constraint Scale/optimize the sink — adding consumers only moves the queue
Consumer lag grows, sink healthy Under-provisioned consumers or hot shard Add shards/partitions, scale consumers, reshard
EFS throughput collapses on a small FS Bursting credits exhausted Switch to Elastic or Provisioned throughput
DynamoDB table throttles during export Full-table Scan consuming RCUs Query, parallel Scan, or Export to S3
Prod RDS performance degrades during extraction Reading from the writer Read replica, or snapshot export to S3

A pipeline blocked by a slow write target does not improve by adding consumer capacity. Diagnose throughput limits and downstream slowdowns separately — they look identical on a dashboard but need opposite fixes.

Anti-Patterns to Reject

  • EBS as a shared dataset for multi-node training.
  • Multi-AZ RDS treated as read scaling (it is failover).
  • DynamoDB Scan for training-data export.
  • scroll for large sequential OpenSearch extracts instead of PIT + search_after.
  • Uncompressed CSV in a repeatedly queried curated layer.
  • Training extracts against a production database writer.
  • Merging sources without a declared authoritative source — silently corrupts labels.
  • Assuming Kinesis ordering across shards — it is per-shard only.