Skill 1.1.8: Ingest and Store Diverse Data Types (Text, Images, Audio) for AI/ML
Task 1.1: Data Preparation for ML and AI – Collect and store data
Core objective: Land heterogeneous, mostly unstructured data from producers into a store a training job, endpoint, or RAG pipeline can read efficiently — at the right cost, under the right compliance controls.
Mental model: Source → Ingestion path → Landing zone (raw, immutable) → Curated store (format-optimized) → Consumer (training / endpoint / vector store)
- Modality drives format; access pattern drives store. Never default to whatever the producer emitted.
- Unstructured ≠ unmanaged. Every image/audio object still needs metadata, partitioning, and lineage to be findable.
- Raw zone stays as-ingested (replayable); curated zone is re-formatted (columnar, compressed, partitioned).
Modality → Format → Landing Zone
| Modality | Raw format | Curated / training format | Store & notes |
|---|---|---|---|
| Text / NLP | JSON, JSONL, TXT, HTML | Sharded Parquet / JSONL | S3, prefix by source+date |
| Images | JPEG, PNG, WebP, TIFF | RecordIO / TFRecord / WebDataset shards, or Parquet (URI + label) | S3 — avoid millions of 20 KB objects |
| Audio | WAV, FLAC (lossless); MP3, OPUS (lossy) | Mel-spectrogram arrays in Parquet/NumPy shards | S3 for datasets; Kinesis Video Streams for live |
| Video | MP4 (H.264/H.265) | Frame extracts + clip-offset manifest | S3 + Kinesis Video Streams (fragments, HLS/RTSP) |
| Tabular | CSV, JSON | Parquet / ORC (columnar, splittable, predicate pushdown) | S3 + Glue Data Catalog; RDS/DynamoDB as source |
| Embeddings / vectors | float32 / float16 arrays | Native vector index | OpenSearch, RDS/Aurora pgvector, S3 Vectors |
| Features | Event records | Offline: Parquet in S3 | SageMaker Feature Store (online + offline) |
| Telemetry / logs | JSON records | Parquet partitioned by time | Kinesis Data Streams → Firehose → S3 |
Gotcha: Columnar (Parquet/ORC) wins for analytic reads across few columns; row-oriented (JSON/CSV) wins for record-at-a-time writes and interchange. The curated layer is almost always columnar even when raw is not.
Storage Service Decision Matrix
| Service | Data shape | Concurrent readers | Use when |
|---|---|---|---|
| S3 | Any object, unlimited scale | Massive, parallel | Default AI/ML data lake, media, model artifacts |
| S3 Express One Zone | Objects, single AZ | Massive, parallel | Ultra-low-latency training data / checkpoints |
| Amazon EBS | Block device, one instance, one AZ | 1 (io1/io2 multi-attach ≤16, same AZ) | Local scratch, single-node training, high-IOPS DB |
| Amazon EFS | POSIX file system, multi-AZ | Thousands of instances | Shared namespace for distributed jobs |
| FSx for Lustre | HPC parallel FS | Thousands, very high throughput | Large-scale distributed training; S3-linked |
| FSx for Windows/ONTAP/OpenZFS | SMB / NFS / ZFS | Many | Windows workloads, NetApp/NFS migration |
| RDS / Aurora | Relational rows | Many (connection-bound) | Structured records; pgvector for embeddings in-place |
| DynamoDB | Key-value / document | Massive, single-digit ms | Metadata index, online lookups, session state |
| OpenSearch Service | Documents + k-NN vectors | Many | Semantic search, RAG retrieval, log analytics |
| SageMaker Feature Store | Features by entity+timestamp | Online: low latency / Offline: batch | Eliminate training-serving skew |
| Kinesis Data Streams / MSK | Ordered record stream | Multiple consumers | Continuous ingestion, real-time feature pipelines |
Gotcha: EBS is single-instance, single-AZ — it can never back a multi-node distributed training job. Shared namespace ⇒ EFS or FSx for Lustre.
Ingestion Paths
| Path | Services | Notes |
|---|---|---|
| Batch / file | S3 multipart upload, DataSync, Transfer Family (SFTP/FTPS/FTP) | Multipart for >100 MB; parallelize |
| Streaming | Kinesis Data Streams, MSK, Managed Flink, Kinesis Video Streams | Flink = stateful processing; KVS = media-specific |
| Delivery to lake | Kinesis Data Firehose | Buffering hints + format conversion to Parquet/ORC via Glue catalog; dynamic partitioning |
| DB extraction | Glue JDBC crawlers + Spark ETL, DMS | Glue Data Catalog holds discovered schemas |
| Bulk migration | Snowball Edge / Snow Family, Direct Connect | Terabytes–petabytes, poor connectivity |
| Merge / join | Glue ETL (Spark), EMR, Athena CTAS | Resolve keys, timezones, nullability before training |
Gotcha: Adding consumer capacity to a pipeline whose downstream write target is saturated moves the queue, not the backlog. Fix the bottleneck, not the consumers.
S3 Layout & Performance Rules
- Partition by access pattern (
source=/date=/hour=) — enables partition pruning; never one giant prefix. - Compact small files (target 128 MB–1 GB Parquet) via Glue/EMR compaction or Firehose.
- Multipart upload for objects >100 MB: ≤10,000 parts, 5 MB–5 GB each.
- Lifecycle policies transition raw → IA → Glacier Instant Retrieval → Deep Archive; expire junk.
- Storage classes: Standard (hot), Intelligent-Tiering (unknown/changing), Standard-IA, One Zone-IA, Glacier IR/Flexible/Deep Archive, S3 Express One Zone.
- Checksums (CRC32/CRC64NVME/SHA-256) for integrity; S3 Inventory + Batch Operations for at-scale audit/dedup.
- Metadata: object metadata/tags + sidecar JSONL manifests + Glue Data Catalog for traceability to source.
SageMaker Training Data Access Modes
| Mode | Behavior | Best for |
|---|---|---|
| File | Downloads full dataset to EBS before training | Small datasets, random access, many epochs |
| Pipe | Streams from S3 on the fly | Large datasets; requires RecordIO-Protobuf/WebDataset or Augmented Manifest |
| FastFile | POSIX-like on-demand streaming, random access | Large Parquet/CSV, no conversion needed |
Gotcha: Pipe mode requires RecordIO or augmented manifest files — plain JPEG/CSV fails. Augmented manifests also carry per-object labels, removing a separate label file.
Purpose-Built Stores
SageMaker Feature Store — one definition, two stores:
- Online store: low-latency lookup for inference; optional TTL.
- Offline store: S3 + Glue catalog, Parquet, supports time-travel / point-in-time joins to prevent label leakage.
- Requires a record identifier and an event time feature; supports batch and streaming ingestion.
Vector stores:
| Option | Strength | Trade-off |
|---|---|---|
| OpenSearch Service | Managed k-NN (HNSW/FAISS), hybrid keyword+vector, filters | Highest cost/ops |
| RDS/Aurora PostgreSQL + pgvector | Embeddings beside relational rows, SQL joins, transactions | Scaling limits, index build cost |
| S3 Vectors | Cheapest at scale, object-storage durability | Higher query latency — large, infrequently queried sets |
Gotcha: A vector store answers "what is similar?"; a feature store answers "what are this entity's values now / at time T?" Confusing the two is the classic distractor.
Security & Compliance
- At rest: SSE-S3 (default), SSE-KMS (audit + key control), DSSE-KMS (double layer), SSE-C. EBS/EFS/FSx/RDS/DynamoDB/OpenSearch are all KMS-integrated.
- In transit: TLS everywhere; enforce
aws:SecureTransportin bucket policies. - Network: S3 Gateway VPC endpoint (no NAT/IGW cost); Interface endpoints (PrivateLink) for Kinesis, SageMaker, Glue, OpenSearch, Feature Store.
- Access: IAM roles (never long-lived keys) + bucket policies + Lake Formation for table/column/LF-Tag control.
- Discovery: Macie for PII in S3; Comprehend for PII in text.
- Retention / WORM: S3 Object Lock (governance/compliance mode) + Glacier Vault Lock.
- Guardrails: Block Public Access at account + bucket level; deny-by-default.
Anti-Patterns & Distractor Traps
- Millions of tiny files in one prefix → request throttling + slow training. Compact and partition.
- CSV/JSON in the curated layer for analytic reads → full scans; use Parquet/ORC.
- EBS for multi-node training → impossible; use EFS or FSx for Lustre.
- Large binaries in RDS/DynamoDB → store an S3 URI instead.
- DynamoDB/EFS as a "vector database" → no native similarity search (pgvector is the relational exception).
- No lifecycle policy on raw media → runaway storage cost.
- Public or unencrypted buckets → instant compliance failure.
- Separate training and inference feature pipelines → drift; use one Feature Store definition.
- FSx for Lustre scratch for durable data → not replicated; use persistent, or keep the source in S3.
Hard rules to memorize: S3 object ≤5 TB, single PUT ≤5 GB, multipart ≤10,000 parts. Kinesis Data Streams record ≤1 MB, 1 MB/s in + 2 MB/s out per shard, retention 24 h default → 365 d max. EFS = multi-AZ shared POSIX; EBS = one AZ. FSx for Lustre links an S3 bucket as its repository.