Skill 1.1.2: Make storage decisions and configure storage services based on cost, performance, data structure, and data compliance
Task 1.1: Data Preparation for ML and AI — Collect and store data
Objective: Land data where training jobs and AI applications read it efficiently — selecting service, tier, and format to satisfy four competing axes: cost, performance, data structure, compliance.
The axes pull against each other: lowest $/GB (Glacier, S3-IA, EFS-IA) = highest latency + retrieval fees; lowest latency (EBS io2, DynamoDB, S3 Express One Zone) = highest cost; strictest compliance (Object Lock, residency, CMK) constrains region, tier, and format — enforce compliance in storage config, not in training code.
Service selection by data structure & access pattern
| Service | Data model | Access pattern | Multi-node | Primary ML role |
|---|---|---|---|---|
| S3 | Object | HTTP GET/PUT, high throughput, higher per-op latency | Yes (unlimited) | Default landing zone; training source; vector buckets |
| S3 Express One Zone | Object (directory bucket) | Single-digit-ms, single AZ | Yes | Latency-sensitive checkpoints / high-IOPS training |
| EBS | Block | Raw device, ONE instance (single AZ) | No* | Instance-local scratch, high IOPS/throughput |
| EFS | File (POSIX/NFS) | Shared namespace, thousands of clients | Yes | Distributed training needing one file namespace |
| FSx for Lustre | Parallel file | Sub-ms, hundreds of GB/s | Yes | HPC/high-throughput training; S3-linked |
| FSx for NetApp ONTAP / Windows | File | SMB/NFS, multi-protocol | Yes | Migrated enterprise file workloads |
| RDS / Aurora | Relational | SQL, ACID, joins | Read replicas | Structured queryable records; pgvector |
| DynamoDB | Key-value / document | Single-digit-ms key lookup | Yes (managed) | Feature lookup, high-QPS metadata |
| OpenSearch Service | Search index | Full-text + k-NN vector | Yes | Vector store for RAG / semantic search |
| SageMaker Feature Store | Feature groups | Online (ms) + offline (batch) | Yes | Shared train/serve feature definitions |
Gotcha: EBS attaches to one instance at a time, in one AZ. Multi-attach (io1/io2) requires a cluster-aware file system — never the answer for "several instances need the same files." That's EFS or FSx.
"X vs Y" decision matrix — common distractors
| Requirement | Choose | Reject |
|---|---|---|
| Many instances, one shared file namespace | EFS / FSx | EBS (single attach) |
| Lowest-latency local disk, single node | EBS gp3 / io2 | S3 (per-op latency) |
| Cheapest durable landing zone at any scale | S3 | EFS / EBS (per-GB cost) |
| Sub-ms shared HPC throughput | FSx for Lustre | EFS Standard |
| Structured records with joins/transactions | RDS / Aurora | DynamoDB (no joins) |
| Predictable single-digit-ms lookup at scale | DynamoDB | RDS (connection limits) |
| Similarity search over embeddings | OpenSearch / pgvector / S3 vectors | Feature Store |
| One feature definition for train + inference | Feature Store | Any general-purpose DB |
Vector store selection
| Specification | Choose | Trade-off |
|---|---|---|
| Embeddings already beside relational rows in PostgreSQL | RDS/Aurora PostgreSQL + pgvector | Zero data movement; scales with DB instance |
| Large-scale ANN, metadata filtering, hybrid text+vector | OpenSearch Service (k-NN) | Highest capability and cost |
| Massive vector sets, infrequent queries, cost-first | S3 vector buckets / indexes | Cheapest, higher query latency |
Feature Store: online store = low-latency latest-value lookup for inference; offline store = full history + point-in-time-correct joins for training. One feature definition across both eliminates training-serving skew.
Data format decision
| Format | Layout | Best for | Avoid when |
|---|---|---|---|
| Parquet / ORC | Columnar, compressed | Analytic reads touching few columns across many rows; curated/refined layers | Record-at-a-time appends |
| JSON / CSV | Row / document, text | Simple interchange, raw ingestion, streaming writes | Repeated large analytic scans |
| Avro | Row, schema in header | Streaming writes + schema evolution | Column-pruned analytics |
| .zip / .tar.gz archives | Opaque blob | Never for query | Always — full download + decompress required |
Gotcha: Columnar = column pruning / projection pushdown → engine reads only required columns → less I/O, faster, cheaper. This is why the curated layer is columnar even when the raw layer is not.
Gotcha: Small-files problem — thousands of tiny S3 objects crush Athena/Spark (per-object listing + open overhead). Fix with compaction, ~128 MB–1 GB target files, and partitioning on query predicates.
Cost levers
| Service | Levers |
|---|---|
| S3 | Storage class + Lifecycle policies; Intelligent-Tiering; version cleanup; request/retrieval costs |
| EBS | gp3 (baseline 3,000 IOPS / 125 MB/s, independently scalable) vs io2 Block Express vs st1/sc1; snapshot lifecycle |
| EFS | Standard vs Infrequent Access vs One Zone; throughput mode Elastic / Provisioned / Bursting |
| FSx for Lustre | Scratch (no replication, cheapest, temporary) vs Persistent (replicated, long-lived) |
| RDS | Instance class; Multi-AZ costs 2×; Reserved Instances; storage autoscaling |
| DynamoDB | On-demand vs provisioned + auto scaling; Standard-IA table class; TTL |
S3 classes — fastest to cheapest: Standard → Intelligent-Tiering → Standard-IA → One Zone-IA → Glacier Instant Retrieval → Glacier Flexible (min–hrs) → Deep Archive (12–48 hrs)
Gotcha: Glacier classes add retrieval latency + retrieval cost + minimum storage duration (30/90/180 days). Never correct for an actively read training dataset.
Performance levers
- S3 throughput: prefix partitioning for parallel GETs; multipart upload for large objects; Transfer Acceleration for cross-region ingest; byte-range fetches; SageMaker Pipe mode / FastFile to stream rather than download.
- EFS: Elastic throughput auto-scales with workload — default unless sustained and predictable.
- FSx for Lustre: link to S3 for lazy load and write-back of results.
- EBS gp3: scale IOPS and throughput independently of capacity — resize without adding GB.
Security, compliance, governance
| Control | Applies to |
|---|---|
| IAM roles (not keys) + least privilege; bucket policies; S3 Access Points | All stores |
| At rest: SSE-S3, SSE-KMS (CMK), DSSE-KMS, SSE-C; EBS/EFS/FSx/RDS KMS; DynamoDB AWS-owned or CMK | Configure at creation |
| In transit: TLS/HTTPS endpoints; EFS/FSx in-transit encryption toggle | All stores |
| VPC endpoints: Gateway for S3 + DynamoDB; Interface (PrivateLink) for EFS, FSx, RDS, KMS, SageMaker | Keeps traffic off the public internet |
| S3 Block Public Access, Object Ownership / bucket-owner enforced | Account + bucket level |
| S3 Object Lock (WORM) + Versioning | Retention / immutability mandates |
| AWS Lake Formation | Fine-grained table/column/row access over the Glue Data Catalog |
| Amazon Macie | PII discovery and classification in S3 |
| SageMaker network isolation + VPC config + KMS volume key | Training / processing jobs |
Gotcha: Residency, retention, and read-access rules are satisfied by region choice, Object Lock, KMS key policy, and Lake Formation grants — not by training code.
Gotcha: Gateway endpoints support S3 and DynamoDB only. Everything else needs an Interface endpoint.
Anti-patterns (high-frequency exam traps)
- EBS for shared multi-node training data → use EFS/FSx.
- CSV/JSON in the curated layer → full-column scans; use Parquet/ORC.
- Glacier/Deep Archive for an actively read training set → retrieval latency + minimum-duration charges.
- Millions of tiny S3 objects → compaction + partitioning.
- Embeddings in an RDBMS without a vector extension, or in S3 when sub-100 ms ANN is required.
- Confusing Feature Store with a vector store → identity/latest value vs similarity.
- Training on one feature definition, serving another → training-serving skew; fix with Feature Store.
- Creating an unencrypted volume then wanting KMS → cannot retrofit in place; create a new encrypted volume from a snapshot.
- Assuming EFS is always multi-AZ → One Zone classes are single-AZ and do not survive AZ loss.