Skip to content

Task 1.1: Ingest and Store Data – MLA-C01 Study Notes

Overview

This task focuses on the early stages of the ML engineering lifecycle (collect → store → prepare for processing). You must select the right storage service and data format, ingest from batch or streaming sources, extract/merge data efficiently, and make cost/performance trade-offs.

Key decision questions you will see on the exam:

  • Is the data structured / semi-structured / unstructured?
  • Batch or streaming?
  • Access pattern (immediate, infrequent, archival)?
  • Need for SQL, schema evolution, or direct SageMaker integration?
  • Operational overhead vs. cost?

Amazon S3 is almost always the default landing zone for raw/unprocessed ML data because of its durability, integration with Glue/EMR/SageMaker/Athena, and lifecycle policies. EFS/FSx appear only when shared POSIX file-system semantics are required.

Core Knowledge Map (Exam-Relevant)

Category Must-Know Items Typical Exam Keywords
Formats CSV/JSON (row), Parquet/ORC/Avro (columnar), RecordIO (SageMaker) “schema evolution”, “splittable”, “compression”, “S3 Select compatible”
Storage S3 (Standard, Intelligent-Tiering, Glacier Instant/Deep Archive), EFS, FSx for ONTAP, EBS “data lake”, “centralized repository”, “immediate access”, “12-hour retrieval”
Ingestion S3 Transfer Acceleration, Kinesis Data Streams/Firehose, MSK, DMS → S3, Glue, EMR “real-time”, “GZIP”, “CDC”, “Pipe mode”
Extraction / Merge S3 Select, Athena, Glue ETL/Spark, SageMaker Data Wrangler & Feature Store “reduce latency/cost”, “join feature groups”, “Spark connector”
Cost/Perf Lifecycle rules, storage classes, Provisioned IOPS, capacity troubleshooting “cost-effective”, “operational overhead”, “scalability”

Hands-on Exercises (Minimized Cost)

All exercises use tiny datasets (≤ 5 MB), the cheapest possible configurations, and explicit cleanup steps. Expected total cost if you follow the timings: < $1 (usually a few cents). Run everything in one region (e.g., us-east-1). Delete resources the same day.

Exercise 1: Cost-Effective S3 Data Lake + Lifecycle for ML Pipeline

Goal: Practice storage-class decisions, lifecycle policies, and S3 as the centralized ML repository (directly maps to the “unprocessed IoT data / 6-month vs 6-year” exam scenarios).

Steps

  1. Create a bucket ml-ingest-lab-<your-initials> (Block Public Access ON).
  2. Create three prefixes: raw/, processed/, archive/.
  3. Upload a 1–2 MB CSV (any public dataset or generate with Python) to raw/data.csv.
  4. In the bucket Management → Lifecycle rules:
  5. Rule “processed-to-glacier”: prefix processed/, transition to S3 Glacier Flexible Retrieval after 30 days (simulates the 6-month rule; you can set 1 day for lab).
  6. Rule “raw-to-deep-archive”: prefix raw/, transition to S3 Glacier Deep Archive after 1 day.
  7. (Optional, still free) Enable S3 Inventory or simply note the storage class of the object.
  8. Cleanup: empty and delete the bucket.

Expected Result

  • Objects appear in the correct storage class after the transition period (or immediately if you force with a 0/1-day rule for testing).
  • You can still use aws s3 cp or the console to retrieve; Deep Archive shows “restore required”.

Exam Traps & Tips Related to This Exercise

  • Trap: Choosing S3 One Zone-IA when the question says “highly available” or “centralized repository”.
  • Tip: “Unprocessed / data lake / pipeline” → S3 Standard + lifecycle. “Immediate access for 6 months, then archive” → Standard → Glacier.
  • Trap: Suggesting EFS for raw IoT data (EFS is file-system, higher cost, no native Glue/Athena integration).
  • Cost reminder: Glacier transitions are free; retrieval later costs money—never restore large objects in the lab.

Exercise 2: Format Selection + Conversion (CSV → Parquet) + S3 Select / Athena

Goal: Choose columnar formats for access patterns, convert with minimal Glue usage, and extract subsets cheaply (S3 Select / Athena). Covers “Parquet for faster queries”, “S3 Select works on CSV/JSON/Parquet”, and schema-evolution talking points.

Steps

  1. Keep the same bucket. Upload a small CSV (raw/sales.csv – 10–20 columns, < 2 MB).
  2. Create a Glue crawler (IAM role with S3 read + Glue service role):
  3. Source = s3://…/raw/
  4. Run crawler once (takes ~1 min, negligible cost).
  5. Create a tiny Glue ETL job (Spark, Glue 4.0, 2 DPUs, worker type G.1X):
  6. Script skeleton (edit in console): read CSV → DynamicFrame → apply mapping → write Parquet to s3://…/processed/.
  7. Job parameters: --enable-job-insights false, timeout 10 min.
  8. Run once (usually finishes in 2–4 min → ~$0.03–0.06).
  9. Confirm Parquet files land under processed/.

  10. S3 Select test (CLI or console “Query with S3 Select”):
    bash aws s3api select-object-content --bucket ml-ingest-lab-… \ --key processed/part-00000.parquet \ --expression "SELECT col1, col2 FROM S3Object LIMIT 5" \ --expression-type SQL \ --input-serialization '{"Parquet":{}}' \ --output-serialization '{"CSV":{}}' /dev/stdout

  11. (Optional) Create Athena table on the Parquet location and run SELECT count(*) … (first 1 TB/month free-ish, but keep query tiny).

  12. Cleanup: delete Glue job, crawler, database/table, and the processed/ objects.

Expected Result

  • Parquet files are smaller than the original CSV.
  • S3 Select returns only the projected columns (proves reduced data transfer).
  • Athena query succeeds without ETL into a warehouse.

Exam Traps & Tips Related to This Exercise

  • Trap: Using Athena for “real-time” insights on a Firehose stream (Athena is batch).
  • Tip: “More compact storage + faster queries” or “Pipe mode into SageMaker” → convert to Parquet/ORC via Glue or DMS.
  • Trap: Claiming S3 Select works on Avro or ORC (it does not; only CSV, JSON, Parquet).
  • Tip: Columnar + compression + splittable = better for schema evolution and ML training I/O.
  • Cost control: Never leave Glue jobs with >2 DPUs or long timeouts; always set a 5–10 min timeout.

Exercise 3: Lightweight Ingestion Path into SageMaker Feature Store (Offline) + Data Wrangler Preview

Goal: Practice the “ingest into SageMaker Data Wrangler / Feature Store” skill with the absolute minimum billable resources. Also shows merging concept via simple join later.

Steps

  1. Create a small CSV or Parquet with 3–4 features + an ID column and event time (upload to S3).
  2. In SageMaker Canvas or Studio (if you already have a running domain; otherwise create a free-tier-eligible or smallest ml.t3.medium Studio domain only for this lab):
  3. Open Data Wrangler → Import from S3 → select your file.
  4. Apply 1–2 simple transforms (e.g., drop nulls, one-hot a column).

  5. Export → “Add to Feature Store” or use the Python SDK in a notebook:
    python from sagemaker.feature_store.feature_group import FeatureGroup fg = FeatureGroup(name="lab-fg-…", sagemaker_session=sess) fg.load_feature_definitions(data_frame=df) fg.create(s3_uri=f"s3://{bucket}/offline-store/", …, enable_online_store=False) # offline only! fg.ingest(data_frame=df, max_workers=1, wait=True)

  6. Verify offline store files appear under the S3 prefix (Iceberg or Glue table format).

  7. (Optional merge demo) Create a second tiny feature group and note that Canvas can join them.
  8. Immediate cleanup:
  9. Delete the feature group (fg.delete()).
  10. Shut down any Studio kernel / app.
  11. Delete the SageMaker domain if you created one solely for the lab (or at least delete the user profile).
  12. Empty the offline-store prefix.

Expected Result

  • Feature group status becomes Created.
  • Offline store S3 objects are queryable by Athena.
  • No online store charges (we disabled it).
  • Total Studio/Feature Store cost for a 15-minute session on ml.t3.medium is typically < $0.10.

Exam Traps & Tips Related to This Exercise

  • Trap: Thinking Feature Store ingestion is only real-time; batch via Data Wrangler, Spark connector on EMR, or ingest() API are all valid.
  • Tip: “Engineer features then ingest” → Data Wrangler → Feature Store.
  • Trap: Leaving online store enabled (hourly charges) or forgetting to delete the feature group.
  • Tip: For pure training pipelines the offline store (S3) is enough and far cheaper.
  • Cost control: Always set enable_online_store=False unless the question requires low-latency serving.

Bonus Micro-Exercise (5 min, almost free): S3 Transfer Acceleration & Extraction Options

  • Enable Transfer Acceleration on the bucket.
  • Time a small upload from a distant region/client vs. normal endpoint.
  • Practice aws s3api select-object-content on both CSV and Parquet (reinforces extraction skill).
  • Disable Acceleration afterward (no ongoing cost).

Quick Exam Trap Checklist for Task 1.1

  • S3 vs EFS/FSx: S3 wins for data lakes / ML pipelines / unstructured; EFS only for shared file-system access.
  • Real-time keyword → Kinesis Data Streams + Managed Flink / Firehose + Lambda; not Athena.
  • Firehose destinations: can land in S3 or Redshift (via intermediate S3 copy), cannot be a direct source for Redshift Streaming Ingestion.
  • Format conversion before SageMaker: Parquet/RecordIO preferred for Pipe mode (faster, less disk).
  • Operational overhead: prefer Glue (serverless) over EMR + Data Pipeline when the question asks to “reduce management”.
  • Capacity/scalability debugging: look at Kinesis shard count, EBS IOPS burst credits, S3 request rate prefixes, Glue DPU limits.
  • Cost: always pair storage-class + lifecycle; never pick One Zone when HA is mentioned.

Cleanup Ritual (Do This Every Lab)

# Example nuke script outline
aws s3 rb s3://ml-ingest-lab-… --force
aws glue delete-job --job-name …
aws glue delete-crawler …
aws sagemaker delete-feature-group --feature-group-name …
# Then delete Studio domain / user profile if created

By completing the three exercises you will have touched every major knowledge and skill bullet in Task 1.1 while keeping the bill under a dollar. Re-read the sample questions in the transcript after each exercise—you will now recognize the exact keywords that map to the services you just used. Good luck!