Task 1.1: Data Preparation for ML and AI - Collect and store data.

Skill 1.1.5: Ingest from and write by using appropriate data formats (for example, Apache Parquet, JSON, CSV, ORC) based on data access patterns.

1. Skill Overview

Selecting the right data format is a fundamental design decision in an ML pipeline. It affects storage cost, query performance, ETL complexity, and even the accuracy of the data that reaches your model.

The choice should always be driven by how the data will be read — not by how it was produced. The format you use for raw ingestion is often different from the format you use for curated, analytics-optimized datasets that feed model training.

Key formats tested on the MLA-C02 exam:

Format	Category	Best Used For
CSV	Row-oriented, text	Simple interchange, human readability, small flat datasets
JSON	Row/document-oriented, text	Nested and semi-structured data, APIs, streaming events
Apache Parquet	Columnar, binary	Analytical queries, ML training datasets, large-scale data lakes
ORC	Columnar, binary	Hive-based analytics, large-scale queries, often used in EMR/Hadoop

The core tension is row-oriented vs columnar. Understanding this difference is the foundation of this skill.

2. Row-Oriented vs Columnar Formats

Row-Oriented Formats (CSV, JSON)

Data is stored record by record, with all fields of a single record stored together.
Optimized for write-heavy or transactional workloads where you frequently read entire rows or append single records.
Inefficient for analytical queries that only need a few columns across millions of rows, because the query engine must scan all columns even if only one is requested.

Columnar Formats (Parquet, ORC)

Data is stored column by column, with values for a single field stored contiguously across many records.
Optimized for read-heavy analytical workloads that scan many rows but only a subset of columns.
Enable column pruning and predicate pushdown, drastically reducing the amount of data scanned.

Visual Comparison

Access Pattern	Row-Oriented (CSV/JSON)	Columnar (Parquet/ORC)
Read entire row by ID	✅ Excellent	❌ Poor
Read few columns across many rows	❌ Poor	✅ Excellent
Append new records frequently	✅ Good	⚠️ More complex (batch recomposition)
Nested data support	JSON: ✅ CSV: ❌	✅ Both support nesting
Storage efficiency with repeated values	❌ Poor	✅ High compression
Schema enforcement	Schema-on-read	Schema embedded in file

3. Decision Framework for Choosing a Data Format

The following mermaid diagram illustrates a simplified decision process:

graph TD
    A[What is the primary data access pattern?]
    A -->|Frequent writes, record-at-a-time appends, streaming ingestion| B{Raw data shape?}
    A -->|Frequent analytical reads, few columns over many rows, ML training| C{Do you need nested data?}

    B -->|Simple flat tables| D[CSV]
    B -->|Nested or semi-structured| E[JSON]

    C -->|No - flat tabular| F[Parquet or ORC]
    C -->|Yes - nested structures| G[Parquet or ORC (both support nesting)]

    F --> H[Choose based on ecosystem: Parquet for AWS/Spark, ORC for Hive-centric]
    G --> H

Important nuance: The choice is not binary. A common pattern is to ingest raw data as JSON/CSV in a raw S3 bucket for durability and schema flexibility, then transform it to Parquet/ORC in a curated S3 bucket for efficient downstream analytics and ML training.

4. Deep Dive on Individual Formats

4.1 CSV (Comma-Separated Values)

Type: Row-oriented text format
Schema: None enforced; schema inferred by reader (schema-on-read)
Strengths:
Universally readable, human-friendly
Simple to produce and consume
Fine for small datasets or one-time exports
Weaknesses:
No native support for nested data
Full scans required even for single column queries
Data type ambiguity (e.g., dates, leading zeros, commas in fields)
Poor compression and no internal predicate pushdown
When to use:
Data interchange between simple systems
Small, flat tabular datasets
Prototyping and ad hoc analysis

4.2 JSON (JavaScript Object Notation)

Type: Row/document-oriented text format
Schema: No fixed schema; supports nested and semi-structured data
Strengths:
Supports nested objects, arrays, and variable structures
Self-describing and widely used by APIs and streaming services
Great for capturing raw event data with evolving schemas
Weaknesses:
Repetition of keys increases file size
Parsing is CPU-intensive compared to binary formats
Query engines must read and parse entire documents to extract fields, unless using specialized optimizations
When to use:
Raw ingestion from webhooks, IoT devices, clickstreams, application logs
Storing events in a raw data lake before transformation

4.3 Apache Parquet

Type: Columnar binary format
Schema: Embedded in file metadata; supports complex nested structures
Strengths:
Excellent compression due to column-wise encoding (e.g., run-length, dictionary)
Enables column pruning and predicate pushdown — massive reduction in data scanned
Splittable even when compressed, enabling distributed processing (e.g., Spark, EMR)
Deeply integrated with AWS services: Athena, Glue, EMR, SageMaker, Redshift Spectrum, Kinesis Data Firehose
Weaknesses:
More complex to write; less suited for high-frequency, small-row appends
Not human-readable
When to use:
Curated data lake layers for analytics and ML
Training datasets read repeatedly by distributed training jobs
Querying with Athena or Redshift Spectrum

4.4 ORC (Optimized Row Columnar)

Type: Columnar binary format
Schema: Embedded in file footer/stripe
Strengths:
Similar columnar benefits to Parquet (compression, pruning, predicate pushdown)
Tight integration with Apache Hive; ACID transactions when used with Hive tables
Weaknesses:
Less broadly adopted across AWS-native services compared to Parquet
When to use:
Hive-centric data warehouses on Amazon EMR
Environments already standardized on ORC tables

Parquet vs ORC: For the exam, treat them as functionally similar for most analytical use cases. If the question is AWS-specific and you need a columnar format, Parquet is generally the safer default answer.

5. Format Choice Based on Access Patterns

Access Pattern	Recommended Format	Reason
High-volume streaming events, raw capture	JSON (often line-delimited)	Flexible schema, easy to ingest from Kinesis/Kafka
Frequent analytical queries scanning few columns over many rows	Parquet or ORC	Column pruning and predicate pushdown reduces I/O and cost
Distributed ML training reading entire dataset many times	Parquet/ORC with partitioning	Efficient compression, splittable, reduces data scanned
Simple flat data, small volume, human inspection needed	CSV	Universally accessible, simple
API responses, nested hierarchical data, semi-structured	JSON	Native support for nesting and schema flexibility
Batch ETL output that will be queried by Athena / Spectrum	Parquet	Optimized for Athena, lower cost per query
Hive table with frequent updates/ACID requirements	ORC	Supports ACID when used with Hive

6. Real-World Scenario-Based Examples

Scenario 1: Analytics and ML Training on a Data Lake

Context: A company collects millions of e-commerce transactions daily. Data analysts query the data for business reports, and ML engineers use the same dataset to train a fraud detection model.

Requirements: Queries typically filter by date and customer segment, reading only 5–10 of 100+ columns. Training jobs run weekly and read the entire dataset.

Solution:

Ingest raw JSON events from the website into an S3 raw bucket.
Use AWS Glue ETL to transform and write the data as Apache Parquet in a curated S3 bucket, partitioned by year/month/day.
Query with Amazon Athena for analytics and use SageMaker Training jobs reading Parquet directly.

Why Parquet? Column pruning reduces Athena scan costs, and partitioning eliminates irrelevant partitions. Compression reduces storage costs.

Scenario 2: Streaming Clickstream Data

Context: A mobile app emits clickstream events at high velocity. The team wants near real-time monitoring and historical analysis.

Requirements: Events are deeply nested with varying fields. Real-time dashboards need to ingest continuously; historical analysis runs occasionally.

Solution:

Use Amazon Kinesis Data Firehose to deliver raw events to S3 as JSON (line-delimited) in a raw bucket for immediate availability.
Enable Firehose’s data format conversion feature to automatically convert the JSON to Parquet for a separate S3 prefix used by Athena and ML workloads.

Why JSON for raw? Captures the exact event payload without schema loss.
Why Parquet for analysis? Enables efficient SQL queries and column pruning without scanning entire JSON documents.

Scenario 3: Frequently Updated Operational Data

Context: A small internal application maintains a list of 1,000 products. The data is updated daily by an admin, and the ML team occasionally exports it for a model.

Requirements: Low frequency of reads, high simplicity, no distributed processing.

Solution: Store the data as CSV in an S3 bucket. Export to CSV for model input.

Why CSV? The dataset is small and flat; overhead of columnar formats not justified. Human readability is valuable for debugging.

Scenario 4: Nested IoT Sensor Data

Context: IoT devices send sensor readings including nested metadata (device info, location, sensor array). The data is stored for both real-time alerting and batch ML model training.

Requirements: Need to preserve nested structure. Training reads all records and many fields.

Solution:

Ingest as JSON into S3 raw.
Convert to Parquet for the curated dataset. Parquet supports nested data with efficient column access.
Train model using Spark on EMR reading Parquet.

Why not keep as JSON? While JSON preserves nesting, scanning 10 TB of JSON for training would be slow and expensive. Parquet allows nested structure and efficient pruning.

7. AWS Services and Format Considerations

AWS Service	How It Relates to Data Formats
Amazon S3	Storage layer for any format. No inherent format preference, but compression and partitioning affect performance.
AWS Glue	ETL jobs can read/write CSV, JSON, Parquet, ORC. DynamicFrame supports format conversion.
Amazon Athena	Serverless SQL. Query cost is based on data scanned, so Parquet/ORC are significantly cheaper than CSV/JSON.
Amazon EMR	Spark/Hive clusters. Parquet and ORC are splittable, enabling high parallelism. CSV/JSON may require full scans.
Amazon SageMaker	Training jobs can read Parquet/CSV/JSON from S3. Built-in algorithms may require CSV or recordIO, but general ML frameworks accept all.
Kinesis Data Firehose	Can convert incoming JSON to Parquet/ORC before delivering to S3, using a Glue table schema.
Redshift Spectrum	Queries external S3 data. Columnar formats dramatically improve performance and reduce cost.
AWS Glue Data Catalog	Stores table definitions, including format, compression, and SerDe for CSV/JSON/Parquet/ORC.

8. Common Exam Traps and Tips

✅ Exam Tips

Columnar = Parquet/ORC. If a question mentions “read few columns across many rows” or “analytical queries on large datasets,” choose Parquet or ORC.
Row-oriented = CSV/JSON. If the question emphasizes “simple interchange,” “human-readable,” “streaming raw events,” or “record-at-a-time writes,” choose CSV or JSON.
Kinesis Data Firehose can convert JSON to Parquet/ORC. This is a repeatable exam point. It reduces storage and improves Athena performance.
Parquet is more AWS-native. When a question asks for an efficient format for S3 data queried by Athena or Spectrum, Parquet is the expected answer unless ORC is explicitly tied to Hive.
Partitioning is a separate but related optimization. A columnar format plus partition pruning yields the best performance.
Splittability matters. Parquet/ORC are splittable even when compressed; gzip-compressed CSV/JSON is not splittable, which can cause Spark/EMR jobs to lose parallelism.

⚠️ Exam Traps

Assuming JSON is best for nested data. JSON supports nesting, but Parquet also supports nested structures and provides far better analytical performance. Do not choose JSON solely because the data is nested if the question emphasizes query performance or training efficiency.
Choosing CSV because it is “simple.” Simplicity may be fine for small datasets, but it becomes a trap for large analytical workloads. The exam often tests whether you default to CSV when Parquet is clearly more appropriate.
Confusing compression with format. A gzip-compressed CSV is still row-oriented and does not gain column pruning. Do not assume compression solves the columnar problem.
Thinking ORC and Parquet are fundamentally different. For exam purposes, both are columnar; the primary distinction is ecosystem. If the scenario is AWS-native, Parquet is usually the safer choice.
Overlooking schema evolution. JSON/CSV are schema-on-read, meaning they tolerate changes without file rewriting. Parquet/ORC embed schema but support some evolution; however, large-scale schema changes may require rewriting. If a question highlights frequent schema changes and immediate ingestion, raw JSON may be acceptable initially.
Ignoring write frequency. Columnar formats are not ideal for high-frequency small appends; row-oriented formats handle that better. If the pattern is continuous streaming writes with occasional reads, JSON/CSV may be the right raw format, with conversion later.

9. Summary Table

Format	Orientation	Nested Support	Compression	Typical Use Case in ML Pipeline
CSV	Row	No	Poor (external only)	Small flat datasets, human-readable interchange
JSON	Row	Yes	Moderate (text)	Raw ingestion, API events, streaming logs
Parquet	Columnar	Yes	Excellent (built-in)	Curated data lake, ML training, Athena queries
ORC	Columnar	Yes	Excellent	Hive-based analytics, EMR workloads

Rule of thumb:
Raw layer → JSON or CSV (flexible, fast ingest).
Curated/analytics/training layer → Parquet (or ORC) (efficient reads).

10. Final Review Scenario

Exam-style question:

A company stores IoT sensor data in S3. The data is ingested continuously via Kinesis Data Firehose as JSON. The data science team wants to run daily training jobs that scan only a few numeric columns across millions of records. Storage costs are a concern. Which action best optimizes the training workload?

A. Store the data as gzip-compressed CSV in S3
B. Enable Firehose data format conversion to Apache Parquet and store in a partitioned S3 prefix
C. Store the data as line-delimited JSON in a single S3 bucket
D. Store the data in Amazon EBS for low-latency access

Answer: B
Explanation: Parquet is columnar, enabling column pruning and predicate pushdown, so training jobs read only the required numeric columns. Firehose conversion automates this without custom ETL. Partitioning further reduces scanned data. CSV/JSON would require full scans of all data, and EBS is not suitable for large-scale distributed training.

This study note should equip you to confidently answer questions on data format selection based on access patterns for the AWS Certified Machine Learning – Associate exam.