Task 1.1 - Collect and store data
Overview
Task 1.1, Collect and store data, is about moving data out of the systems that produce it and landing it somewhere a training job or an AI application can read it efficiently.
The storage choice is a design decision, not a default. It's driven by how the data is structured, how often it is read, what it costs to keep, and what compliance rules apply to it.
As you review these lessons for Task 1.1, check that you understand how to do the following.
- Extract data from data sources (for example, Amazon S3, Amazon EBS, Amazon EFS, Amazon RDS, Amazon DynamoDB, Amazon OpenSearch Service).
- Make storage decisions and configure storage services based on cost, performance, data structure, and data compliance.
- Troubleshoot and debug data ingestion and storage issues that involve capacity and scalability.
- Use AWS streaming data sources to ingest data (for example, by using Amazon Kinesis, Apache Flink, Apache Kafka).
- Ingest from and write by using appropriate data formats (for example, Apache Parquet, JSON, CSV, ORC) based on data access patterns.
- Merge data from multiple sources (for example, by using programming techniques, AWS Glue, Apache Spark).
- Configure scalable vector databases for AI applications (for example, OpenSearch Service, Amazon RDS with pgvector, Amazon S3) based on specifications.
- Ingest and store diverse data types (for example, text, images, audio) for AI and ML applications.
- Ingest data into SageMaker Feature Store.
Review AWS services
The services in this task fall into two jobs: reading data out of a source system, and holding data in a store that suits how it will be read. Recognize which job each service does and what shape of data it expects. Review each service one to learn more.
AWS services overview
The services in this task fall into two jobs: reading data out of a source system, and holding data in a store that suits how it will be read. Recognize which job each service does and what shape of data it expects. Review each service one to learn more.
| AWS Service | Description |
|---|---|
| Amazon Simple Storage Service (Amazon S3) | Object storage for datasets, documents, images, and audio. The usual landing zone for ML data and the usual source for training jobs. |
| Amazon OpenSearch Service | Search and analytics, including vector search over embeddings. |
| Amazon Elastic Block Store (Amazon EBS) | Block storage presenting a raw block device to an instance, typically one instance at a time. Suits data read as a local disk. |
| Amazon Elastic File System (Amazon EFS) | A shared file system many instances can mount at once. Suits distributed jobs that need one file namespace. |
| Amazon FSx | Fully managed third-party file systems, including a high-performance option for training that needs very fast access to a shared dataset. |
| Amazon Relational Database Service (Amazon RDS) | Managed relational databases for structured, queryable records. |
| Amazon DynamoDB | A managed key-value and document database for fast lookups by key. |
| AWS Glue | Serverless data integration. Crawlers record table definitions in the AWS Glue Data Catalog, and jobs extract, transform, and load, most commonly on Apache Spark. |
| Amazon EMR | Managed clusters for Apache Spark and other big data frameworks, for transformations that need cluster scale. |
| Amazon SageMaker Feature Store | A purpose-built store for the features a model consumes, so one feature definition serves training and inference. |
Streaming and purpose-built stores overview
Two groups of services in this task carry more depth for a machine learning engineer, because the choice inside each group changes the design. Review each service to learn more.
| AWS Service | Description |
|---|---|
| Amazon OpenSearch Service as a vector store | Stores embeddings and finds nearest neighbors by distance or by cosine similarity, which is what retrieval in a generative AI application needs. |
| Amazon Kinesis Data Streams | Captures a continuous stream of records so one or more consumers can read them, in order within each shard. |
| Amazon Managed Service for Apache Flink | Runs Apache Flink applications that process records as they arrive, with connectors for Kinesis Data Streams and for Kafka topics. |
| Amazon Managed Streaming for Apache Kafka (Amazon MSK) | A managed Apache Kafka service, for teams already standardized on Kafka topics and consumer groups. |
| Amazon RDS for PostgreSQL with the pgvector extension | Adds vector similarity search to a relational database, so embeddings sit beside the relational records they describe. |
| Amazon S3 vector buckets and vector indexes | Hold embeddings in object storage, which suits large vector sets that tolerate less frequent querying. |
| Amazon SageMaker Feature Store | Keeps an online store for low-latency lookup of the most recent record and an offline store for the history training reads. |
Knowledge Check
An application needs to find the documents closest in meaning to a user's question, and the embeddings already sit alongside relational records in a PostgreSQL database. Which option adds similarity search without moving the data? (Choose 1)
- A. Amazon RDS for PostgreSQL with the pgvector extension
- B. Amazon DynamoDB
- C. Amazon Elastic File System (Amazon EFS)
- D. Amazon Elastic Block Store (Amazon EBS)
Answer & Explanation
Correct Answer: A. Amazon RDS for PostgreSQL with the pgvector extension
Explanation:
The pgvector extension enables vector storage and similarity search (e.g., L2 distance, inner product, cosine distance) directly inside PostgreSQL. Since the relational records already reside in an RDS PostgreSQL database, adding pgvector allows performing vector searches on embeddings sitting right alongside their relational data without needing to migrate or sync data to a separate vector database.
Why the other options are incorrect: * B. Amazon DynamoDB: A NoSQL key-value/document database. It cannot natively run PostgreSQL extensions or query relational records in place. * C. Amazon EFS: A shared POSIX file system designed for file-level access across multiple instances, not a queryable database engine. * D. Amazon EBS: A raw block storage service attached to individual EC2 instances, serving as a disk volume rather than a database layer with vector search capabilities.
Review AWS skills
Extracting and merging source data
In practice, this group of skills lets the machine learning engineer assemble one training dataset out of systems that were never designed to be read together.
Be comfortable doing the following:
- Pulling records out of object storage, block and file storage, relational databases, key-value stores, and search indexes, and recognizing which access pattern each one is built for.
- Joining data that arrives from several sources into a single dataset, whether with code, a managed integration service, or a distributed processing framework.
- Cataloging what you extracted, so a later job can find the table definition instead of rediscovering the schema.
Extraction is usually the simpler half. The harder half is reconciling sources that disagree: the same customer keyed differently in two systems, timestamps recorded in different zones, a field that is optional in one source and required in another. A crawler can record the schema it finds, but deciding which source wins on a conflict is an engineering judgment you make deliberately and document.
Knowledge Check
Two source systems describe the same entities with different keys and overlapping records. What must be settled before the merged dataset is usable for training? (Choose 1)
- A. Which storage class the merged output uses
- B. How many compute instances the merge job runs on
- C. Whether the output is written before or after the catalog is updated
- D. Which source is authoritative when the two disagree on a field
Answer & Explanation
Correct Answer: D. Which source is authoritative when the two disagree on a field
Explanation:
When merging overlapping datasets from different source systems (data deduplication / entity resolution), field-level conflicts inevitably arise. Establishing conflict resolution rules—specifically defining which source system serves as the single source of truth (authoritative source) when attributes disagree—is mandatory to ensure data quality, consistency, and label integrity prior to model training.
Why the other options are incorrect: * A. Which storage class the merged output uses: Storage tiering (e.g., S3 Standard vs. S3 Glacier) impacts storage cost and access latency, but does not affect the semantic correctness or usability of the training data. * B. How many compute instances the merge job runs on: Instance scaling dictates processing speed and resource cost for the ETL pipeline, not the logical validity of the data. * C. Whether the output is written before or after the catalog is updated: Glue/Hive metastore catalog update sequencing affects metadata discovery timing, but does not resolve data contradictions between overlapping source records.
Choosing storage, formats, and data types
In practice, this group of skills lets the machine learning engineer pick storage and a file format that match how the data will actually be read, rather than defaulting to whatever produced it.
Be comfortable doing the following:
- Weighing cost, performance, data structure, and compliance together when you select and configure a storage service, and recognizing that these four pull against each other.
- Choosing a file format from the access pattern: row-oriented formats such as JSON and CSV suit record-at-a-time writes and simple interchange, and columnar formats such as Apache Parquet and ORC suit analytic reads that touch a few columns across many rows.
- Storing text, images, and audio together with the metadata that makes them findable and traceable back to their source.
The format decision has a direct cost consequence. A columnar format lets a query engine read only the columns it needs, so the same analytic query scans less data than it would over row-oriented files. That is why a curated layer in a data lake is usually written in a columnar format even when the raw layer is not. Compliance cuts the other way and can constrain both format and location: where data may live, how long it is kept, and who may read it are requirements you satisfy in the storage configuration, not in the training code.
Knowledge Check
A curated dataset is queried repeatedly, and each query reads a small number of columns across many rows. Which format choice BEST fits that access pattern? (Choose 1)
- A. Uncompressed CSV
- B. One JSON document per record
- C. A single archive file containing the whole dataset
- D. A columnar format such as Apache Parquet or ORC
Answer & Explanation
Correct Answer: D. A columnar format such as Apache Parquet or ORC
Explanation:
Columnar storage formats like Apache Parquet or ORC organize data by column rather than by row. When a query only requests a small subset of columns across many rows, columnar formats allow query engines (such as Amazon Athena, AWS Glue, or Apache Spark) to perform column pruning (projection pushdown). The engine reads only the storage blocks containing the required columns from S3, dramatically reducing I/O operations, execution time, and query costs.
Why the other options are incorrect:
* A. Uncompressed CSV: CSV is a row-oriented, uncompressed text format. Query engines must scan every single column in every row, leading to massive I/O overhead and slow performance.
* B. One JSON document per record: JSON is a row-/document-oriented text format. Every document must be fetched and parsed in its entirety to retrieve even a single field.
* C. A single archive file containing the whole dataset: Storing data as a compressed archive (e.g., .zip or .tar.gz) lacks columnar indexing. The entire file must be downloaded and decompressed before reading any data.
Streaming ingestion, capacity, and scaling
In practice, this group of skills lets the machine learning engineer ingest data that never stops arriving, and diagnose the ingestion path when it falls behind.
Be comfortable doing the following:
- Ingesting from AWS streaming sources, and recognizing the division of labor between the service that holds the stream and the application that processes records as they arrive.
- Reading from Kafka topics as well as from AWS-native streams, since many organizations already run Kafka.
- Recognizing capacity and scalability symptoms in an ingestion path, and separating a throughput limit from a downstream slowdown.
Streaming failures are rarely mysterious once you know where to look. A consumer that falls further behind over time is either under-provisioned for the incoming rate or blocked on something downstream, and the two look similar from a dashboard but need opposite fixes. Adding consumer capacity to a pipeline that is actually blocked on a slow write target just moves the queue. The same reasoning applies to batch ingestion: a job that slows as the dataset grows is usually reading more data than it needs, which points back at the layout and format decision rather than at compute.
Knowledge Check
A stream consumer falls further behind the incoming record rate each hour, and the write target it loads into is already at its throughput limit. What does adding consumer capacity accomplish? (Choose 1)
- A. Very little, because the constraint is the downstream write target
- B. It clears the backlog, because consumer capacity is the binding constraint
- C. It changes the order in which records are read
- D. It reduces the size of each record
Answer & Explanation
Correct Answer: A. Very little, because the constraint is the downstream write target
Explanation:
In streaming architectures (e.g., Amazon Kinesis Data Streams or Apache Kafka), total pipeline throughput is constrained by the system's narrowest bottleneck. Because the downstream write target (e.g., a database, data warehouse, or index) is already at its throughput limit, adding more consumer instances or threads will simply cause the extra consumers to be throttled or blocked by the target. To resolve the backlog, you must scale up or optimize the downstream write target first.
Why the other options are incorrect: * B. It clears the backlog, because consumer capacity is the binding constraint: Consumer capacity is not the binding constraint in this scenario; the write target is the limiting factor. * C. It changes the order in which records are read: Scaling consumer capacity does not inherently alter the strictly ordered read sequence within individual stream shards or partitions. * D. It reduces the size of each record: Scaling consumer workers has no impact on the payload size or schema of incoming stream records.
Vector stores and feature stores
In practice, this group of skills lets the machine learning engineer stand up the two purpose-built stores that AI and ML workloads depend on, and keep straight which one answers which question.
Be comfortable doing the following:
- Configuring a scalable vector store for an AI application, and selecting among a search service, a relational database with a vector extension, and object storage based on the specification you were given.
- Ingesting features into a feature store, and recognizing that the store keeps a low-latency copy for lookup and a historical copy for training.
These two stores are easy to confuse because both sit between raw data and a model. A vector store answers a similarity question: given this embedding, what else is close to it. A feature store answers an identity question: given this entity, what are its current feature values, and what were they at the moment a historical event occurred. The second half of that matters. Serving inference from a feature definition that differs from the one training used is a common source of models that score well in testing and disappoint in production, which is exactly the problem a shared feature definition removes.
Knowledge Check
What does a feature store provide that a general-purpose database does not provide on its own? (Choose 1)
- A. Similarity search over embeddings
- B. Automatic selection of a model algorithm
- C. Encryption of data at rest
- D. One feature definition that serves both training and inference
Answer & Explanation
Correct Answer: D. One feature definition that serves both training and inference
Explanation:
A key purpose of a feature store (like Amazon SageMaker Feature Store) is to provide a single, centralized repository for feature engineering logic. It manages an offline store (optimized for batch reads and historical time-travel queries for model training) and an online store (optimized for high-throughput, low-latency lookups during inference). This single-definition design guarantees consistency across training and production, preventing training-serving skew.
Why the other options are incorrect:
* A. Similarity search over embeddings: Vector search is a capability of dedicated vector databases or search engines (e.g., Amazon OpenSearch Service, pgvector), not a standard feature store function.
* B. Automatic selection of a model algorithm: Algorithm selection and hyperparameter optimization are handled by AutoML services (such as Amazon SageMaker Autopilot), not data storage layers.
* C. Encryption of data at rest: General-purpose relational and NoSQL databases (e.g., Amazon RDS, DynamoDB) natively support data-at-rest encryption via AWS KMS.
Test your Knowledge
Knowledge Check
A distributed training job runs across several instances, and every instance needs to read from the same file namespace at the same time. Which storage service fits that requirement? (Choose 1)
- A. Amazon Elastic Block Store (Amazon EBS)
- B. Amazon DynamoDB
- C. Amazon Elastic File System (Amazon EFS)
- D. Amazon Relational Database Service (Amazon RDS)
Answer & Explanation
Correct Answer: C. Amazon Elastic File System (Amazon EFS)
Explanation:
Amazon EFS provides a serverless, fully managed POSIX-compliant shared file system that can be mounted concurrently by hundreds or thousands of compute instances (such as EC2 or SageMaker training instances). This provides all nodes in a distributed training job concurrent access to a single, shared file namespace without requiring data duplication.
Why the other options are incorrect: * A. Amazon Elastic Block Store (Amazon EBS): EBS provides block-level storage mounted as a local disk to a single instance at a time, making it unsuitable for a multi-node shared file namespace. * B. Amazon DynamoDB: A NoSQL key-value and document database designed for low-latency row/item lookups, not a shared file system interface. * D. Amazon Relational Database Service (Amazon RDS): A managed relational SQL database for structured tabular data, not a file storage layer.