Skip to content

1.1

Task 1.1: Ingest and store data

Knowledge of:

  • Data formats and ingestion mechanisms (for example, validated and non-validated formats, Apache Parquet, JSON, CSV, Apache ORC, Apache Avro, RecordIO)
  • How to use the core AWS data sources (for example, Amazon S3, Amazon EFS, Amazon FSx for NetApp ONTAP)
  • How to use AWS streaming data sources to ingest data (for example, Amazon Kinesis, Apache Flink, Apache Kafka)
  • AWS storage options, including use cases and tradeoffs

Skills in:

  • Extracting data from storage (for example, Amazon S3, Amazon EBS, Amazon EFS, Amazon RDS, Amazon DynamoDB) by using relevant AWS service options (for example, Amazon S3 Transfer Acceleration, Amazon EBS Provisioned IOPS)
  • Choosing appropriate data formats (for example, Parquet, JSON, CSV, ORC) based on data access patterns
  • Ingesting data into Amazon SageMaker Data Wrangler and SageMaker Feature Store
  • Merging data from multiple sources (for example, by using programming techniques, AWS Glue, Apache Spark)
  • Troubleshooting and debugging data ingestion and storage issues that involve capacity and scalability
  • Making initial storage decisions based on cost, performance, and data structure

Let's get started with the first task statement from Domain 1, which is to ingest and store data. This task statement is split into three lessons. Let's start with an overview and talk about the first two steps of the machine learning engineering lifecycle, collecting data and processing data. So after you collect and store your data, you can then begin to process that data. This would be the third step of the well-architected machine learning lifecycle, which is data processing. Let's pause here and go back a bit further for the needed fundamentals and overview for this exam, and talk about the steps of the data engineering lifecycle, which are generation, storage, ingestion, transformation, and serving data. So let's quickly cover the first two steps of the data engineering lifecycle before we talk about collecting and processing data in AWS, which is the focus of this task statement. The first step is generation, which is where the data originates from. So it's your source system such as a database, IoT device, and more. And for the generation stage, the data engineer or the machine learning engineer consumes the data from a source system, but might not control that source system. The second step is storage, which is a focus for this task statement, and the first two steps of the machine learning engineer lifecycle. For this exam, ensure you can choose an appropriate data store to store the data. Also ensure you understand different use cases and different AWS storage services to ingest your data from the source system into AWS, and ensure you understand how to extract that data and ingest and process that data into your machine learning algorithm to train your model. A few questions to ask yourself are, which AWS storage service is best for your use case? What format is the data in? What are the access patterns? Is the data streaming? And does the data need to be merged from multiple sources? Here is a question. "You need to store unprocessed data from IoT devices for a new machine learning pipeline. The storage solution should be a centralized and highly available repository. What AWS storage service do you choose to store the unprocessed data?" For this exam, you need to understand the use cases and trade-offs for the AWS storage services. And for cost optimization, understand the different storage classes of those services and the lifecycle policies. Let's also point out a few keywords in this question, which are "store unprocessed data," "pipeline," and "centralized," and "highly available repository." Do you store the unprocessed data in an Amazon Elastic File System, Amazon EFS, or do you store the data in Amazon S3 and create a data lake for the unprocessed data? You could use Amazon EFS to store the data, but Amazon S3 is a better choice because it has direct integration with other AWS services, such as AWS Glue, Amazon EMR, Amazon SageMaker and more, that you can use to build a machine learning pipeline. And remember that "pipeline" was a keyword. Amazon S3 has a use case for unstructured data, and again, integrates with other AWS services to build your pipeline. Also, another key phrase was unprocessed data. Amazon EFS supports file storage and a use case is scalable data storage. Here's a question. "You are designing a highly scalable data repository for your machine learning pipeline. You need immediate access to the processed data from your pipeline for 6 months. Your unprocessed data must be accessible within 12 hours and stored for 6 years. What is your cost-effective storage solution?" One solution is to set an S3 lifecycle with lifecycle configuration rules that defines actions to store your objects in S3. For this solution, you can store your processed data in an Amazon S3 Standard storage class, create a lifecycle policy to move your processed data into Amazon S3 Glacier after 6 months. And for the unprocessed data, you can create another lifecycle policy to move that data into Amazon S3 Glacier Deep Archive. Let's pause and look at two key phrases for this scenario question. "Immediate access" and "highly available." So for this question, Amazon S3 One-Zone storage class would not be a best-choice answer, because with both of the One-Zone storage class options, your data is only stored in one availability zone. I added a link under additional resources for the S3 storage classes. I also added a link for other AWS storage options that also have lifecycle policies. What if we added another requirement for the storage solution to support SQL querying capabilities? Well, with Amazon S3, you can use Amazon Athena for ad hoc SQL queries. You cannot run SQL queries in Amazon DynamoDB unless you use PartiQL. And for Amazon Redshift, it does support SQL queries, but would not be the best-choice answer. I'm gonna stop this lesson here, and in the next lesson, we will continue talking about task statement 1.1.

Let's continue with task statement 1.1, which is to ingest and store data, and pick up where we left off and continue talking about ingesting data. There are many different types of datasets to transform different pieces of data before using that data to train your model. By using the right data structures and algorithms, machine learning engineers can optimize performance, improve scalability, and reduce processing time. Ensure you know which AWS services can work with and store structured, semi-structured, and unstructured data. And ensure you know how to model structured, semi-structured, and unstructured data in AWS to choose the appropriate storage and processing options to get valuable insights from different types of data. You can work with unstructured, semi-structured, and structured data formats with your data stored in Amazon S3 by using Amazon Athena, and query data without the need for data preparation or extract, transform, and load processes. And by using columnar data formats, can help with your schema evolution. You might also consider compression when choosing data formats or using splittable formats. For data ingestion, you might use Lambda to write code that handles data ingestion tasks, such as parsing, transforming, and loading the data into a data warehouse or database. For data processing, you might use Amazon EMR and AWS Glue to write code using programming languages to define data processing jobs. You can use frameworks such as Apache Spark to distribute and parallelize data processing tasks across a cluster of EC2 instances. For real-time data processing, you can use Kinesis and write code to consume and process streaming data. And you can use the AWS SDKs and Kinesis Client Libraries to interact with the data streams. To perform complex data transformations, you can use AWS Glue and write code using the AWS Glue Python-based ETL framework to define transformation logic, such as data cleansing, aggregation, normalization, and schema evolution. For workflow orchestration, you can use AWS Step Functions to build serverless workflows by defining state machines. Then, you can use programming concepts to define the workflow's structure, conditions, and actions. For the exam, ensure you also understand how to use AWS streaming data sources to ingest data, such as Amazon Kinesis, Apache Flink, and Apache Kafka. Here is a question. You are using Amazon Data Firehose delivery stream to ingest GZIP compressed data records from an on-premises application. You need to configure a solution for your data scientist to perform SQL queries against the data stream for real-time insights. What is your solution? One solution is to use the AWS Managed Service for Apache Flink application and a Lambda function to transform the data before it is processed by the SQL code, and then send the data to the application for real-time analysis. Another solution might be to store the data in an S3 bucket and use Amazon Athena to run queries. This would work, but remember, a keyword for this scenario question was real-time. And Athena does not provide real-time insights. Also, Athena cannot consume data directly from the Firehose delivery stream in real-time. Here is another streaming scenario. Can you migrate data to Amazon S3 using AWS Database Migration Service from an on-premises or other supported database sources? Yes, one solution is to use Amazon S3 as a target in an AWS DMS task, and both full load and change data capture data is written to comma-separated value format by default. For more compact storage and faster queries, you can use Apache Parquet as the storage format. And after that data is migrated from AWS DMS to Amazon S3, you can use Amazon SageMaker, which has a faster Pipe mode implementation that accelerates the data transfer speeds for data that is streamed from Amazon S3 into SageMaker. This helps your training jobs start sooner, finish quicker, require less disk space, and reduces your cost to train machine learning models on SageMaker. Let's say that you are a machine learning engineer and you need to process a large amount of customer data, analyze the data, and get insights so that analysts can make further decisions. To accomplish this task, you need to store the data in a data structure that can handle large volumes of data and efficiently retrieve it as fast as possible. What is your solution? One solution is to use Amazon EMR with Hadoop Distributed File System to store the data and then you can process the data to extract insights. You choose the appropriate algorithm, such as the MapReduce algorithm, to process the data in your data processing pipelines. Finally, you optimize the performance of your data processing pipeline by using techniques such as data partitioning, caching, and parallel processing. For this task statement, you also need to understand how to ingest data into Amazon SageMaker Canvas and SageMaker Feature Store. In SageMaker, records are added to your feature groups through ingestion, depending on the use case, storage configuration, and more. Or you can use the Data Wrangler experience in SageMaker Canvas to engineer features and then ingest your features into your Feature Store. After the feature group has been created, you can also select and join data across multiple feature groups to create new engineered features in SageMaker Canvas and then export your dataset to an S3 bucket. Let's pause and check your depth and ask a question. Is there another way to bring your data into Feature Store? Yes, you can also use Amazon EMR for batch data ingestion through an Apache Spark connector. I'm gonna stop this lesson here, and in the next lesson, we'll continue talking about task statement 1.1.

Let's continue with task statement 1.1 and talk about extracting data from storage by using AWS service options. Let's go back to our earlier question of building a scalable and cost-efficient repository to store thousands of existing training datasets and new sets to train models. The storage solution was Amazon S3. What are options to extract data from an S3 bucket? You can use Amazon S3 Select and SQL statements to filter the contents of an Amazon S3 object and retrieve only the subset of data that you need. This also reduces the cost and latency to retrieve the needed data. As far as data formats, Amazon S3 Select works on objects stored in CSV, JSON, or Apache Parquet format. It also works with objects that are compressed with GZIP or BZIP2 and server-side encrypted objects. For Amazon S3, you can also use Amazon S3 Transfer Acceleration to extract data from your S3 bucket. If your data is stored in Amazon EBS volumes, ensure you understand the relationship between the maximum performance of your EBS volumes, the needed size and number of input and output operations, and the time it takes for each action to complete. On instances without support for EBS-optimized throughput, network traffic might compete with traffic between your instance and your EBS volumes. But on Amazon EBS optimized instances, the two types of traffic are kept separate. Let's move on and talk about troubleshooting and debugging data ingestion and storage issues that involve capacity and scalability. Here's a question. You were asked to redesign and reduce the operational management and the cost of the transformation pipeline that ingests training data from multiple custom sources. Currently, the pipeline uses Amazon EMR and AWS Data Pipeline. What is your solution? One solution would be to create an ETL job using AWS Glue to transform the data and deliver the results into an S3 bucket. Here's another question. You have been asked to redesign and reduce the operational overhead and use AWS services to detect anomalies in transaction data and assign anomaly scores to malicious records. The records are streamed in real-time and stored in an Amazon S3 data lake for processing and analysis. What is your solution? One solution is to use Firehose to stream transaction data and Amazon Managed Service for Apache Flink RANDOM_CUT_FOREST function to detect anomalies. Could you also use Firehose to stream transaction data into Amazon S3 with the SageMaker Random Cut Forest to detect anomalies? This is a valid solution, but it won't meet the requirements of reducing the operational overhead, because SageMaker requires more operational management than Amazon Managed Service for Apache Flink. Here's another scenario. You've been asked to improve the process time to ingest and store data in Amazon Redshift to conduct near real-time analytics. One solution would be to use Kinesis Data Stream to ingest the location data. Load the streaming data into the cluster using Amazon Redshift streaming ingestion. Could you also use Amazon Managed Streaming for Apache Kafka to ingest the location data and use Amazon Redshift Spectrum to deliver the data in the cluster? No. Redshift Spectrum is an Amazon Redshift feature to query data in Amazon S3 without loading that data into Amazon Redshift tables. Redshift Spectrum is not capable of moving data from S3 to Amazon Redshift. Could you use Firehose to ingest the location data and load the streaming data into the cluster using Amazon Redshift streaming ingestion? No, and I added this specifically because Firehose only streams data to certain AWS services. We added a link for Firehose destinations. Firehose is not a valid streaming source for Amazon Redshift streaming ingestion. Could you use Firehose to ingest the location data and set the Amazon Redshift cluster as a destination? No, you can configure Amazon Redshift as a destination for Firehose. But Firehose does not actually load the data directly into Amazon Redshift. It stages the data in Amazon S3, and then copies the data into Amazon Redshift using the copy command. Here's another question. You are migrating a data analysis solution to AWS. The application produces the data as CSV files in near real time. You need a solution to convert the data format to Apache Parquet before saving it to an S3 bucket. What is your solution? Well, my first thought is to use Kinesis Data Streams and create a streaming AWS Glue ETL job to convert the data into Apache Parquet. Also, as the streaming data is processed in the AWS Glue job, you have access to Spark Structured Streaming to implement data transformations to enrich or cleanse the data for analysis. Here's one last question. You are using Firehose to ingest data records from on premises. The records are compressed using GZIP compression. How can you perform SQL queries against the data stream to gain real-time insights and reduce the latency for queries? One solution is to use an AWS Managed Service for Apache Flink application configured with Lambda to transform the data. You can configure AWS Managed Service for Apache Flink to transform data before it's processed by your SQL code and use AWS Lambda to convert formats, enrich data, filter data, and more. After the data is transformed by your function, the Managed Service sends the data to your applications SQL code for real-time analytics, and provides Lambda blueprints for use cases such as converting GZIP and Kinesis Producer Library formats to JSON. You can use these blueprints without any change or writing your own custom functions. Could you transform the data with Amazon Kinesis Client Library and deliver the results to an Amazon OpenSearch cluster? No, KCL is a client library used for writing consumer applications for Kinesis Data Streams, and it is designed to read from Kinesis Data Streams not from a Firehose delivery stream. Let's get started with the second task statement from Domain 1, which is to transform data and perform feature engineering.