Skip to content

1.3

Task 1.3: Ensure data integrity and prepare data for modeling

Knowledge of:

  • Pre-training bias metrics for numeric, text, and image data (for example, class imbalance [CI], difference in proportions of labels [DPL])
  • Strategies to address CI in numeric, text, and image datasets (for example, synthetic data generation, resampling)
  • Techniques to encrypt data
  • Data classification, anonymization, and masking
  • Implications of compliance requirements (for example, personally identifiable information [PII], protected health information [PHI], data residency)

Skills in:

  • Validating data quality (for example, by using DataBrew and AWS Glue Data Quality)
  • Identifying and mitigating sources of bias in data (for example, selection bias, measurement bias) by using AWS tools (for example, SageMaker Clarify)
  • Preparing data to reduce prediction bias (for example, by using dataset splitting, shuffling, and augmentation)
  • Configuring data to load into the model training resource (for example, Amazon EFS, Amazon FSx)

Let's get started with the third task statement from Domain 1, which is to ensure data integrity and prepare data for modeling. This task statement is split into three lessons. The first step of the machine learning lifecycle is collecting a lot of quality data to input into your algorithm. After the data has been collected machine learning engineers process the data to ensure the quality of the data is high, ensure there is no missing data, ensure there is no erroneous data, and manipulating the data to be as efficient as possible to load into the next two steps of the machine learning lifecycle. For the exam ensure you understand bias and variance, which have roots in statistics, and it is important for machine learning engineers. The correct balance of bias and variance is important to build machine learning algorithms that create accurate results from their models. Let's talk about AWS services, features, and tools that can be used for pre-training bias metrics for numeric, text, and image data, and let's talk about Amazon SageMaker again. SageMaker has a feature to improve your machine learning models by detecting potential bias and helping explain the predictions that models make. It helps you to identify various types of bias in pre-training data and in post-training that can emerge during model training or when the model's in production. You can also evaluate a language model for model quality and responsibility metrics using foundation model evaluations. Amazon SageMaker Clarify can help you understand why your machine learning model made a specific prediction, and whether this bias impacts this prediction during training or inference. SageMaker Clarify also provides tools that can help you build less biased and more understandable machine learning models. SageMaker Clarify can also generate model governance reports that you can provide to risk and compliance teams and external regulators. Bias can be measured before training and after training, and monitored against baselines after deploying models to endpoints for inference. Pre-training bias metrics are designed to detect and measure bias in the raw data before it's used to train a model. The metrics used are model-agnostic because they do not depend on any model outputs. However, there are different concepts of fairness that require distinct measures of bias, and Amazon SageMaker Clarify helps to provide bias metrics to quantify various fairness criteria. I added a link for a cheat sheet for guidance and links to the pre-training bias metrics, so make sure you dive deeper into that documentation. I'm gonna stop this lesson here, and in the next lesson we will continue talking about task statement 1.3.

Let's continue with task statement 1.3, which is to ensure data integrity and prepare data for modeling and talk about strategies to address class imbalances in numeric text and image datasets. Which AWS service can you use to help address class imbalances? To train a machine learning model, you need a large, high-quality labeled dataset. You can label your data using SageMaker Ground Truth with Ground Truth built-in task types or create your own custom labeling workflow. Here's a question: what's the difference between augmented and synthetic data? Synthetic data is generated data that does not use the original dataset. Data augmentation artificially creates new training data from the existing training data. You can use the SageMaker image classification algorithm and generate more training data using data augmentation techniques. It is a supervised learning algorithm that supports multi-label classification and it uses a convolutional neural network that can be trained from scratch or trained using transferred learning when a large number of training images are not available. Here's a question: what is the difference between augmented and synthetic data? Synthetic data is generated data that does not use the original dataset. Data augmentation artificially creates new training data from the existing training data. You can use the SageMaker image classification algorithm and generate more training data using data augmentation techniques. It is a supervised learning algorithm that supports multi-label classification and it uses a convolutional neural network that can be trained from scratch or trained using transferred learning when a large number of training images are not available. Here's a question: what is the recommended input format for the SageMaker image classification algorithms? Apache MXNet RecordIO or raw images in .jpg or .png format. You can also use a data augmentation type hyper-parameter to configure your input images to be augmented in multiple ways. In task statement 1.2, we talked about how SageMaker Canvas provides numerous machine learning data transforms to clean, transform, and featurize your data. SageMaker Canvas has built-in transforms, which you can use to transform columns without any code. You can also add custom transformations. Some transforms operate in place while others create a new output column in your dataset. Here's a question: what can you do if you need data that is taken at regular intervals? Temporal or time series data usually has observations that aren't taken at regular intervals. One solution is to re-sample your data. For example, a dataset could have some observations that are recorded hourly or other observations that are recorded every 2 hours. If you are using a forecasting algorithm, it might require the observations to be taken at regular intervals. Re-sampling helps to establish regular intervals for the data in your dataset. You could either up-sample or down-sample a time series and you can re-sample both numeric and non-numeric data and use the re-sample operation to re-sample your time series data. Here's a question: why is it important to randomize training data? Randomizing your data when appropriate is important before training the data and you can split the data into training data and testing data. If we're trying to teach the algorithm to recognize the differences between images of a fish and a manatee and we input the images in groups that are not randomized, starting with the images of a fish, then the algorithm will think everything looks like a fish and the algorithm will be confident. And if we're changing our weights and biases as we're processing through the images, the model will start to have weights and biases that are prejudiced toward the fact that everything looks like a fish and trying to change that inside of the model becomes difficult. But if your data is randomized and you input the images into the model, then the model will see a difference of all of the class types that are in the dataset and each sample image going in will be randomized and it will adjust the weights and biases iteratively as it goes through all of the class types. Let's pause and go back to where I stated when appropriate. What is an example of when you should not randomize the data? Well, what if you're looking for the relationships between data? An example is if you have a time series dataset of records throughout the day, starting at midnight to midnight the next day. So in this case, you might use a recurrent neural network to load in data points and make a prediction of possible times to turn off your pool's heater based on what has happened in the past. So in this scenario, if you randomize the data, then you lose the context of the data. The ordering of the dataset is important, so you would not randomize the data. For the exam, also ensure you understand why you split the data into training data and testing data. I'm gonna stop this lesson here and in the next lesson we'll continue talking about task statement 1.3.

Let's continue with task statement 1.3 and talk about data formats. The actual data format and the way you present the data to the algorithm will depend on the format that the algorithm is expecting in the framework you are using. If you're using scikit-learn, Keras, or others, you need to look at the chosen algorithm, and it will explain how it needs to have the data formatted and loaded. If you're using Python code at a lower level, you can use NumPy arrays and load that in with your data into the algorithm. You will need to complete some preprocessing of your data, such as normalization or one-hot encoding. If you're using higher level implementation of algorithms such as SageMaker, it has prebuilt algorithms that expect data in formats such as JSON documents or manifest files. Also, there are different modes that you can provide data to an algorithm. For example, an image classifier with images saved separately in an S3 bucket, and the algorithm will have the needed access. For file mode, the algorithm will need to copy the images to an Amazon EBS drive, then iteratively process each of the files. This can take longer because of the extra copying and processing of each file. For pipe mode, you can have a recordIO file format type that takes all of the images and will create one file with all of the images. This is much faster than file mode and loading individual files one at a time. You can also use data quality rules in AWS Glue DataBrew. And AWS Glue Data Quality helps to measure and monitor the quality of your data and works with the Data Quality Definition Language. Here's a question. What are the two entry points for AWS Glue Data Quality? Both the AWS Glue Data Catalog and AWS Glue ETL jobs. When you create a training job in SageMaker, you specify the location of a training dataset and an input mode for accessing the dataset. For data location, SageMaker supports Amazon S3, Amazon EFS, and Amazon FSx for Lustre. The input modes determine whether to stream data files of the dataset in real time, or download the whole dataset at the start of the training job. For this task statement, also ensure you understand encryption and techniques to encrypt data in AWS. Here's a question. Your datasets contain confidential information and you must ensure that the data is encrypted and redacted. You need to build a machine learning model using these datasets. What is your solution? One solution is to store your data in Amazon S3 and choose AWS Key Management Service, or AWS KMS, and use AWS Glue to redact the sensitive information within the dataset. Then create an Amazon SageMaker Notebook and choose KMS encryption for the notebook. And then to encrypt the machine learning storage volume that is attached to the notebooks, preprocessing jobs, training jobs, hyper parameter tuning jobs, batch transform jobs, and endpoints, you can pass an AWS KMS key to SageMaker. Let's say you're using SageMaker Studio to analyze and build your models. You have five terabytes of data stored in an Amazon Redshift cluster. You need to choose a secure solution to export a subset of this data stored from your cluster and input that subset of data as your training data into Amazon S3. What is your solution? One solution is to use SageMaker Canvas to import data from Amazon Redshift. You can integrate a data preparation flow to preprocess and feature engineer your data, and then query data from the cluster and output the results to an S3 bucket. For added security, Amazon Redshift credentials can be stored as secrets in Secret Manager and use a library to retrieve the connection credentials at runtime. Another solution might be to create an Apache Spark job in AWS Glue to query data from the cluster and export the results to Amazon S3. Here's another question. You have a model that only accepts preprocessed data as input in generating real-time predictions. You are using scikit-learn to perform data processing and feature engineering on raw data before using the data for model training in SageMaker. So how can you preprocess this data most efficiently? One solution is to reuse the scikit-learn code in the preprocessing of your data using an inference pipeline. And you can add SageMaker Spark ML Serving and scikit-learn containers that reuse the data transformers developed for the training models. Then the inference pipeline can be considered as a SageMaker model that you can use to make real-time predictions or to process batch transforms directly without any external preprocessing. And to get us rolling into Domain 2, ensure you understand how to transform the output of a model, specifically the softmax activation function, the rectified linear unit function, and the Sigmoid function, and the hyperbolic tangent function. Let's get started with our first walkthrough question.