Task Statement 1.3: Describe the ML development lifecycle
Objectives:
- Describe and differentiate components of an AI/ML pipeline.
- Describe sources of FM models (for example, open source pre-trained models, training custom models).
- Describe methods to use a model in production (for example, managed API service, self-hosted API).
- Identify relevant AWS services and features for each stage of an AI/ML pipeline (for example, Amazon Bedrock, Amazon Q, Amazon Quick, Kiro, SageMaker AI).
- Describe fundamental concepts of ML operations (MLOps) (for example, experimentation, repeatable processes, scalable systems, managing technical debt, achieving production readiness, model monitoring, model re-training).
- Describe model performance metrics (for example, accuracy, precision, recall, F1 score) and business metrics (for example, cost per user, development costs, customer feedback, return on investment [ROI]) to evaluate ML models.
Lesson 1
Let's get started with the third task statement from Domain 1, which is to describe the ML development lifecycle. This task statement is split into seven lessons.
Let's begin with describing ML pipelines and some AWS services that are used in the pipeline stages. A machine learning pipeline is a series of interconnected steps that start with a business goal and finish with operating a deployed ML model. It starts with defining the problem, collecting and preparing training data, training the model, deploying, and finally, monitoring it.
As we'll see, some of these steps are an iterative process, and this is repeated until certain objectives are accomplished.
Machine learning models are dynamic by design. They are re-trained with new data, continually evaluated against performance and business metrics, monitoring for drifts and bias, and adjusted or rebuilt as needed. As a result, many prefer to think of the ML pipeline as a lifecycle, where parts, or even all of it, are repeated even after the model is deployed. Let's look at each phase in the ML pipeline in detail.
We'll discuss some of the AWS services that can be used to obtain the objects in each stage.
The development of an ML model should always begin with identifying the business goal. An organization considering ML should have a clear idea of the problem to be solved and the business value to be gained. More than just an idea, you must be able to measure business value against specific business objectives and success criteria. Without clear success criteria, you won't be able to evaluate the model or even determine if ML is the best solution.
You'll need to align stakeholders to gain consensus on what the goal of the project is. After you determine your criteria for success, evaluate your organization's ability to move toward the target. The target should be achievable and provide a clear path to production.
Determine if ML is the appropriate approach for delivering your business goal. Evaluate all the options that you have available for achieving the goal. Determine how accurate the resulting outcomes would be while considering the cost and scalability of each approach.
Ensure that enough relevant high-quality training data is available to the algorithm. Carefully evaluate the data to make sure that the correct data sources are available and accessible. Formulate the ML question in terms of input, desired outputs, and the performance metric to be optimized.
With the ML problem in mind, investigate all available options. Start with the simplest solution before determining that more complexity is required to meet the business objectives.
Remember to perform a cost-benefit analysis to see if the project should move to the next phase. AWS has introduced a number of AI services to democratize ML and make it accessible to anyone. They have identified many common use cases and developed easy, consumable, and fully trained ML models that are fully hosted by them. Because these services are pay as you go, it makes sense to evaluate them to see if they can meet the business goals. Many of these services allow you to customize their outputs.
For example, with Amazon Comprehend, you can create a custom classifier that uses your own categories by supplying it with your training data. If a hosted service doesn't achieve the objectives, the next consideration should be building your own model by starting with an existing one.
For example, for generative AI use cases, Amazon Bedrock lets you start with a fully trained foundation model. You can fine-tune this model with your own data using transfer learning. For other use cases, Amazon SageMaker has a number of open source pre-trained models to jumpstart your model development.
The most difficult and costly approach is to train your own model from scratch. As we will see in later sections, this is not only the most technically challenging, but also requires the most responsibility for security and compliance.
SageMaker JumpStart provides pre-trained AI foundation models and task-specific models for computer vision and natural language processing problem types. These are pre-trained on large public datasets. You have the option of fine-tuning the model with incremental training using your own dataset. This is a process known as transfer learning. Using a pre-trained model is a large savings in cost and development time over creating a custom model from scratch.
I'm going to pause this lesson. And in the next lesson, we will continue with task statement 1.3.
Lesson 2
Let's continue with a third task statement from Domain 1, which is to describe the ML development lifecycle.
Let's proceed to the next stages of the pipeline, which is collecting and processing training data. Start by identifying the data needed and determining the options for collecting the data. You'll need to know what training data you will need to develop your model and where it is generated and stored.
To collect the data, you'll need to know if it's streaming data or whether you can load it in batch process. You'll need to configure a process known as extract, transform, and load, ETL, to collect the data from possibly multiple sources and store it in a centralized repository. Remember that models should be re-trained frequently with new data, so your process needs to be repeatable.
You will need to know if the data is labeled or how you will be able to label it. This can be one of the longest parts of the process because accurately labeled data likely does not already exist.
Data preparation includes data pre-processing and feature engineering. Exploratory data analysis, EDA, with visualization tools can help to quickly gain a deeper understanding of data. You can use data wrangling tools for interactive data analysis and to prepare data for model building.
Data with missing or anomalous values might need to be filtered out or repaired. PII data should be masked or removed. After reprocessing your data, you are almost ready to start training. But first, you need to decide how best to split up your data. Typically, you will need to create three datasets from the available data. A common recommendation is that about 80% of the data should be used for training the model, 10% should be set aside for model evaluation, and 10% for performing the final test before deploying the model to production.
Finally, you need to determine which characteristics of the dataset should be used as features to train the model. This is the subset that is relevant and contributes to minimizing the error rate of a trained model. You should reduce the features in your training data to only those that are needed for inference.
Features can be combined to further reduce the number of features. Reducing the number of features reduces the amount of memory and computing power required for training. Now let's look at some of the many AWS services available for data ingestion and preparation.
AWS Glue is a fully managed ETL service. You can create and run an ETL job with a few clicks in the AWS Management Console. You simply point AWS Glue to your data stored on AWS. Then AWS Glue discovers your data and stores the associated metadata, the table definition and schema, in the AWS Glue Data Catalog.
After it's cataloged, your data is immediately searchable, queryable, and available for ETL. AWS Glue generates the code to execute your data transformations and data loading processes. In addition to defining your own data transformation, AWS Glue has built-in transformations for things like dropping duplicate records, filling in missing values, and splitting your dataset. AWS Glue can extract, transform, and load data from a large variety of data stores. These include relational databases, data warehouses, and other cloud, or even streaming services, such as Amazon Managed Streaming for Apache Kafka, or Amazon MSK, and Amazon Kinesis.
The AWS Glue Data Catalog contains references to data that is used as sources and targets of your ETL jobs in AWS Glue. The AWS Glue Data Catalog tables include an index to the location, schema, and runtime metrics of your data. You use the information in the Data Catalog to create and monitor your ETL jobs. Typically, you run a crawler to take inventory of the data in your data stores, but you can also enter the information in the tables manually.
AWS Glue can crawl your data sources and automatically determine the data schema by using classifiers. It writes the schema to tables in the Data Catalog. It's important to understand that the source data itself is not written to the data. Only metadata, such as the location and schema, are stored in the Data Catalog.
The AWS Glue ETL jobs use this information to collect, transform, and store the data in the target data store, which is typically an S3 bucket.
AWS Glue DataBrew is a visual data preparation tool that enables users to clean and normalize data without writing any code. You can interactively discover, visualize, clean, and transform raw data. DataBrew makes smart suggestions to help you identify data quality issues that can be difficult to find and time-consuming to fix. You can save transformation steps in a recipe, which you can update or reuse later with other datasets and deploy on a continuing basis.
DataBrew provides more than 250 built-in transformations, with a visual point-and-click interface for creating and managing data transformation jobs. These include removing nulls, replacing missing values, fixing schema inconsistencies, creating column-based functions and more. You can also use DataBrew to evaluate the quality of your data by defining rule sets and running profiling jobs.
To train a supervised machine learning model, you need a large, high-quality labeled dataset. SageMaker Ground Truth helps you build high-quality training datasets for your machine learning models. SageMaker Ground Truth active learning uses machine learning model to label your training data. It will automatically label data that it can label, and the rest is given to a human workforce. You can use human workers from Amazon Mechanical Turk, a workforce with over 500,000 independent contractors. Or use an internal private workforce that uses your own employees or contractors.
You can use Amazon SageMaker Canvas to prepare, featurize, and analyze your data. With Amazon SageMaker Canvas, you can simplify the feature engineering process by using a single visual interface. Using the SageMaker Data Wrangler data selection tool, you can choose the raw data that you want from various data sources and import it with a single click.
SageMaker Canvas contains over 300 built-in transformations so that you can quickly normalize, transform, and combine features without having to write any code.
Amazon SageMaker Feature Store is a centralized store for features and associated metadata, so features can be easily discovered and reused. Feature Store makes it easy to create, share, and manage features for ML development.
Feature Store accelerates this process by reducing repetitive data processing and curation work required to convert raw data into features for training an ML algorithm. You can create workflow pipelines that convert raw data into features and add them to feature groups.
Lesson 3
Let's continue with the third task statement from Domain 1, which is to describe the ML development lifecycle.
The next phase of the pipeline is where we get to train, tune, and evaluate the model. In this phase, we are teaching the model, through an iterative process of training, tuning, and evaluating. During training, the machine learning algorithm updates a set of numbers, known as parameters or weights.
The goal is to update the parameters in the model in such a way that the inference matches the expected output. This can't be done in one iteration, because the algorithm has not learned yet. It has no knowledge of how changing weights will shift the output closer toward the expected value. Therefore, it watches the weights and outputs from previous iterations, and shifts the weights to a direction that lowers the error in generated output.
This iterative process stops either when a defined number of iterations have been run, or when the change in error is below a target value. There are usually multiple algorithms to consider for a model. The best practice is to run many training jobs in parallel, by using different algorithms and settings. This is known as running experiments, which helps you land on the best-performing solution. Each algorithm has a set of external parameters that affect its performance, known as hyperparameters. These are set by the data scientists before training the model. These include adjusting things like how many neural layers and nodes there will be in a deep learning model. The optimal values for the hyperparameters can only be determined by running multiple experiments with different settings.
To train your model using SageMaker, you create a training job which runs your training code on a fleet of ML compute instances managed by SageMaker. To create a training job, you specify the URL of the S3 bucket containing your training data. You also specify the compute resources you want to use for training, and the output bucket for the model artifacts. You specify the algorithm by giving SageMaker the path to a Docker container image that contains the training algorithm. In the Amazon Elastic Container Registry, Amazon ECR, you can specify the location of SageMaker provided algorithms and deep learning containers, or the location of your custom container, containing a custom algorithm.
You also need to set the hyperparameters required by the algorithm. After you create the training job, SageMaker launches the ML compute instances, and uses the training code and the training dataset to train the model. It saves the resulting model artifacts and other outputs in the S3 bucket you specified for that purpose.
We've seen that machine learning is an iterative process. You need to experiment with multiple combinations of data, algorithms, and parameters, all while observing the impact of incremental changes on model accuracy. This iterative experimentation can result in thousands of model training runs and model versions.
Amazon SageMaker experiments is a capability of Amazon SageMaker that lets you create, manage, analyze, and compare your machine learning experiments. An experiment is a group of training runs, each with different inputs, parameters, and configurations. It features a visual interface to browse your active and past experiments, compare runs on key performance metrics, and identify the best-performing models.
Amazon SageMaker automatic model tuning, AMT, also known as hyperparameter tuning, finds the best version of a model, by running many training jobs on your dataset. To do this, AMT uses the algorithm and ranges of hyperparameters that you specify. It then chooses the hyperparameter values that create a model that performs it best, as measured by a metric that you choose. For example, suppose that you are tuning a binary classification model.
You can have automatic model tuning find the combination of hyperparameters that maximizes a metric known as the area under the curve. To use automatic model tuning, you can figure a tuning job that runs several training jobs inside a loop. You specify completion criteria as the number of jobs that are no longer improving the metric. The job will run until the completion criteria are satisfied.
Lesson 4
Let's continue with the third task statement from Domain 1, which is to describe the ML development lifecycle. Next, let's talk about how we can deploy our model so it can be used for inferences.
Now that we have a fully trained, tuned, and evaluated model, we need to make it available for use. The first decision is whether you need batch or real-time inferencing or something in between. Recall that batch is for when you need a large number of inferences, and it's okay to wait for the results. Perhaps the batch process runs overnight on the data collected from the previous day sales. This is the most cost-effective approach because cloud computing resources are only running once per day.
With real-time inference, you deploy your model so that it can respond to requests immediately. For example, when using generative AI. Clients interact with your model by using a REST application programming interface, API. An API is a set of actions that are made available over an HTTP connection.
For example, a web application can send a POST request containing input data and your endpoint, which will pass the request to a compute resource that is running the model. The resulting model output is sent back to the client in the response to the request.
In this example, Amazon API Gateway can serve as the interface with the clients and forward requests to an AWS Lambda function, which is running the model. In both cases, the inference code and model artifacts are typically deployed as a Docker container.
Docker containers are very versatile and can be run on any compute resource with a container runtime installed. On AWS, this could include AWS Batch, Amazon Elastic Container Service, or Amazon ECS, Amazon Elastic Kubernetes Service, or Amazon EKS, AWS Lambda, Amazon Elastic Compute Cloud, or Amazon EC2, and others.
Depending on the service these options will require you to configure and manage the inference endpoint, which might also include managing updates, patches, scalability, network routing, and security. For reduced operational overhead you can choose to host your model with Amazon SageMaker.
Amazon SageMaker can automatically deploy your model on hosted endpoints that it fully manages on your behalf. To use SageMaker inferencing, you just point SageMaker to your model artifacts in an S3 bucket and a Docker container image in Amazon ECR. You select which inference option such as batch, asynchronous, serverless, or real time, and SageMaker creates the endpoint and installs your model code.
For real-time, asynchronous, and batch inference, SageMaker runs the model on EC2 ML instances, which can be inside an auto scaling group. You select the number and instance type of the ML instances that you want to use.
There is also an inference recommender tool within SageMaker that can test out different configuration options with your model, so you can pick the best one. For the serverless inference option, SageMaker runs your code on Lambda functions.
When you create an endpoint or endpoint configuration, you must choose an inference option. Amazon SageMaker supports four option types. The best choice depends on the business requirements of your ML inference workload. These endpoints are fully managed and support auto scaling. Batch transform provides offline inference for large datasets. It's useful when running inference if a persistent endpoint is not required and you can wait for the results. It can support large datasets that are gigabytes in size.
Asynchronous inference is ideal when you want to queue request and have large payloads with processing times. SageMaker will scale your endpoint down to zero so that you aren't charged for periods without requests.
Serverless inference can be used to serve model inference requests in real time without directly provisioning compute instances, or configuring scaling policies to handle traffic variations. Because it uses Lambda, you only pay when functions are running or pre-provisioned, so it is a good choice if your model has periods without requests.
Real-time inference is ideal for inference workloads where you need real-time interactive responses from your model. Use real-time inference for a persistent and fully managed endpoint REST API that can handle sustained traffic backed by the instance type of your choice. The ML instances remain available to receive requests and return a response in real time.
Lesson 5
Let's continue with the third task statement from Domain 1, which is to describe the ML development lifecycle. No matter how great your model performs initially, model performance could degrade over time for reasons such as data quality, model quality, and model bias.
The final stage of the ML pipeline is to monitor your model. The model monitoring system must capture data, compare the data to the training set, define rules to detect issues, and send alerts. This process repeats on a defined schedule when initiated by an event or when initiated by human intervention.
For most ML models, a simple scheduled approach for re-training daily, weekly, or monthly is usually enough. The monitoring system should detect data and concept drifts, initiate an alert, and send it to an alarm manager system, which could automatically start a re-training cycle.
Data drift is when there are significant changes to the data distribution compared to the data used for training. Concept drift is when the properties of the target variables change. Any kind of drift results in model performance degradation.
Amazon SageMaker Model Monitor, which is a capability of Amazon SageMaker, monitors models in production and detects errors so you can take remedial actions. You define a monitoring schedule that collects data from your endpoints and detects changes against the baseline. It analyzes the data based upon built-in rules or rules that you define. You can view the results in Amazon SageMaker Studio and see which rules were violated. The results are also sent to Amazon CloudWatch, where you can configure alarms to take remedial actions, such as starting a re-training process.
Automation is an important part of implementing and operating repeatable and reliable business processes. So let's look at how we can use automation in our ML pipelines.
MLOps is about using these established best practices of software engineering and applying them to machine learning model development. It's about automating manual tasks, testing, and evaluating code before release, and responding automatically to incidents.
MLOps can streamline model delivery across the machine learning development lifecycle. Because the cloud uses API based services, everything is treated as software. This includes the infrastructure used in ML pipelines. The entire infrastructure can be described in software and deployed and redeployed in repeatable fashion. This lets data scientists quickly spin up the infrastructure needed to build and test a model so they can run experiments and make continual improvements.
Like DevOps, version control is critical for tracking lineage and being able to inspect a past configuration. With MLOps, everything gets versioned, including the training data. Other key MLOps principles are monitoring deployments to detect potential issues and automating re-training because of issues or data and code changes.
One of the benefits of MLOps is productivity, automation and providing self-service environments and infrastructure let data engineers and data scientists move forward. Another benefit is repeatability. Automating all the steps in the ML lifecycle helps ensure a repeatable process, including how the model is trained, evaluated, version, and deployed. This also improves reliability because it provides the ability to deploy not only quickly, but with increased quality and consistency.
For compliance, MLOps can improve auditability by versioning all inputs and outputs from data science experiments to source data to trained models. This means that we can demonstrate exactly how the model was built and where it was deployed. The final benefit is improvements to data and model quality.
MLOps lets us enforce policies that guard against model bias and track changes to data statistical properties, and model quality over time. Amazon SageMaker Pipelines offers the ability to orchestrate SageMaker jobs and author reproducible ML pipelines.
SageMaker Pipelines can deploy custom built models for inference in real time with low latency, run offline inferences with batch transform and track lineage of artifacts. They can institute sound operational practices in deploying and monitoring production workflows, deploying model artifacts, and tracking artifact lineage through a simple interface.
You can create a pipeline using the SageMaker SDK for Python or define the pipeline using JSON. The pipeline can contain all the steps to build and deploy a model, and can also include conditional branches based on the output of a previous step. Pipelines can be viewed in SageMaker Studio. This example pipeline is for a model that infers the age of an abalone based on its size.
Lesson 6
Let's continue with the third task statement from domain 1, which is to describe the ML development lifecycle.
It's important to mention a few other services for MLOps. Repositories are where you keep versions of your code and models.
SageMaker Feature Store is a repository for the feature definitions of your training data and SageMaker Model Registry is a centralized repository for your trained models and history. We've already seen how SageMaker Pipelines can orchestrate your ML pipeline, but there are a few other options.
One is AWS Step Functions, which lets you define a workflow with a visual drag-and-drop interface. It gives you the ability to build serverless workflows that integrate various AWS services and custom application logic.
Apache Airflow is an open source tool used to programmatically author, schedule, and monitor sequences of processes and tasks referred to as workflows. With Amazon Managed Workflows for Apache Airflow, you can use Apache Airflow and Python to create workflows without having to manage the underlying infrastructure for scalability, availability, and security.
We've mentioned how models need to be evaluated against target metrics. Let's describe some of these in more detail. A confusion matrix is used to summarize the performance of a classification model when it's evaluated against test data. The simplest way would be a binary classification model where the output is a simple binary result, yes or a no, positive or a negative.
A confusion matrix is a table with actual data typically across the top and the predicted values on the left. Let's continue to use our fish classification model to illustrate. If the model sees a picture of a fish and accurately predicts that it is a fish, it is called a true positive. If the model sees a picture of something that is not a fish and accurately predicts that is not a fish, that is called a true negative. If the model sees a picture of something that is a fish and incorrectly predicts that it's not a fish, that is called a false negative. If the model sees a picture of something that is not a fish and incorrectly predicts that it is a fish, it is called a false positive.
Let's make up an example where we ran 100 labeled test images through the model. The confusion matrix shows the number of true and false positives and negatives. One metric that is sometimes used to judge a model's performance is accuracy, which is simply the percentage of correct predictions.
Accuracy measures how close the predicted class values are to the actual values. Values for accuracy metrics vary between zero and one. A value of one indicates perfect accuracy and zero indicates complete inaccuracy. The formula for accuracy is the number of true positives plus true negatives divided by the total number of predictions.
Using our confusion matrix, that works out to 25 plus 40, which is 65, divided by 100, which gives us an accuracy of 0.65 or 65%. Though accuracy is understandable, it is not a good metric when the dataset is imbalanced. For example, suppose that 90 of 100 of our test images are fish, then the model only has to predict that all our images are fish and it'll get an accuracy of 0.9 or 90%.
Precision measures how well an algorithm predicts true positives out of all the positives that it identifies. The formula is the number of true positives divided by the number of true positives, plus the number of false positives. This is a good quality metric to use when your goal is to minimize the number of false positives.
For example, we don't want to label a legitimate email as spam. Using our confusion matrix, we at a precision of 0.55 or 55% for our fish model. If we want to minimize the false negatives, then we can use a metric known as recall. For example, we want to make sure that we don't miss if someone has a disease and we say they don't. The formula is the number of true positives divided by the number of true positives, plus the number of false negatives. Using our confusion matrix, we arrive at a recall of 0.625 for the model.
There is a tradeoff between precision and recall because you can't optimize a model for both. For example, suppose we want to make sure we diagnose everyone who has a disease and not miss anyone. Then we're going to increase the likelihood of diagnosing people who don't have the disease as having it.
Recall, is also known as sensitivity or the true positive rate. However, if recall and precision are both important to us, the F1 score balances precision and recall by combining them in a single metric. Our fish model has better recall than precision, which means it'll be better at detecting true positives, but it will also have some false negatives. In this scenario, optimizing the F1 score is the best compromise.
I'm going to pause this lesson here and in the next lesson we'll wrap up task statement 1.3.
Lesson 7
Let's continue with the third task statement from Domain 1, which is to describe the ML development lifecycle and jump back into talking about metrics to evaluate your models.
Another metric we can calculate from our confusion matrix is the false positive rate, which is the false positives divided by the sum of the false positives and true negatives. In our example, this metric shows us how the model is handling the images that are not fish. It is a measure of how many of the predictions were of fish out of the images that were not fish.
Closely related to the false positive rate is the true negative rate, which is the ratio of the true negatives to the sum of the false positives and true negatives. It is a measure of how many of the predictions were of not fish out of the images that were not fish.
The area under the curve, also known as AUC metric, is used to compare and evaluate binary classification by algorithms that return probabilities, such as logistic regression. To map the probabilities into discrete predictions such as true or false, these are compared against a threshold value.
A threshold is a value that the model uses to make a decision between the two possible classes. It can converts the probability of a sample being part of a class into a binary decision. For example, when the threshold is set to 0.6, anytime the model is 60% confident that the image is of a fish, it will classify it as a fish.
The true positive rate is plotted against the false positive rate for increasing threshold values. The threshold values are represented by the red dashed line in the graph. The relevant curve is called the receiver operating characteristic curve. You can see that increasing the threshold results in fewer false positives, but more false negatives. AUC is the area under this receiver operating characteristic curve.
AUC provides an aggregated measure of the model performance across the full range of thresholds. AUC scores vary between zero and one. A score of one indicates perfect accuracy and a score of one half, or 0.5, indicates that the prediction is no better than a random classifier.
Recall that in linear regression, we're fitting a line to the points in a dataset. The distance between the line and the actual values is the error. A metric that we can use to evaluate a linear regression model is called the mean squared error, MSE. To compute it, we take the difference between the prediction and actual value, square the difference, and then compute the average of all square differences. MSE values are always positive. The better a model is at predicting the actual values, the smaller the MSE value is.
Another metric that is commonly used is the root mean squared error, which is the square root of the mean squared error. The advantage of using this square root of the MSE is that the units match the dependent variable. In our example, if the height is measured in inches, then the MSE will be in square inches, but the RMSE is in inches, so the RMSE is easier for us to interpret.
Because the errors are squared, the mean squared error and root means squared error metrics emphasize the impact of outliers. These are good metrics, but incorrect predictions can be very costly. If that is not desired, a different metric called mean absolute error averages the absolute values of the errors, so it doesn't emphasize the large errors.
We've seen that ML solutions need to be rooted in solving business problems, so let's end this domain by discussing business metrics. Recall that the first step in the ML pipeline is to define the business goal. From there, we determine how to measure successful achievement of the goal.
Business metrics help us quantify the value of a machine learning model to the business. Good metrics could be cost reduction, a percentage increase in users or sales, a measurable improvement in customer feedback, or any measurable metric that is important to the business. It is important to also estimate the risks of using AI and ML and potential costs incurred from errors.
For example, one potential cost could be the loss of sales or customers. After the model is in production, you need to be able to collect the data to support the metrics and compare the actual results with the original business goals. Also, consider the actual cost of building and operating the model and compare this cost with the initial cost benefit model. This way you'll be able to calculate the return on investment.
AWS allows you to define cost allocation tags that are assigned to the resources that you create. For example, you can define a tag with the name of ML project and the name of your project as the value. You add that tag to all the resources used in your pipeline. Then you can filter the cost reports in AWS Cost Explorer to determine the actual AWS charges incurred for the project.
Okay, let's get started with our walkthrough questions.