4.1
Task 4.1: Monitor model inference
Knowledge of:
- Drift in ML models
- Techniques to monitor data quality and model performance
- Design principles for ML lenses relevant to monitoring
Skills in:
- Monitoring models in production (for example, by using Amazon SageMaker Model Monitor)
- Monitoring workflows to detect anomalies or errors in data processing or model inference
- Detecting changes in the distribution of data that can affect model performance (for example, by using SageMaker Clarify)
- Monitoring model performance in production by using A/B testing
Let's get started with the first task statement from Domain 4, which is to monitor model inference. This task statement is split into two lessons. Let's talk about the sixth step of both the machine learning engineer and the well-architected machine learning lifecycles, which is to make inference and model monitoring. In AWS, your model monitoring system must automatically capture data, compare that data to the training set, define rules to detect issues and send alerts. The issues detected in the monitoring phase include data quality, model quality, bias drift and feature attribution drift. There is often a need to re-train a model using new or updated data to ensure that the model is able to effectively learn and predict based on the most current data available, and to be able to incorporate additional data into a machine learning model, there must be a mechanism implemented to analyze existing model performance against defined metrics and initiate an alarm or a re-training event when the model variance reaches a specific threshold or proactively re-train the model over time based on new known data. For a model to predict accurately, the data that it is making predictions on must have a similar distribution as the data on which the model was trained. Because data distributions can be expected to drift over time, deploying a model is not a one-time exercise, but rather a continuous process. It is a good practice to continuously monitor the incoming data and re-train your model on newer data if you find that the data distribution has deviated significantly from the original training data distribution. To track your model over time through metrics such as model accuracy and drift, you can use Amazon SageMaker Model Monitor and set alerts that notify you when there are deviations in your model's quality. For example, you can monitor your endpoint's health through metrics such as invocation errors and model latency using CloudWatch metrics. The SageMaker endpoint invocation metrics can provide information about your endpoint's performance. You can set up continuous monitoring with a real-time endpoint or a batch transform job that runs regularly or on schedule monitoring for asynchronous batch transform jobs. For the exam ensure you understand how to monitor the data and the model's predictions, run analysis of model performance against defined metrics to identify errors due to data and concept drift, automate model re-training to mitigate those errors on a fixed schedule intervals or when the model variance reaches a defined threshold, or when the new data becomes available. You can develop a re-training pipeline using Amazon SageMaker Pipelines for orchestration using step creation and management, and you can use SageMaker Projects to orchestrate and manage building custom images for processing, training and inference, data preparation and feature engineering, training models, evaluating models, deploying models and monitoring and updating the models, which is the focus for this task statement. You can also use AWS Step Functions Data Science SDK for Amazon SageMaker to automate training of the machine learning model. How does this work? Well, you define the steps in the workflow and set up alerts to start the flow, then to detect the new training data in an S3 bucket, you can use AWS CloudTrail or Amazon CloudWatch Events to start an AWS Step Function workflow to initiate re-training task in your training pipeline. I'm gonna stop this lesson here, and in the next lesson, we'll continue talking about task statement 4.1.
Let's continue with task statement 4.1, which is to monitor model inference, and let's start with a question. How do you understand the effects of data drift on model performance and implement strategies that monitor and adapt to data drift through re-training? One solution is to use SageMaker Model Monitor and SageMaker Clarify to maintain high-quality machine learning models and to detect model and concept drift in real time. Model Monitor is integrated with SageMaker Clarify to help identify potential bias in your machine learning models, and you can configure alerts, receive notifications and take immediate action. Here's another question. Your model performance has degraded over time for reasons such as data quality, model quality, model bias and model explainability. You want to continuously monitor the quality of the machine learning model in real time. You also want to identify the right time and frequency to re-train and update the model. You need to configure alerts to notify and initiate actions if any drift in model performance is observed. What is your solution? You can monitor model performance with Model Monitor and establish a baseline during the training and before the model is in production, then collect data while in production and compare changes in model inferences. You can use SageMaker Clarify to identify model bias, and you can configure alerting systems with CloudWatch to send notifications for unexpected bias or changes in data quality, then you can monitor the model performance in production by using A/B testing. What is your solution to also configure automatic scaling for your monitored metrics? Remember SageMaker includes automatic scaling capabilities for your hosted model to dynamically adjust underlying compute supporting and endpoint based on a demand. If you also want to monitor endpoint metrics, SageMaker also outputs endpoint metrics for monitoring the usage and health of the endpoint. SageMaker Model Monitor helps you to monitor your models in production and provides alert when data quality issues appear. You can create a mechanism to aggregate and analyze model prediction endpoint metrics using services, such as Amazon OpenSearch Service, which supports Kibana for dashboards and visualization. The traceability of hosting metrics back to version inputs helps to analyze changes that could be impacting current operational performance, and remember to include human in the loop monitoring to monitor model performance efficiently. When automating decision processes, the human labeling of model results is a reliable quality test for model inferences. You can compare human labels with model inferences to estimate model performance degradation, and you can perform mitigation as model re-training. You can use Amazon Augmented AI to get human reviews. How do you monitor workflows to detect anomalies or errors in the data processing or model inference? One solution is to use AWS Glue Data Quality anomaly detection to apply machine learning algorithms on data statistics over time to detect abnormal patterns and hidden data quality issues that are hard to detect through rules. You can also use Amazon Lookout for Metrics, which is a service that finds anomalies in your data, determines their root causes and helps you to quickly take action. You can add alerts and configure the alerts to run a Lambda function or send notification to an Amazon SNS topic. Also, Monitor Model has model quality monitoring jobs that monitor the performance of a model by comparing the predictions that the model makes with the actual ground truth labels that the model attempts to predict. Here's a question. You deployed a machine learning model in SageMaker that provides product recommendations, but you have noticed that the total number of customers purchasing the recommended products has been declining over the last few weeks. What is your solution to improve the accuracy of the model and ensure it is providing high-value predictions? If you detect a significant change in customer data, you can re-train your model using both the original, updated, and new data. Can you re-train the model periodically using the original data and apply regularization to improve prediction accuracy? Regularization does reduce overfitting, but re-training the model with the original data is unlikely to produce better results. Can you tune the hyper parameters to reduce model drift? Well, this solution also would be working with the original data. Instead, you need to re-train the model based on the original data plus the new data to give product recommendations. Monitoring is also important to maintain the performance of your AWS CodePipeline and storage of your metrics, traces and logs can be in CloudWatch and AWS CloudTrail. Let's get started with the second task statement from Domain 4, which is to monitor and optimize infrastructure and costs.