Skip to content

4.2

Task 4.2: Monitor and optimize infrastructure and costs

Knowledge of:

  • Key performance metrics for ML infrastructure (for example, utilization, throughput, availability, scalability, fault tolerance)
  • Monitoring and observability tools to troubleshoot latency and performance issues (for example, AWS X-Ray, Amazon CloudWatch Lambda Insights, Amazon CloudWatch Logs Insights)
  • How to use AWS CloudTrail to log, monitor, and invoke re-training activities
  • Differences between instance types and how they affect performance (for example, memory optimized, compute optimized, general purpose, inference optimized)
  • Capabilities of cost analysis tools (for example, AWS Cost Explorer, AWS Billing and Cost Management, AWS Trusted Advisor)
  • Cost tracking and allocation techniques (for example, resource tagging)

Skills in:

  • Configuring and using tools to troubleshoot and analyze resources (for example, CloudWatch Logs, CloudWatch alarms)
  • Creating CloudTrail trails
  • Setting up dashboards to monitor performance metrics (for example, by using Amazon Quick Sight, CloudWatch dashboards)
  • Monitoring infrastructure (for example, by using Amazon EventBridge events)
  • Rightsizing instance families and sizes (for example, by using SageMaker AI Inference Recommender and AWS Compute Optimizer)
  • Monitoring and resolving latency and scaling issues
  • Preparing infrastructure for cost monitoring (for example, by applying a tagging strategy)
  • Troubleshooting capacity concerns that involve cost and performance (for example, provisioned concurrency, service quotas, auto scaling)
  • Optimizing costs and setting cost quotas by using appropriate cost management tools (for example, AWS Cost Explorer, AWS Trusted Advisor, AWS Budgets)
  • Optimizing infrastructure costs by selecting purchasing options (for example, Spot Instances, On-Demand Instances, Reserved Instances, SageMaker AI Savings Plans)

Let's get started with the second task statement from Domain 4, which is to monitor and optimize infrastructure and cost. This task statement is split into four lessons. Monitoring is an important part of maintaining the reliability, availability, and performance of the machine learning lifecycle. AWS provides monitoring services and tools that you can use to monitor, report, and take action automatically, when appropriate to remediate, but you also have to ensure observability. For the exam, ensure you know how to implement the needed collection of monitoring of metrics, logs, and traces for observability and transparency, into your infrastructure, system, events, actions, API calls, and more, and ensure you understand how to implement a resiliency framework, using best practices. To make your application observable, it must be instrumented to share traces, metrics, and logs, to understand the raw data collected, gain insights from it, and add more visibility. AWS provides Amazon CloudWatch, AWS X-Ray, CloudWatch Logs Insights, Amazon GuardDuty, Amazon Inspector, AWS Security Hub, and more, to provide monitoring and observability. What AWS services are available to help you troubleshoot latency and performance issues in AWS? You can use CloudWatch to log and monitor SageMaker. For example, CloudWatch can be used to monitor SageMaker endpoints, and detect anomalies in their performance, which can help optimize resource usage and reduce costs. CloudWatch provides metrics related to SageMaker endpoints, such as endpoint latency, endpoint invocations, CPU and memory utilization, and data input and output rates. To keep track of batch and real-time predictions, you can monitor the predict count metric, according to the request mode dimension, and to keep track of batch and real-time predictions, you can monitor the predict count metric according to the request mode dimension. CloudWatch Logs can monitor information in the log files, and notify you when certain thresholds are met. You can create CloudWatch Event rules that react to a status change in a SageMaker training, hyperparameter tuning, or batch transform job. Using EventBridge, you can automate SageMaker, and respond automatically to events, such as training job status change, or endpoint status change. CloudWatch also provides several features to help analyze logs and metrics, such as CloudWatch application insights, to collectively define and monitor metrics and logs, for an application across different AWS resources, CloudWatch anomaly detection to surface anomalies for your metrics, and CloudWatch Logs Insights, to interactively search and analyze your log data in CloudWatch Logs. You can also use CloudWatch Service Lens to correlate traces, metrics, and logs and alarms, for diagnosing issues. SageMaker is also integrated with AWS CloudTrail, and provides a record of actions taken by a user role, or an AWS service in SageMaker. AWS Lambda automatically logs each invocation of your Lambda functions. You can also add custom log statements to your Lambda functions, using programming languages. CloudWatch Lambda Insights is a monitoring and troubleshooting solution for serverless applications, running on AWS Lambda, that collects, aggregates, and summarizes system-level metrics. And to ensure auditing and traceability in AWS, you can deploy logging and monitoring solutions that capture and analyze activities, and events across your infrastructure and services. What AWS services are available to ensure traceability? One service is AWS X-Ray, that provides end-to-end tracing and analysis, to visualize and understand the interactions between components of your application, performance optimization and issue resolution. And to ensure the accuracy and reliability of the data for traceability and audits, it is important to verify and clean your data. AWS provides services for data verification and cleaning, such as Lambda, Athena, QuickSight, Jupyter Notebooks, and Amazon SageMaker Canvas. I'm gonna stop this lesson here, and in the next lesson, we'll continue talking about task statement 4.2.

Let's continue with task statement 4.2, which is to monitor and optimize infrastructure and cost. Here's a question. You have a current S3 bucket that is shared by your team to store datasets for your CI/CD pipeline. The S3 bucket policy was accidentally changed and caused a denial of a download for a needed artifact and stopped the deployment. What solution can you add to ensure you are notified of any policy changes or issues? One solution is to create a CloudTrail trail to send logs to a CloudWatch logs group, create a CloudWatch metric filter for an S3 bucket policy event on the CloudWatch log group, and then create a CloudWatch alarm with a metric threshold of greater than or equal to one, and send a notification whenever this metric threshold is breached. You can also configure AWS CodeDeploy to use the rollback when alarm thresholds are met. Alert notifications and remediation actions are important for your entire system to ensure high availability, fault tolerance, and the security of your data. Here is a question. For data protection, what AWS service can you use to ensure you can trace integrity of each file and prevent the files from being tampered with? If you enable the log file integrity feature of AWS CloudTrail, CloudTrail will create a hash for every log file it delivers. Then each hour, CloudTrail creates and delivers a file that references the log files for the last hour and contains a hash of each. For the exam, ensure you understand automation and CloudWatch alarms. Let's say that you need to ensure all EC2 instances with low utilization are terminated to optimize cost. What is your solution? One solution is to integrate a CloudWatch event rule with AWS Trusted Advisor to detect Amazon EC2 instances with low utilization, then create a trigger with a Lambda function and create a second trigger to invoke a second Lambda function to terminate the underutilized Amazon EC2 instances. Here's another question. For your machine learning pipeline, you have a large Hadoop cluster with more than 100 Amazon EC2 instance nodes. Currently, you forward email notifications from the AWS Health service to the team if there is an instance that needs maintenance or an instance type that may be retired soon. Notifications are important because the team must decide how the needed maintenance will affect the cluster, but also to ensure that the Hadoop distributed file system component can recover from failure. How do you automate this notification process? One solution is to create an EventBridge rule for AWS Health and select the EC2 service and events you need notifications for, along with creating an Amazon Simple Notification Service topic and subscribe your team to that topic. Tracking metrics, logs, and events helps in the process of auditing your application, systems, and infrastructure in AWS to troubleshoot and identify root cause and implementing real-time logs to CloudWatch using subscription filters can deliver logs to Kinesis Stream, a Firehose stream, or Lambda for processing and analysis. I'm gonna stop this lesson here, and in the next lesson, we'll continue talking about task statement 4.2.

Let's continue with task statement 4.2, which is to monitor and optimize infrastructure and costs. To ensure cost optimization with real-time endpoint monitoring, you can use SageMaker tools and techniques, CloudWatch metrics and alerts, SageMaker Model Monitor, and Cost Explorer. The AWS Well-Architected Framework cost optimization pillar covers best practices, and right-sizing your instance is the first step in cost optimization. For machine learning, it would be right-sizing your EC2 instances and your database instances. You can monitor SageMaker notebook instances through CloudWatch custom metrics to optimize resource allocation and performance. SageMaker notebooks can be appropriately sized to minimize overutilized notebooks, which could cause errors or wasted spend. For the exam, ensure you understand how to choose the right instance family for your model, such as memory optimized, compute optimized, inference optimized, and more. I added a link under additional resources for instance families and sizes to dive deeper and ensure you know the different instance families and their general usage patterns to choose the best instance for cost and requirements. Here are a few questions you should ask. Do you need a solution that is more compute optimized for some type of batch processing, or perhaps you need a memory-optimized instance for processing large datasets in memory. The second cost optimization pillar is to increase elasticity. Ensure you understand auto scaling and the AWS Elastic Load Balancers for your use cases, requirements, and cost optimization, too. Here's a question. How do you troubleshoot latency with your SageMaker endpoint? For a single endpoint, you can use CloudWatch to monitor the latency metrics, model latency, and overhead latency for a SageMaker endpoint. If you use a SageMaker multi-model endpoint, you can also use CloudWatch to track model loading wait time, model downloading time, model loading time, and model cache hit. And then to reduce latency, ensure you benchmark the model outside of a SageMaker endpoint to test performance. If SageMaker Neo supports your model, then you can compile the model and SageMaker Neo will optimize the model to run up to twice as fast. If AWS Inferentia supports your model, then it can add up to three times higher throughput and up to 45% lower cost per inference compared to the AWS GPU-based instances. For further troubleshooting, remember that the inference code might affect the model latency, depending on how the code handles the inference. Any delays in code, increase the latency. Also, infrequent requests might also lead to an increase in overhead latency, and an overused endpoint might cause higher model latency. To dynamically increase and decrease the number of instances that are available for an endpoint, you can add auto scaling to the endpoint. Also, the first invocation for an endpoint might have an increase in latency because of a cold start, but you can prewarm the endpoint and send test requests. The third pillar for cost optimization is choosing the right pricing model. Ensure you understand the different pricing options, such as On-Demand, Savings Plans, Reserved Instances, Spot Instances, Dedicated hosts, Dedicated instances, Scheduled instances, Capacity reservations, and so on. For example, which pricing option would you choose if you had a batch processing job that was designed to handle interruptions and you wanted to minimize your cost? My first thought is spot instances. What pricing option would you choose if you couldn't handle interruptions, but you wanted to save as much as possible, and you want to have the flexibility of using EC2, Fargate, or Lambda for your compute needs? For this question, dive deeper into savings plans such as Compute Saving Plan, EC2 Savings Plan, and the SageMaker Saving Plan. The fourth pillar of cost optimization is to match your storage to your usage. Know the different use cases and requirements for different storage to match your workloads, and we talked about this under lessons for Domain 1, so go back and read those lessons if you need a refresher. The fifth pillar of cost optimization is continual improvement with measuring and monitoring to optimize your architecture. You need measures in place to monitor and track your metrics, measure your usage and your cost, and set alarms to immediately take action. You can also use other services like AWS Cost Explorer, AWS Trusted Advisor, AWS Budgets, and more to help define your metrics, set target goals, define and enforce your tagging strategy, use cost allocation tags, and make sure you review regularly for any infrastructure changes. I'm gonna stop this lesson here, and in the next lesson, we'll continue to talk about task statement 4.2.

Let's continue with task statement 4.2, which is to monitor and optimize infrastructure and costs. Let's wrap up this task statement and talk about implementing a monitoring and cost management solution to understand your resource needs, how costs are allocated, and measure and monitor your architecture. CloudWatch helps to track our metrics and alarms and immediately take actions, but what other services are available to help? Trusted Advisor can help to optimize costs, increase performance, improve security and resilience, and scale. But there are more services to help too, such as the AWS Well-Architected Framework Tool, AWS Cost Explorer, and more. It is crucial to define your metrics, set target goals, define and enforce your tagging strategy, use cost allocation tags, and make sure you review regularly for any infrastructure changes. This will require a good understanding of Cost Management tools. For example, how do you use cost allocation tags? One way to use tags is to filter views in Cost Explorer to help analyze costs. Also ensure you understand the capabilities of the AWS Cost Management tools. For example, if you wanted high-level, interactive financial reports, would you use AWS Cost Explorer or the AWS Cost and Usage Report? Cost Explorer gives you a high-level view that you can then drill down into more specifics. AWS Cost and Usage Reports breakdown costs by the hour, day, month, product, resource tags, and more. We've mentioned alarms and alerting with CloudWatch, but you can also set up alarms and alerts to automate notifications when your bill exceeds a certain threshold. You can create billing alarms, free-tier alarms, and also alarms with AWS Budgets. Ensure you understand how to optimize costs and set cost quotas by using appropriate Cost Management tools. Dive deeper into Cost Explorer, AWS Cost Management, AWS Organizations, and AWS Control Tower to help centrally manage billing, control access, compliance, security, and share resources too. There are also data and storage lifecycle rules that we did not mention under Domain 1 that you can use to automatically delete data that is no longer needed, and services such as Amazon Data Lifecycle Manager and AWS Backup that help you automatically delete old Amazon EBS snapshots or backups from other AWS services. And this can help you reduce the storage costs within an AWS account. There are also different options for optimizing the data migration calls for hybrid environments such as AWS DataSync, the Snow Family, the AWS Transfer Family, and AWS Storage Gateway. Architecting for data transfer can help to minimize data transfer costs. For example, you might use content delivery networks such as Amazon CloudFront, SageMaker Neo, or using dedicated private connections with AWS Direct Connect. Amazon SageMaker Inference Recommender helps you select the best instance types, configurations, optimizations to deploy your model to a real-time or serverless inference endpoint that delivers the best performance at the lowest cost for your machine learning models and workloads. AWS Compute Optimizer helps to analyze the configuration and utilization of metrics of your AWS resources and generates optimization recommendations to reduce the cost and improve the performance of your workload. Tags are so important for cost tracking and allocation techniques. You can use the AWS Cost Optimization Hub to identify, filter, and aggregate AWS cost optimization recommendations across your AWS accounts and Regions, which you can use to evaluate which recommendation provides the best price-performance tradeoff. Let's get started with the third task statement from Domain 4, which is to secure AWS resources.