Skip to content

3.2

Task 3.2: Create and script infrastructure based on existing architecture and requirements

Knowledge of:

  • Difference between on-demand and provisioned resources
  • How to compare scaling policies
  • Tradeoffs and use cases of infrastructure as code (IaC) options (for example, AWS CloudFormation, AWS CDK)
  • Containerization concepts and AWS container services
  • How to use SageMaker AI endpoint auto scaling policies to meet scalability requirements (for example, based on demand, time)

Skills in:

  • Applying best practices to enable maintainable, scalable, and cost-effective ML solutions (for example, automatic scaling on SageMaker AI endpoints, dynamically adding Spot Instances, by using Amazon EC2 instances, by using Lambda behind the endpoints)
  • Automating the provisioning of compute resources, including communication between stacks (for example, by using CloudFormation, AWS CDK)
  • Building and maintaining containers (for example, Amazon ECR, Amazon EKS, Amazon ECS, by using bring your own container [BYOC] with SageMaker AI)
  • Configuring SageMaker AI endpoints within the VPC network
  • Deploying and hosting models by using the SageMaker AI SDK
  • Choosing specific metrics for auto scaling (for example, model latency, CPU utilization, invocations per instance)

Let's get started with the second task statement from Domain 3, which is to create and script infrastructure based on existing architectures and requirements. This task statement is split into two lessons. Let's talk about how to apply best practices to ensure your machine learning model is maintainable, scalable, and cost-efficient for your machine learning solutions. Let's start with scalability. SageMaker hosting service offers 100 instance types to host your model. You can choose when to auto scale and how many instances to provision or remove. SageMaker supports three auto scaling options. The first is target tracking. You select the value of an Amazon CloudWatch metric of your choice, such as the average CPU utilization or throughput that you want to achieve as a target, and SageMaker will automatically scale in or scale out the number of instances to achieve the target metric. The second option is to choose step scaling. You create and manage the CloudWatch alarms that initiate the scaling process. When an alarm is breached. Amazon EC2 auto scaling initiates the scaling policy associated with that alarm. The third option is scheduled scaling, which lets you specify a recurring schedule for scaling your endpoint in and out based on anticipated demand. AWS recommends that you combine these scaling options for better resilience. Sometimes, scaling up and down can provide better resilience for some traffic needs and scaling in and out. Here's a question. How do you scale up or down an existing endpoint? You must create a new production variant and shift traffic. For example, if you want a system to handle 1,000 requests per second scale up instead of out, Accelerator instances such as G5 and Inf1 can process higher numbers of requests on a single host. What is the solution if you have unpredictable traffic patterns or you don't want to set up scaling policies? You can use serverless inference for an endpoint and SageMaker manages the auto scaling for you. Serverless inference also integrates with AWS Lambda for high availability, built-in fault tolerance, and automatic scaling. Here's another question. You are using a SageMaker endpoint to predict outcomes. You need to ensure that the SageMaker model scales. What is your solution? One solution is to configure auto scaling on the SageMaker endpoint. When you host a model on SageMaker, you can also configure auto scaling to adjust the number of instances in response to the actual workload. Could you instead use an AWS Inferentia instance? No, AWS Inferentia is designed for accelerating inference workloads at a reduced cost. These instances can offer improved inference performance, but they can have failures due to a surge in a load. SageMaker multi-model endpoints support automatic scaling to help manage replicas of models to ensure model scale based on your traffic patterns. You can also optimize the cost of training your models by using managed spot instances, and for the exam, ensure you understand how to choose metrics for auto scaling. What metrics would you need to monitor for your models? My first thoughts are model latency, CPU utilization, invocations per instance, and explanations per instance. And we've talked about using Amazon CloudWatch to monitor SageMaker. You can build customized dashboards for your CloudWatch metrics that can display multiple metrics and include text and images. I'm gonna stop this lesson here, and in the next lesson, we'll continue talking about task statement 3.2.

Let's continue with task statement 3.2, which is to create and script infrastructure based on existing architecture and requirements. Let's continue talking about how to apply best practices for maintainable, scalable, and cost-efficient machine learning solutions. Let's talk about maintainability and automating the provision of your resources, communication, and security. Using Infrastructure as Code, you can use AWS CloudFormation templates to specify the definitions and configurations of your infrastructure that you're going to deploy. You can also use the AWS Cloud Development kit, or AWS CDK, for repeatable deployments. Then for your pipelines, you can use continuous integration and continuous delivery to implement, test, and deploy your pipelines. There's also the AWS Serverless Application model, or AWS SAM, to package and deploy serverless pipelines, such as Lambda functions, AWS Step Functions, and Amazon DynamoDB tables. For version control in AWS CloudFormation, you can commit files to a source code repository, such as AWS CodeCommit, Amazon S3, or GitHub. Then your CI/CD pipeline can be automated to run when a new version of the file is committed to the source code repository. The goal is to optimize your code to reduce runtime for data ingestion, transformation, training, and more. You can also use SageMaker Canvas ready-to-use models or bring your own model to generate predictions. The ready-to-use models in Canvas can extract insights from your data for different use cases and are powered by AWS AI services, such as Amazon Rekognition, Amazon Textract, and Amazon Comprehend. In an earlier lesson, I said that SageMaker provides APIs, SDKs, and a command line interface that you can use to create and manage notebook instances, and train and deploy models. I added a link under additional resources for the different SDKs and example notebooks. Another part of this task statement is building and maintaining containers. I don't wanna repeat myself because we already covered how SageMaker uses containers and more in previous lessons. But I added links under additional resources for you to dive deeper. Let's also talk about another best practice, which is ensuring security and secure connections. How do you configure secure connections for your Amazon Virtual Private Cloud, or Amazon VPC, networks and your SageMaker endpoints? Instead of connecting over the internet, you can configure and connect to the SageMaker API or to SageMaker Runtime through an interface endpoint. When you use a VPC interface endpoint, communication between your Amazon VPC and the SageMaker API or Runtime is within an AWS network, and these connections can be set in your CloudFormation templates, which ties back to maintainability and scalability. Each VPC endpoint is represented by one or more elastic network interfaces with private IP addresses in your Amazon VPC subnets. So if you have an application inside your Amazon VPC, it uses AWS PrivateLink to communicate with SageMaker Runtime, and then SageMaker Runtime communicates with the SageMaker Endpoint. The VPC Interface Endpoint connects to your Amazon VPC directly to the SageMaker API or SageMaker Runtime using AWS PrivateLink without using an internet gateway, NAT device, VPN connection, or an AWS Direct Connect connection. You can also connect to SageMaker Studio Classic and connect your notebook instances from your Amazon VPC through an interface endpoint in your Amazon VPC instead of connecting over the internet. Here's a question. You are using SageMaker notebooks to clean and train your data. You received a new security requirement that all training data must be stored in a private S3 bucket. What is your solution? One solution is to host the SageMaker notebooks in a private subnet of an Amazon VPC and associate a VPC endpoint for the SageMaker notebook and Amazon S3. Could you host the SageMaker notebook in a private subnet of a VPC, create an outbound network access control list rule that blocks access from the public internet? No, a network ACL will prevent your SageMaker notebooks from connecting to SageMaker APIs, which are needed to perform SageMaker operations. We will talk more about network ACLs in Domain 4. Let's get started with the third task statement from Domain 3, which is use automated orchestration tools to set up continuous integration and continuous delivery pipelines.