Skip to content

3.1

Task 3.1: Select deployment infrastructure based on existing architecture and requirements

Knowledge of:

  • Deployment best practices (for example, versioning, rollback strategies)
  • AWS deployment services (for example, Amazon SageMaker AI)
  • Methods to serve ML models in real time and in batches
  • How to provision compute resources in production environments and test environments (for example, CPU, GPU)
  • Model and endpoint requirements for deployment endpoints (for example, serverless endpoints, real-time endpoints, asynchronous endpoints, batch inference)
  • How to choose appropriate containers (for example, provided or customized)
  • Methods to optimize models on edge devices (for example, SageMaker Neo)

Skills in:

  • Evaluating performance, cost, and latency tradeoffs
  • Choosing the appropriate compute environment for training and inference based on requirements (for example, GPU or CPU specifications, processor family, networking bandwidth)
  • Selecting the correct deployment orchestrator (for example, Apache Airflow, SageMaker Pipelines)
  • Selecting multi-model or multi-container deployments
  • Selecting the correct deployment target (for example, SageMaker AI endpoints, Kubernetes, Amazon ECS, Amazon Elastic Kubernetes Service [Amazon EKS], AWS Lambda)
  • Choosing model deployment strategies (for example, real time, batch)

Let's get started with the first task statement from Domain 3, which is to select deployment infrastructure based on existing architecture and requirements. This task statement is split into three lessons. Let's talk about the fifth step of the machine learning lifecycle, which is to host the model. A machine learning framework is a way to abstract away complexity of low-level code, and the many dimensions of the data. You do not have to code anything yourself, you can just use the frameworks which put the code together, and provide a library to get started building your machine learning solutions. For the exam, ensure you know the different frameworks, such as TensorFlow, PyTorch, scikit-learn, MXNet, Neural Networks, and the frameworks under AWS. There are also other frameworks that are listed as machine learning frameworks, that are not really frameworks, such as Keras and Gluon. Let's talk about different methods for hosting an inference, before getting into specifics of hosting in AWS. The model is a combination of the machine learning frameworks, configurations, and binary data. You can host the machine learning model, and then make inference. Inference is the process of using the trained model to create an output from new data. For the exam, ensure you understand the difference between batch inference and real-time inference, and use cases for both. Part of this task statement, and the next one too, is understanding how to deploy your machine learning workflows with containers. You can also use SageMaker to train algorithms, and the way that works is that you will deploy a Docker container to run a training job, and you can run hyperparameter tuning jobs, which is a batch way to run multiple training jobs, and monitor the performance metric, and tuning the jobs and tweaking your hyperparameters as much as it can for you. SageMaker can also perform inference, and host models, and create endpoints. SageMaker Inference Recommender reduces the time required to deploy machine learning models in production by automating load testing and model tuning across SageMaker machine learning instances. SageMaker provides multiple inference options, such as real-time inference, serverless inference, batch transform, and asynchronous inference. Let's ask a few questions on SageMaker hosting services. After you train your machine learning model, you can deploy your model using SageMaker to get predictions. What SageMaker hosting service do you use if you need persistent real-time endpoints that make one prediction at a time? SageMaker real-time hosting services and real-time inferences. What if you have workloads that have idle periods between traffic spurts, and can tolerate cold starts, what would you use? Serverless inference. What if your requests have large payload size, up to one gigabyte, long processing times, and near real-time latency requirements? What do you use? SageMaker Asynchronous Inference. What if you need to get predictions for an entire dataset, what would you use? SageMaker Batch Transform. SageMaker also provides features to manage resources and optimize inference performance when deploying machine learning models. SageMaker Neo helps you to train your model once, and then run that model anywhere in the cloud, and at the edge. Here's a question. You want to run inference for real-time data and test multiple variants of your model hosted in your development environment. To identify the best model to deploy to production, you are going to perform A/B testing. What is your solution? One solution is to deploy the model variants behind a single SageMaker endpoint, specify a traffic percentage for each variant, and shift all traffic to the best-performing model after the A/B test is done. Instead, could you use API Gateway to an API endpoint for each model variant, advertise the endpoints for A/B testing, delete all model endpoints that perform poorly, and only retain the API endpoint of the best-performing model? Well, this solution is possible, but you would have to publish a number of API endpoints that you would need to monitor and track. Using a single SageMaker endpoint is a less-effort solution for A/B testing, because it will perform load balancing on multiple models. I'm gonna stop this lesson here, and in the next lesson, we'll continue talking about task statement 3.1.

Let's continue with task statement 3.1, which is to select deployment infrastructure based on existing architecture and requirements. As you build compute architectures for machine learning, it is important to understand whether you're dealing with training or inference. If you are doing computer calculations or recognition of imagery, you will need a lot of compute power to train the models, because of the hundreds, thousands, or billions of calculations that are needed. And the processing of these calculations will require compute processing units, graphic processing units, and tensor processing units. And this ties into scaling. So let's talk about some of the aspects of training machine learning models and compute. Ensure you understand how you can use multiple cores and how you can distribute your code among multiple cores. You might need to increase the mini batch size to make use of the available cores. And sometimes, in machine learning, we need to scale vertically and increase the compute size, which increases the number of cores. But horizontal scaling is usually best, because you get more machines, but you need to ensure that the code, framework, and algorithm must support parallel training and running on multiple parallel machines. And this reduces the time to train, but remember that each machine will train a mini batch and they get split between the different machines and different cores. And you also might need to increase the learning rate if you increase the mini batch size. Here's a question. You are training an Apache MXNet model using SageMaker. You want to accelerate this inference workload, but also optimize cost and right size the instance. What is your solution? One solution is to use AWS Inferentia, which is a custom machine learning chip that you can use for high-performance inference predictions. Let's also talk about Amazon Elastic Map Reducer, Amazon EMR. You can use Amazon EMR Studio notebooks, which are managed Jupyter notebooks that you can use as an integrated development environment into the EMR environment to manage the code that runs inside of your EMR cluster. You can run Apache Spark with Python, Scala, or SQL, and you can analyze the data inside EMR from this in-memory perspective. And Amazon EMR with Spark has a library of a code called Spark MLLib. The algorithms are pre-baked into MLLib, and you can use them to train your data that is inside of EMR through Spark. And you can use MLLib to not only train the model, but run the model. And with this design, AWS is supporting this machine learning inside of EMR, but you can also use the SageMaker Spark Library. The SageMaker Spark Library is a specific SDK for your Spark code to interact with the SageMaker services. It can use the k-means algorithm, the principal component algorithm, or XGBoost, which are prebuilt algorithms inside of SageMaker. And you can also extend it out with the estimator model. And if you choose to use SageMaker Spark Library instead of MLLib, you will also not need cluster training and hosted production endpoints, so you do not need to off-size your cluster for the training job you need to complete. And you can host your production endpoints outside of your Amazon EMR cluster and have infrastructure spun up to make inference on your model. AWS Batch is a fully managed service to manage batch compute jobs. And the components of AWS Batch are the compute environments, job queues for prioritization, and job definitions, which are descriptions of what we want the job to look like. The options for AWS Batch compute environments are Fargate and EC2. If you use Fargate as your compute environment, you need to choose the max virtual centralized processing unit, or vCPUs, that you need to have available. If you choose EC2, you can set the maximum, minimum, and desired vCPU, or you can select GPUs. And for both, you can use on-demand instances or spot instances. I'm gonna stop this lesson here. And in the next lesson, we'll continue talking about task statement 3.1.

Let's continue with task statement 3.1, which is to select deployment infrastructure based on existing architecture and requirements. SageMaker provides infrastructure and model deployment options for inference, and it integrates with MLOp tools to help you scale your model deployment, reduce inference costs, manage models and production, and reduce the operational overhead. After you've built and trained your model, you can use SageMaker Inference to start getting predictions or inferences from your model. You can set up an endpoint that returns inferences or run batch inferences from your model. If you have multiple models that use different frameworks and require their own containers, then you can use the option of host multiple models, which use different containers behind one endpoint. With this option, you get many of the benefits of multi-model endpoints and can deploy a variety of frameworks and models. If you want to host the models with pre-processing and post-processing logic behind an endpoint, then you can use serial inference pipelines. Inference pipelines are fully managed by SageMaker and provide lower latency because all the containers are hosted on the same Amazon EC2 instances. You can also use an existing Docker container in SageMaker or create a new Docker container and receive guidance on how to run your own inference code. After you have an endpoint and understand the general inference workflow, you can use features of SageMaker Inference to improve your inference workflow. Another feature of SageMaker Inference is SageMaker MLOps which helps to automate the steps in your machine learning workflow and practice CI/CD. SageMaker also supports using your own third-party Git repositories for creating a CI/CD system. For your machine learning pipelines, you can use SageMaker Model Registry to manage your model versions and the deployment and automation of your models. And you can use deployment guardrails in SageMaker Inference to update your models in production to control the traffic switch from the current model in production to a new one. If you need to run large-scale machine learning and deep learning applications for use cases such as image or speech recognition, natural language processing, personalization forecasting, or fraud detection, you can use the Inf1 instance with a real-time endpoint. You can then deploy a model on Inf1 instances, compile your model with SageMaker Neo, and choose an Inf1 instance for your deployment option. Ensure you understand the features of SageMaker to manage resources and optimize inference performance when deploying your models. Let's get started with the second task statement from Domain 3, which is to create and script infrastructure based on existing architectures and requirements.