Task 3.1 - Manage deployment infrastructure for ML and AI model types

Overview

Task 3.1, Manage deployment infrastructure for ML and AI model types, is about deciding where a model runs and how requests reach it.

The requirements do most of the deciding: who is waiting for the answer, how quickly they need it, how the work arrives, and how large each unit of work is. Those answers point at a hosting option, and the hosting option then constrains what else is available to you.

As you review these lessons for Task 3.1, check that you understand how to do the following.

Select appropriate compute environments and deployment targets.
Select deployment orchestrators and multi-model or multi-container deployment strategies.
Select model inference strategies (for example, real-time and batch processing).
Evaluate and select appropriate foundation model (FM) deployment options.
Deploy models that were built outside of AWS into AWS environments (for example, Amazon SageMaker AI, Amazon Bedrock Custom Model Import).
Deploy and configure agents for specific tasks, integration with other services and tools, and agent communication protocols.
Configure FM deployment, model hosting, and resource allocation.
Apply Retrieval Augmented Generation (RAG) system configurations (for example, retrieval strategies, reranking).

AWS services overview

The services in this task answer two questions: what hosts the model, and what surrounds it once it is hosted. Recognize which ones are managed hosting for a model you supply, which ones serve a foundation model for you, and which ones are general compute you would operate yourself. The following carousel covers Amazon SageMaker AI, Amazon SageMaker JumpStart, Amazon Bedrock, Amazon Bedrock Custom Model Import, Amazon Bedrock AgentCore, Amazon Bedrock Knowledge Bases, Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Kubernetes Service (Amazon EKS), AWS Lambda, and Amazon Elastic Compute Cloud (Amazon EC2). Review each one to learn more.

AWS Service	Description
Amazon SageMaker AI	The managed service for hosting a model you supply, with four inference options: real-time endpoints, serverless endpoints, asynchronous endpoints, and batch transform jobs.
Amazon SageMaker JumpStart	A catalog of pretrained models that can be deployed to a preconfigured endpoint without authoring container settings first.
Amazon Bedrock	Serves foundation models through an API, so there is no endpoint or instance to size. Capacity is either on demand or reserved through Provisioned Throughput.
Amazon Bedrock Custom Model Import	Brings a model customized elsewhere into Amazon Bedrock from Amazon S3, for a set of supported open-weight architectures, and serves it through the same invocation API.
Amazon Bedrock AgentCore	A set of modular services for running agents in production, including a serverless runtime, memory, a gateway that exposes existing APIs and functions as tools, identity, and observability.
Amazon Bedrock Knowledge Bases	Manages the retrieval side of a generative AI application, from ingesting documents into a vector store to answering a query with retrieved passages.
Amazon Elastic Container Service (Amazon ECS)	Runs containers as long-lived services or scheduled tasks, for inference an organization wants to operate itself rather than hand to managed hosting.
Amazon Elastic Kubernetes Service (Amazon EKS)	Managed Kubernetes, for teams whose deployment orchestration already lives in Kubernetes manifests and controllers.
AWS Lambda	Event-driven compute that suits lightweight inference and the glue around a deployment, such as invoking an endpoint or reacting to a pipeline event.
Amazon Elastic Compute Cloud (Amazon EC2)	Instances you configure and operate directly, which is the right answer when a workload needs control that managed hosting does not expose.

Knowledge Check

A machine learning engineer must serve a model to an internal application that sends a handful of requests in short bursts, with long idle stretches between them, and the team wants no instances to manage. Which hosting option fits that description? (Choose 1)

A. A serverless endpoint in Amazon SageMaker AI
B. A batch transform job in Amazon SageMaker AI
C. A container service running the model as a long-lived service
D. An imported model in Amazon Bedrock

Answer & Explanation

Correct Answer: A. A serverless endpoint in Amazon SageMaker AI

Explanation:
Amazon SageMaker Serverless Inference is specifically designed for workloads with intermittent or bursty traffic patterns and long idle periods. It automatically provisions, scales, and manages compute capacity based on incoming traffic, scaling down to zero instances during idle stretches so you only pay for the precise processing duration and data transferred. It eliminates infrastructure management and avoids paying for idle compute instances.

Why the other options are incorrect: * B. A batch transform job in Amazon SageMaker AI: Batch transform is designed for offline processing of large, pre-assembled static datasets rather than responding to dynamic, real-time application requests as they arrive. * C. A container service running the model as a long-lived service: Long-lived container services (or standard real-time SageMaker endpoints) keep instances running continuously, incurring infrastructure costs during long idle stretches and requiring instance/cluster scaling management. * D. An imported model in Amazon Bedrock: Bedrock Custom Model Import is specifically for importing supported open-weight Foundation Model architectures into Bedrock, rather than general serverless hosting for custom traditional ML or deep learning models.

AWS skills overview

In this lesson, you review the skills behind Task 3.1 as a light refresher. Each task groups a handful of related skills into a few topics. Read through them and notice which ones already feel familiar, and which ones feel shaky enough that you want to come back to them before exam day.

Choosing compute environments, targets, and orchestrators

In practice, this group of skills lets the machine learning engineer place a model on the smallest amount of infrastructure that satisfies the requirement, and decide how many models share it.

Be comfortable doing the following:

Selecting a compute environment and a deployment target, and separating the cases that managed hosting handles from the cases that need a container service, a Kubernetes cluster, or instances you operate.
Selecting a deployment orchestrator, which in practice means recognizing whether the organization's existing control plane is a managed pipeline service, a container scheduler, or Kubernetes.
Choosing among a single-model endpoint, a multi-model endpoint, and a multi-container endpoint, and knowing what each one assumes about the models sharing it.

The endpoint packing decision is the one worth rehearsing. A multi-model endpoint hosts many models on one fleet behind one shared serving container, so it assumes the models use the same framework, and it loads and unloads them from memory as traffic arrives. Infrequently used models therefore pay an occasional load penalty on first invocation. A multi-container endpoint is the opposite assumption: the containers differ, and they either run in sequence as an inference pipeline or are invoked directly by name. A model with much higher throughput or a much tighter latency requirement than its neighbors belongs on its own endpoint, because sharing a fleet means sharing its limits.

Knowledge Check

A team hosts many small models that share one framework, and traffic is uneven, with a few models called constantly and the rest called rarely. Which endpoint shape suits that pattern? (Choose 1)

A. A multi-model endpoint, accepting a load penalty on rarely used models
B. A multi-container endpoint with one container per model
C. A separate single-model endpoint for each model
D. One batch transform job per model, run on a schedule

Answer & Explanation

Correct Answer: A. A multi-model endpoint, accepting a load penalty on rarely used models

Explanation:
Amazon SageMaker Multi-Model Endpoints (MMEs) enable hosting hundreds or thousands of models on a single endpoint using a shared compute fleet and serving container. SageMaker dynamically manages model caching in memory: frequently invoked models stay loaded for fast responses, while rarely invoked models are dynamically loaded from Amazon S3 into memory when requested. This causes an occasional "cold start" load penalty for infrequently used models, but drastically reduces hosting costs and infrastructure overhead.

Why the other options are incorrect: * B. A multi-container endpoint with one container per model: Multi-container endpoints (MCEs) allow running up to 15 different ML containers (often using different frameworks or serving stacks) on a single endpoint, rather than scaling large numbers of small models sharing a single framework. * C. A separate single-model endpoint for each model: Provisioning dedicated endpoints/instances for many small, rarely called models leads to low endpoint utilization and extremely high idle infrastructure costs. * D. One batch transform job per model, run on a schedule: Scheduled batch jobs process static batch datasets offline, which cannot serve dynamic, real-time application traffic.

Selecting an inference strategy

In practice, this group of skills lets the machine learning engineer read a requirement and name the inference option it describes, without reaching for the option that is most familiar.

Be comfortable doing the following:

Matching a real-time endpoint to interactive traffic that a caller is blocked on, where the latency budget is strict and traffic is steady enough to keep capacity busy.
Matching a serverless endpoint to traffic with genuine idle periods, and recognizing that a request arriving after an idle period can meet a cold start unless provisioned concurrency is configured to keep capacity warm.
Matching asynchronous inference to requests that are queued rather than answered inline, which suits larger payloads and longer per-request processing while nobody waits on the connection.
Matching batch transform to a dataset that is already assembled, where compute starts, processes the input from object storage, writes the results back, and shuts down.

Two details separate the options once the obvious sorting is done. First, capacity that can fall to zero behaves differently per option: an asynchronous endpoint can scale its instance count to zero and let the queue hold arriving work, while a real-time endpoint that has scaled in to zero cannot answer an invocation until at least one instance is in service, and attempts during that provisioning window return an error. Scale-in to zero on a real-time endpoint is also not a single setting: it requires inference components plus a policy that scales back out on the metric the service emits when there is no capacity to serve a request. Second, the hosting option constrains the rest of the design. Serverless endpoints, for example, do not offer accelerated instances, VPC configuration, or multi-model hosting, so a requirement that names any of those has already ruled serverless out.

Knowledge Check

Single requests arrive through the day, each one takes minutes of processing, and no user is blocked waiting on the response. Which inference option matches that requirement? (Choose 1)

A. A real-time endpoint
B. Batch transform
C. Asynchronous inference
D. A serverless endpoint

Answer & Explanation

Correct Answer: C. Asynchronous inference

Explanation:
Amazon SageMaker Asynchronous Inference is designed for workloads that require processing individual requests with large payloads (up to 1 GB) or long processing times (up to 15 minutes to 1 hour). Incoming requests are automatically placed in an internal managed queue. The API immediately returns an acknowledgment (202 Accepted) to the caller without blocking, processes the request asynchronously from the queue, and outputs the result to Amazon S3 (with optional Amazon SNS notification). Additionally, it supports scaling the instance count down to zero during idle periods.

Why the other options are incorrect: * A. A real-time endpoint: Real-time endpoints are synchronous and enforce a 60-second processing timeout, making them unsuitable for tasks taking several minutes. Callers remain blocked waiting for a response. * B. Batch transform: Batch transform processes pre-assembled datasets in bulk offline. It is not designed to accept and queue individual dynamic requests arriving on demand throughout the day. * D. A serverless endpoint: Serverless inference is synchronous and capped at a 60-second execution processing limit. Processing tasks running for minutes will time out.

Deploying and hosting foundation models

In practice, this group of skills lets the machine learning engineer serve a foundation model, including one that was customized somewhere else, and allocate resources to it deliberately.

Be comfortable doing the following:

Evaluating foundation model deployment options, and separating a managed API that serves the model for you from an endpoint you size and operate yourself.
Deploying a model built outside AWS, either by importing supported architectures into the managed API or by packaging the artifacts and hosting them on a managed endpoint.
Configuring foundation model deployment, hosting, and resource allocation, which means committing to a throughput posture rather than only to an instance count.

The throughput posture is where most of the design sits. A managed foundation model API removes instance sizing, but it replaces it with a capacity decision: invoke on demand, or reserve model units through Provisioned Throughput so a set level of input and output capacity is held for you. A model customized inside the managed service has to be served through reserved capacity, while a model brought in through Custom Model Import is invoked on demand, which is a real difference in operating posture between two paths that both end in a custom model. Self-hosting inverts the tradeoff: you choose the accelerator, the instance count, and the scaling policy, and you inherit the work of keeping them right. Import also carries its own gate, since only certain model architectures and file layouts are accepted, so confirming that before planning the migration saves a rebuild.

Knowledge Check

An organization customized an open-weight model in its own training environment and now wants to serve it through the managed foundation model API without operating an endpoint. Which capability supports that? (Choose 1)

A. Custom Model Import in Amazon Bedrock
B. A multi-model endpoint in Amazon SageMaker AI
C. A batch transform job in Amazon SageMaker AI
D. Provisioned Throughput for a base model

Answer & Explanation

Correct Answer: A. Custom Model Import in Amazon Bedrock

Explanation:
Amazon Bedrock Custom Model Import enables organizations to import open-weight foundation models (e.g., Llama, Mistral) that were customized or fine-tuned externally (e.g., on Amazon EC2, SageMaker, or on-premises) into Amazon Bedrock. Once imported via Amazon S3, the custom model is hosted in Bedrock's serverless infrastructure and served seamlessly through the standard Amazon Bedrock InvokeModel or Converse APIs—eliminating the need to deploy, configure, or scale SageMaker endpoints directly.

Why the other options are incorrect: * B. A multi-model endpoint in Amazon SageMaker AI: Multi-model endpoints (MMEs) require hosting and managing a SageMaker endpoint instance fleet, which violates the constraint of serving the model through a managed API without operating an endpoint. * C. A batch transform job in Amazon SageMaker AI: Batch transform is designed for offline, asynchronous dataset processing jobs rather than real-time managed foundation model API inference. * D. Provisioned Throughput for a base model: Provisioned Throughput reserves dedicated model processing capacity for base models or models customized directly inside Bedrock, but does not handle the mechanism of importing external custom weights from an outside environment.

Configuring agents and retrieval

In practice, this group of skills lets the machine learning engineer stand up the two application patterns that sit on top of a foundation model, and tune the retrieval that feeds them.

Be comfortable doing the following:

Deploying and configuring an agent for a specific task, including the tools it may call and the identity it uses when it calls them.
Recognizing how an agent reaches other services and tools, including gateways that present existing APIs and functions as tools through an open agent protocol such as the Model Context Protocol.
Applying retrieval configurations to a knowledge base, including how many passages come back, whether the search is semantic or hybrid, which metadata filters narrow it, and whether a reranking model reorders what returns.

Retrieval configuration deserves the closer look, because the levers are independent and they fix different complaints. Raising the number of returned passages widens recall and gives the model more to work with, at the cost of more unrelated text in the prompt. Hybrid search adds a keyword match alongside the vector match, which recovers exact terms that embeddings blur, and it depends on the vector store carrying a filterable text field, so it is not available everywhere. Metadata filters cut the candidate set before similarity is considered at all, which is the right tool when the complaint is that stale or out-of-scope documents keep surfacing. Reranking is a second pass that reorders the candidates the vector search already found, so it improves ordering rather than recall, and it cannot rescue a passage the first pass never retrieved. Query decomposition breaks a compound question into sub-questions so each one retrieves against its own topic.

Knowledge Check

Retrieval returns the right passage, but it arrives well down the list while weaker passages rank above it. Which configuration change addresses that ordering problem directly? (Choose 1)

A. Add a reranking model to reorder the retrieved candidates
B. Reduce the number of returned passages
C. Switch the vector store to a different service
D. Re-embed the documents with more dimensions

Answer & Explanation

Correct Answer: A. Add a reranking model to reorder the retrieved candidates

Explanation:
Two-stage retrieval architectures balance search speed and precision: 1. Stage 1 (Bi-Encoder Vector Search): Quickly retrieves a broad list of \(K\) candidate passages (e.g., top 20–50) from the vector database based on embedding similarity. 2. Stage 2 (Reranker Model): Evaluates the query alongside each retrieved candidate passage using a cross-encoder model (such as a Bedrock or Cohere reranking model) to calculate a deep semantic relevance score.

Adding a reranking model directly fixes the ordering problem by re-scoring candidates and moving the most contextually relevant passage to the top of the context window before passing it to the generator LLM.

Why the other options are incorrect: * B. Reduce the number of returned passages: If the correct passage is currently ranked low (e.g., rank 12 out of 20), lowering top_k (e.g., to 5) completely drops the correct passage from the retrieved set. * C. Switch the vector store to a different service: The vector database vendor (e.g., OpenSearch, Pinecone, pgvector) executes nearest-neighbor searches based on distance metrics; changing stores does not change the inherent mathematical limitations of single-pass vector embeddings. * D. Re-embed the documents with more dimensions: While higher dimensions can slightly increase embedding expressiveness, bi-encoders still struggle with complex query-document alignment compared to a dedicated cross-encoder reranker. Furthermore, re-embedding requires a costly full re-indexing of the knowledge base.

Choosing Inference Infrastructure

Original video transcript

Inference infrastructure looks like a vocabulary list 4 options, 4 definitions, and people memorize them in an afternoon, right? And the definitions are the easy part. The hard part is that real workloads never announce which option they are, so you have to read the requirements for the signal. Exactly. So let's read through some requirements together. All right, I'll give you a scenario. You choose the infrastructure, and we'll question every choice. You're an ML engineer at a payments company deploying a fraud scoring model. Every transaction gets scored while the customer waits, so responses must return in milliseconds. Traffic spikes during shopping events. Availability has to hold, and the budget still matters. So where do you start? Well, I'd start with a question that decides everything, which is who's waiting for the answer. Here the customer is waiting mid-payment. So this is a real-time endpoint in Amazon Sagemaker AI. OK, so let's put a pin in that. Who's waiting for the answer, because we'll ask about it again later. Now, what about those traffic spikes? Well, auto scaling handles them. The endpoint watches an invocation metric and adds instances as traffic climbs, and then releases them as it falls. That protects latency as traffic climbs without paying for peak capacity all day. OK, here's where I want to push. Spiky traffic is exactly what serverless inference advertises. Why not deploy this fraud model serverless and stop managing instances at all? Because serverless fits intermittent traffic with real idle gaps between requests. A request arriving after an idle period can hit a cold start, and payment can't absorb that kind of delay. Fraud scoring has constant traffic and a strict latency budget, so it stays on a real-time endpoint. OK, so the split here is clear. Steady traffic with a strict latency budget goes real-time, and traffic with real idle gaps goes serverless unless you pay to keep capacity warm. Exactly, and cost still gets managed through right-sized instances and auto scaling. An end point can even scale in to zero. Requests then fail until it provisions an instance again, so a payment can't absorb it. Nice. So now let's change companies and ask the question again. You're at an insurance company. Every night, millions of scanned claim documents need processing through a document model. The results are due by morning. The files arrive together in bulk, and the company wants to pay only while the work runs. So start the same way. Sure, uh, I ask first, who's waiting for the answer. In this case, nobody. The claims arrive overnight and the results are due by morning. So real time is the wrong tool entirely for this. So which option would you go with? The whole data set arrives on a schedule. So this is a batch transform. It starts the compute, processes millions of documents in bulk, writes the results to Amazon S3, and releases everything. There's no endpoint sitting idle between nights, which is exactly the payment model they asked for. OK, but let me complicate it even further because real workloads are messy. OK. Suppose some claims arrive during the day, one at a time, and each one takes minutes to process. Batch waits for the nightly run and a real-time request would time out. What then? Well, that middle case is asynchronous inference. Requests go into a queue. The endpoint works through them, and it accepts payloads far larger than a real-time request allows. Nobody's blocked waiting, and the endpoint can scale down to zero when the queue's empty. So now all four options have a job. Whole data set on a schedule is batch, and queued single requests are asynchronous, right? And noticed that that one question sorted all of it. Who's waiting? How fast do they need it, and how does the work arrive? The service names follow from those answers. All right, so let's review what changed between the two companies. The deployment skills stayed identical, but every infrastructure choice swapped. The real-time endpoint became batch transform. Standing capacity became compute that starts and stops, and the latency budget became a completion deadline. Each swap traces back to who was waiting, and that's the reasoning that matters because the requirements signal the infrastructure, and the wording of a requirement is the whole clue. So here's a useful self-check. If every one of those swaps felt predictable, you're well prepared for these infrastructure decisions. And if one of them surprised you, especially the serverless call and the fraud model, that swap is your gap. Go back and understand why the choice moved, and then keep studying.

Selecting the correct Amazon SageMaker AI inference option relies on analyzing three core workload signals: who is waiting for the answer, how fast they need it, and how the data arrives.

Inference Strategy Decision Matrix

Inference Option	Workload Pattern	Primary Drivers & Constraints	Scaling & Cost Behavior
Real-Time Endpoint	Constant or spiky traffic requiring millisecond latencies.	Live user/process actively blocked waiting for a response (e.g., mid-payment fraud scoring).	Uses Auto Scaling; maintains warm capacity to prevent cold-start delays.
Serverless Inference	Intermittent traffic with significant, extended idle gaps.	Low/unpredictable volume that can tolerate occasional cold starts.	Scales automatically to zero; charges strictly per execution time and payload size.
Batch Transform	Pre-assembled datasets arriving in bulk on a fixed schedule.	No live caller; job completion bounded by a deadline (e.g., overnight claims processing).	Spins up compute cluster on demand, processes data, outputs to S3, and terminates.
Asynchronous Inference	Large payloads (> 6 MB) or long processing times (up to 15–60 mins).	Requests arrive dynamically; callers are unblocked and receive a queued response.	Managed SQS queue handles requests; endpoint can scale down to zero when idle.

Scenario Breakdown

Scenario 1: Payment Fraud Scoring

Requirements: High availability, millisecond response budget, spiky traffic during shopping events.
Selection: Real-Time Endpoint with Auto Scaling
Key Distinctions:
Why not Serverless? Serverless inference handles spiky traffic well, but requests arriving after an idle stretch encounter cold starts. A payment pipeline cannot absorb cold-start latency.
Cost Control: Handled via right-sized instances and metric-based Auto Scaling rather than scaling to zero.

Scenario 2: Overnight Bulk Document Processing

Requirements: Millions of scanned files arrive together in bulk each night; results are due by morning.
Selection: Batch Transform
Key Distinctions:
Why not Real-Time/Serverless? There is no endpoint needed to listen for dynamic incoming requests.
Cost Control: Compute provisions automatically at the scheduled run time, writes predictions directly to Amazon S3, and releases all resources immediately upon completion.

Scenario 3: Long-Running Daytime Document Processing

Requirements: Individual documents arrive one at a time during the day, taking several minutes each to process.
Selection: Asynchronous Inference
Key Distinctions:
Why not Real-Time? Synchronous real-time endpoints enforce strict timeout limits (60 seconds) and would time out on multi-minute tasks.
Why not Batch? Batch transform expects a pre-assembled, bulk dataset on a schedule rather than individual requests arriving dynamically.
Mechanism: Requests enter a managed queue, return an immediate acknowledgment, process asynchronously, and scale compute to zero when the queue clears.

Core Architectural Rules

Who is waiting? If a user or downstream microservice is blocked waiting mid-transaction, use Real-Time. If no one is waiting, use Batch or Asynchronous.
Real-Time vs. Serverless: If traffic is spiky but continuous, choose Real-Time with Auto Scaling. Reserve Serverless for true intermittent traffic with extended idle gaps where cold starts are acceptable.
Batch vs. Asynchronous: Use Batch Transform for static datasets arriving all at once. Use Asynchronous Inference for single, incoming requests that take minutes to process or exceed real-time payload limits.

Quick recall

Without rewatching, write the one question the conversation keeps returning to: who is waiting for the answer. Then write the four pairings it lands on: steady traffic to real-time, idle gaps to serverless, a whole dataset to batch, and queued requests to asynchronous.

Find your gap

The video ends on a self-check. If every swap between the two companies felt predictable as it went, you are well prepared for these infrastructure decisions. If one surprised you, especially the serverless call on the fraud model, that swap is your gap. If necessary, return to the skills lesson above and work out why the choice moved.

Knowledge Check

AnyCompany scores every payment while the customer waits, traffic climbs sharply during promotions, and the latency budget is strict. Which deployment approach BEST fits that requirement? (Choose 1)

A. A serverless endpoint, so no instances have to be managed
B. A real-time endpoint with an auto scaling policy on an invocation metric
C. An asynchronous endpoint, so requests queue during promotions
D. A batch transform job triggered every few minutes

Answer & Explanation

Correct Answer: B. A real-time endpoint with an auto scaling policy on an invocation metric

Explanation:
This scenario features two critical constraints: 1. Low-Latency Synchronous Requirements: Payments are scored in real time while the customer waits mid-transaction, imposing a strict latency budget. 2. Traffic Volatility: Promotion events drive sharp traffic spikes.

A SageMaker Real-Time Endpoint paired with Application Auto Scaling (tracking an invocation metric like SageMakerVariantInvocationsPerInstance) ensures warm compute instances process incoming requests immediately without cold starts. When promotional traffic surges, Auto Scaling automatically provisions additional instances to maintain throughput and latency SLAs, then scales back in when traffic subsides.

Why the other options are incorrect: * A. A serverless endpoint, so no instances have to be managed: Serverless Inference scales to zero during idle periods. However, when traffic arrives after an idle period, it introduces cold-start latency (typically 1–5+ seconds), which violates a strict latency budget for payment authorization. * C. An asynchronous endpoint, so requests queue during promotions: Asynchronous endpoints queue incoming requests for non-blocking background processing. Because the customer is actively waiting for payment confirmation, queuing requests introduces unacceptable delays. * D. A batch transform job triggered every few minutes: Batch Transform processes pre-assembled datasets offline in bulk on a schedule. It cannot process individual real-time inference requests as they arrive.