Task 3.2 - Provision and configure resources for ML and AI workloads

Overview

Task 3.2, Provision and configure resources for ML and AI workloads based on existing architecture and requirements, is where a hosting decision becomes real resources.

It covers the capacity posture, the container the model runs in, the network the endpoint sits inside, the scaling signal that adds and removes capacity, and the stores and runtimes that retrieval and agents depend on. The phrase that carries the most weight is existing architecture, because these choices are constrained by what the organization already runs.

As you review these lessons for Task 3.2, check that you understand how to do the following.

Optimize resource provisioning between on-demand and provisioned resources for performance and cost efficiency.
Automate compute resource provisioning with integrated communication between stacks and orchestration services.
Build and maintain containers for ML and AI workloads.
Configure SageMaker AI endpoints within VPC network environments.
Deploy and host models programmatically (for example, by using the SageMaker AI SDK for Python, AWS CLI, Boto3).
Select specific metrics for auto scaling implementations.
Create and manage Amazon Bedrock knowledge bases with vector database configurations, document indexing, and retrieval - optimization.
Implement retrieval pipelines to meet business needs.
Implement agent state management systems.
Implement AI-specific resource scaling for GPU workloads.
Deploy agentic workflow infrastructure.

AWS services overview

The services in this task fall into four jobs: hosting and scaling compute, describing that compute as code, placing it inside a private network, and standing up the stores and runtimes that generative AI applications need. Recognize which job each one does before you try to remember its options. Review each service to learn more.

AWS Service	Description
Amazon SageMaker AI	Hosts the model and holds the configuration that matters here: the container image, the instance family and count, and the network settings the model runs under.
Amazon SageMaker HyperPod	Provisions resilient accelerator clusters, orchestrated with either a workload manager or Kubernetes, and replaces faulty hardware automatically so a long training run survives it.
Amazon Elastic Container Registry (Amazon ECR)	Stores and versions the container images a training job or an endpoint pulls, which is where a custom inference container lives.
AWS CloudFormation	Declares resources in a template so the same environment can be created repeatedly, and lets one stack publish an output that another stack imports.
AWS Cloud Development Kit (AWS CDK)	Defines the same infrastructure in a general-purpose programming language and synthesizes it into a template.
Application Auto Scaling	Registers an endpoint variant or an inference component as a scalable target and applies the scaling policy that adds and removes capacity.
Amazon Virtual Private Cloud (Amazon VPC)	Supplies the subnets, security groups, and private endpoints that let a model reach storage and other services without traversing the internet.
Amazon Bedrock Knowledge Bases	Owns the ingestion and retrieval pipeline, including which vector store holds the embeddings and how a query is answered from them.
Amazon OpenSearch Service	A vector store option for a knowledge base, available as a serverless collection or as a managed cluster, and the one that supports hybrid search over a filterable text field.
Amazon Bedrock AgentCore	Supplies the runtime, memory, gateway, and identity an agent needs in production, so agent state and tool access are configured rather than hand-built.

Knowledge Check

Which service registers a hosted model as a scalable target and applies the policy that adds and removes capacity as traffic changes? (Choose 1)

A. Application Auto Scaling
B. AWS CloudFormation
C. Amazon Elastic Container Registry (Amazon ECR)
D. Amazon Virtual Private Cloud (Amazon VPC)

Answer & Explanation

Correct Answer: A. Application Auto Scaling

Explanation:
Application Auto Scaling is the dedicated AWS service that manages scaling for resources beyond standard EC2 Auto Scaling groups. For Amazon SageMaker AI endpoints, Application Auto Scaling registers the endpoint's model variant (or inference component) as a scalable target and dynamically adjusts instance counts (or provisioned concurrency) based on user-defined policies, such as target tracking on SageMakerVariantInvocationsPerInstance or GPU utilization metrics.

Why the other options are incorrect: * B. AWS CloudFormation: CloudFormation is an Infrastructure as Code (IaC) service used to declare and provision AWS resources via templates; it does not monitor traffic or dynamically scale runtime compute capacity. * C. Amazon Elastic Container Registry (Amazon ECR): ECR stores, manages, and versions Docker container images, providing the inference images that SageMaker endpoints pull during initialization. * D. Amazon Virtual Private Cloud (Amazon VPC): VPC manages networking isolation, subnets, routing, and security groups for endpoint compute instances, but does not handle scaling logic.

Review AWS Skills

In this lesson, you review the skills behind Task 3.2 as a light refresher. Each task groups a handful of related skills into a few topics. Read through them and notice which ones already feel familiar, and which ones feel shaky enough that you want to come back to them before exam day.

Provisioning capacity and choosing scaling signals

In practice, this group of skills lets the machine learning engineer decide how much capacity to hold, how much to add on demand, and which signal to scale on.

Be comfortable doing the following:

Weighing on-demand capacity against reserved or provisioned capacity, and tying the choice to how predictable the traffic is rather than to a preference.
Selecting a scaling metric, and separating a predefined invocation metric from a concurrency metric and from a custom resource-utilization metric.
Scaling accelerated workloads, including provisioning resilient accelerator clusters for training and choosing accelerator-backed hosting for inference.

The metric choice is a real discriminator, because the metric decides how fast the policy can react. A predefined target-tracking policy on average invocations per instance is the general-purpose answer, and the underlying metric publishes at standard resolution, so the policy responds on that cadence. High-resolution concurrency metrics publish far more often, which lets a policy scale out much sooner when concurrent requests climb, and that matters for a model whose response streams back over a long window rather than completing in one short call. A custom metric such as processor utilization is the fallback for a workload whose cost driver is not request count, and it needs testing before production because nothing validates that the metric moves in proportion to capacity. Cooldown settings sit on top of all three, and they exist so a burst does not trigger a scale-out that immediately reverses.

Knowledge Check

Concurrent requests to a hosted model climb in seconds, and each response streams back over a long window, so a policy that reacts on a per-minute metric adds capacity too late. Which scaling choice addresses that? (Choose 1)

A. Target tracking on a high-resolution concurrency metric
B. A longer scale-out cooldown on the existing policy
C. A larger target value on the existing invocation metric
D. A scheduled scaling action each morning

Answer & Explanation

Correct Answer: A. Target tracking on a high-resolution concurrency metric

Explanation:
For workloads with rapid traffic surges (e.g., streaming generative AI responses or long-running inferences), standard 1-minute aggregation metrics like InvocationsPerInstance react too slowly.

Amazon SageMaker supports high-resolution CloudWatch concurrency metrics (such as ConcurrentRequestsPerModel or ConcurrentRequestsPerCopy) emitted at sub-minute intervals (10 seconds). Pairing target tracking with a high-resolution concurrency metric allows Application Auto Scaling to detect in-flight request surges almost instantaneously and trigger scale-out events in seconds rather than minutes.

Why the other options are incorrect: * B. A longer scale-out cooldown on the existing policy: Increasing the scale-out cooldown period enforces a longer delay before additional scaling actions can occur, making the auto scaler even slower to respond to traffic spikes. * C. A larger target value on the existing invocation metric: Increasing the target threshold forces each instance to handle more traffic before triggering a scale-out, which delays scaling even further and risks overwhelming the endpoint. * D. A scheduled scaling action each morning: Scheduled scaling works for predictable, recurring calendar events (e.g., daily 9:00 AM office traffic), but cannot dynamically react to unexpected or sudden traffic spikes during the day.

Automating provisioning and deploying programmatically

In practice, this group of skills lets the machine learning engineer create the same environment twice without drift, and deploy a model from code rather than from a console.

Be comfortable doing the following:

Declaring compute and hosting resources as code, and connecting separately owned stacks so a network stack and a hosting stack stay linked instead of duplicating identifiers.
Coordinating provisioning with orchestration services, so a stack deployment, a training job, and an endpoint update run in a defined order with defined failure handling.
Deploying and hosting models programmatically through a language SDK, the command line interface, or the lower-level client library, and knowing which one a given task calls for.

Cross-stack linkage is the detail that keeps automation honest. A template can flag an output for export and another template can import that value, so the subnet identifiers and security group a hosting stack needs come from the network stack that owns them. Restricting the coupling to exported outputs is deliberate: it keeps one team from depending on the internals of another team's stack, and it means a rename inside the network stack does not silently break hosting. The programmatic surface is layered the same way. A high-level SDK builder assembles a model, an endpoint configuration, and an endpoint in a few lines, the lower-level client exposes each API call individually when you need control over one field, and a template is the right tool when the environment itself has to be reproducible rather than just the deployment.

Knowledge Check

A networking team owns the subnets and security groups, and a modeling team owns the endpoint stack. What keeps the endpoint stack using the networking team's values without copying them? (Choose 1)

A. A parameter in the endpoint stack holding the copied identifiers
B. An exported stack output that the endpoint stack imports
C. A scaling policy attached to the endpoint
D. A container image tag shared by both teams

Answer & Explanation

Correct Answer: B. An exported stack output that the endpoint stack imports

Explanation:
In AWS CloudFormation, cross-stack references allow one stack to reuse resources defined by another stack. The networking stack exports its resource values (such as Subnet IDs and Security Group IDs) using the Export field in its Outputs block. The modeling stack can then dynamically reference those values using Fn::ImportValue (or CDK stack imports). This decouples the infrastructure layers while ensuring the modeling stack automatically references the correct networking resources without manual copying or hardcoding.

Why the other options are incorrect: * A. A parameter in the endpoint stack holding the copied identifiers: Manually copying identifiers into stack parameters creates operational overhead and risks configuration drift if the networking team updates or recreates resources. * C. A scaling policy attached to the endpoint: Scaling policies govern Application Auto Scaling instance capacity and have no role in passing network configuration across stacks. * D. A container image tag shared by both teams: Container image tags in Amazon ECR identify container software versions, not network subnets or security groups.

Building containers and configuring endpoint networking

In practice, this group of skills lets the machine learning engineer ship a container that the hosting service can run, and place the resulting endpoint inside the network the organization requires.

Be comfortable doing the following:

Choosing between a prebuilt framework container, an extended version of one, and a container built from scratch, and recognizing what the hosting service expects a serving container to implement.
Maintaining those images over time, which means versioning them in a registry so a redeploy pulls a known image rather than a moving tag.
Configuring an endpoint inside a private network by supplying subnets and security groups, and then supplying the private paths the container needs to reach storage and other services.

Network configuration is where a deployment most often stalls, and the failure is usually a missing path rather than a missing permission. When a model is attached to subnets that have no route to the internet, it cannot reach object storage to pull its own artifacts until a private path to that service exists. Other AWS services it calls need either their own private interface endpoints or a network address translation gateway for outbound access. Subnets in more than one Availability Zone are required even for a single hosting instance, and the execution role needs permission to create and attach the network interfaces the service places into those subnets. The constraint worth remembering is that this configuration is not available on every hosting option: serverless endpoints do not take a network configuration at all, so a requirement for private networking decides the inference option before anything else does.

Knowledge Check

An endpoint is configured with private subnets that have no internet route, and it fails while starting because it cannot reach the bucket holding its model artifacts. What resolves that? (Choose 1)

A. Move the endpoint to a serverless configuration
B. Add a private path from the subnets to the object storage service
C. Attach a larger instance type to the endpoint
D. Register the endpoint as a scalable target

Answer & Explanation

Correct Answer: B. Add a private path from the subnets to the object storage service

Explanation:
When a SageMaker AI endpoint is deployed in private subnets with VPC network isolation enabled (no Internet Gateway or NAT Gateway route), it cannot resolve or reach public AWS service endpoints. To allow the endpoint compute instances to download model artifacts from Amazon S3 (the object storage service) without public internet access, you must create an Amazon S3 VPC Endpoint (Gateway Endpoint) in the VPC and attach it to the private subnet's route table.

Why the other options are incorrect: * A. Move the endpoint to a serverless configuration: Serverless endpoints configured inside isolated VPC subnets face the exact same VPC networking and S3 access constraints. * C. Attach a larger instance type to the endpoint: Instance hardware capacity (vCPU/RAM) does not resolve underlying VPC route table network isolation issues. * D. Register the endpoint as a scalable target: Registering a scalable target connects the endpoint to Application Auto Scaling, which has no effect on network connectivity or model artifact retrieval.

Configuring knowledge bases and retrieval pipelines

In practice, this group of skills lets the machine learning engineer stand up the retrieval half of a generative AI application and tune it against a business requirement rather than a default.

Be comfortable doing the following:

Creating and managing a knowledge base, including selecting the vector store, matching the index dimensions to the embedding model, and mapping the fields that hold the vector, the text chunk, and the metadata.
Indexing documents through an ingestion job, and recognizing that a resync processes only what changed since the last one.
Implementing a retrieval pipeline that meets a stated need, which means combining the search type, the result count, filters, and reranking rather than picking one of them.

The vector store choice is the decision with the longest reach, because it sets what the retrieval pipeline can do later. A search-service collection or managed cluster supports the widest set of retrieval behavior, including hybrid search, which needs a filterable text field alongside the vectors. Object storage for vectors is the low-cost option for large vector sets that are queried less often. A relational option keeps embeddings beside the records they describe, and a graph store supports retrieval that follows relationships between entities. Two mechanical details cause most rebuilds: the index dimensions have to match the embedding model, and the field mapping has to be declared before ingestion, because neither can be changed afterward without reindexing. Metadata is what turns retrieval into a pipeline rather than a lookup, since filters can narrow the candidate set to a subset before similarity is measured at all.

Knowledge Check

A retrieval application must return exact part numbers as well as semantically similar text, and the team needs a vector store that supports both in one query. Which requirement follows from that? (Choose 1)

A. The embedding model must produce binary vectors
B. The vector store must support hybrid search over a filterable text field
C. The knowledge base must use a graph store
D. The ingestion job must run on a fixed schedule

Answer & Explanation

Correct Answer: B. The vector store must support hybrid search over a filterable text field

Explanation:
Dense vector embeddings excel at capturing semantic intent, but they often struggle with exact keyword precision for alphanumeric identifiers like part numbers, product SKUs, or error codes. Hybrid search addresses this by combining dense vector search (k-NN for semantic similarity) with sparse lexical search (like BM25 for exact text and keyword matching) in a single query execution. Vector stores like Amazon OpenSearch Service (used within Amazon Bedrock Knowledge Bases) support hybrid search alongside metadata filtering to deliver both precise string matches and semantically related context.

Why the other options are incorrect: * A. The embedding model must produce binary vectors: Binary vector quantization reduces embedding storage and speeds up distance calculations, but it does not enable exact string keyword matching for specific part numbers. * C. The knowledge base must use a graph store: Graph databases (e.g., Amazon Neptune) store entity relationships, which is a different architectural pattern than combining lexical keyword matching and vector similarity over unstructured/semi-structured text. * D. The ingestion job must run on a fixed schedule: Scheduling data ingestion governs how often the index is updated, but does not affect the query-time execution engine capabilities.

Managing agent state and agentic infrastructure

In practice, this group of skills lets the machine learning engineer run an agent as production infrastructure, with state that survives the right amount of time and tools it can reach safely.

Be comfortable doing the following:

Implementing agent state management, and separating memory that holds a single multi-turn conversation from memory that persists across sessions.
Deploying agentic workflow infrastructure, which means a runtime that isolates each session, an identity the agent uses when it calls anything, and observability that records each step it took.
Exposing existing services to an agent as tools rather than rebuilding them, through a gateway that presents APIs and functions in a protocol the agent already speaks.

The state distinction is the one to hold onto, because getting it wrong shows up as a behavior complaint rather than an error. Short-term memory is the conversation the agent is currently having, and it is what makes a follow-up question resolvable. Long-term memory persists across sessions and can be shared between agents, which is what lets a returning user avoid repeating context, and it is also what makes retention and privacy an explicit design decision instead of an accident. Session isolation belongs to the runtime rather than to the memory store: separate sessions get separate execution environments, so one conversation cannot read another's working state. Observability closes the loop, because an agent's failure is usually a tool call that returned something unexpected, and only a recorded trace of the steps shows which one.

Knowledge Check

An agent handles multi-turn conversations correctly within a session, but returning users must repeat context they supplied on a previous visit. Which capability addresses that? (Choose 1)

A. Long-term memory that persists across sessions
B. Stronger session isolation in the runtime
C. A gateway that exposes more tools to the agent
D. A larger result count in the retrieval configuration

Answer & Explanation

Correct Answer: A. Long-term memory that persists across sessions

Explanation:
In AI agent architectures (such as Amazon Bedrock AgentCore Memory), short-term memory retains conversation history and dialogue context exclusively within a single active session (sessionId). Once a session terminates or expires, that working context is cleared.

To allow returning users to preserve context across visits—such as saved user preferences, historical conversation summaries, or user facts—the agent requires long-term memory (LTM). Long-term memory automatically extracts and consolidates persistent insights scoped to a user/actor namespace (e.g., /preferences/{actorId}/), making that context available across distinct sessions.

Why the other options are incorrect: * B. Stronger session isolation in the runtime: Session isolation prevents execution environments and temporary state from leaking between tenants/users; it enforces security boundaries rather than persisting context across visits. * C. A gateway that exposes more tools to the agent: An API gateway exposes external functions and tools to the agent, but does not handle user context retention or cross-session memory management. * D. A larger result count in the retrieval configuration: Increasing retrieval top_k returns more candidate chunks from a vector store/Knowledge Base during a query, but does not store or carry forward past user interaction history across separate sessions.

Knowledge Check

A compliance requirement states that a hosted model must reach its artifacts and its downstream services without traffic leaving the private network. Which configuration satisfies that requirement? (Choose 1)

A. A serverless endpoint with provisioned concurrency
B. A real-time endpoint with a scaling policy on average invocations per instance
C. A batch transform job reading and writing the same bucket
D. A real-time endpoint configured with private subnets, security groups, and private endpoints for the services it calls

Answer & Explanation

Correct Answer: D. A real-time endpoint configured with private subnets, security groups, and private endpoints for the services it calls

Explanation:
To ensure network traffic never traverses the public internet, SageMaker AI endpoints must be deployed within an Amazon VPC using private subnets and restrictive Security Groups. To communicate privately with required AWS services—such as Amazon S3 (for model artifacts), Amazon ECR (for container images), or other downstream APIs—you must configure VPC Endpoints (AWS PrivateLink) for those services. This guarantees all API calls and data transfers remain entirely on the AWS private network backbone.

Why the other options are incorrect: * A. A serverless endpoint with provisioned concurrency: Provisioned concurrency eliminates cold starts for serverless endpoints, but does not inherently enforce VPC network isolation or private endpoint routing. * B. A real-time endpoint with a scaling policy on average invocations per instance: Scaling policies manage compute capacity based on traffic load, which is completely separate from VPC network topology and network security rules. * C. A batch transform job reading and writing the same bucket: While batch transform can run inside a VPC, this option describes an offline job rather than a hosted endpoint configuration and lacks the private endpoint configuration required for secure downstream service communication.