Task 4.2 - Optimize and manage ML and AI infrastructure costs and performance

Overview

Task 4.2, Optimize and manage ML and AI infrastructure costs and performance, is about keeping a working system affordable without making it slower or less reliable.

The levers differ by workload. Hosted inference is billed for the compute it holds, while foundation model inference on demand is billed for the tokens going in and coming out. So an instinct that moves one bill can do nothing at all to the other.

As you review these lessons for Task 4.2, check that you understand how to do the following.

Select inference instance families to optimize performance and cost.
Configure and use tools to troubleshoot and analyze resources (for example, Amazon CloudWatch, Amazon Bedrock AgentCore - Observability, AWS X-Ray).
Set up dashboards to monitor performance metrics.
Optimize capacity for cost, performance, and reliability.
Optimize costs and set cost quotas by using appropriate cost management tools.
Optimize infrastructure costs by selecting purchasing options.
Evaluate cost implications of using FMs for inference in production.
Monitor agent resource consumption patterns.
Manage FM inference costs with usage optimization.
Monitor AI-specific cost patterns (for example, token usage optimization, embedding computation costs, vector database storage optimization).

AWS services overview

AWS Service / Feature	Description
Amazon Bedrock	Serves foundation model inference. On-demand invocation is billed by token, and batch inference and provisioned throughput are the alternative capacity modes for work that is asynchronous or predictable.
Amazon SageMaker AI	Hosts inference on instance families that are selected per endpoint, scales endpoint capacity toward demand, and is the usage that SageMaker AI Savings Plans covers.
Amazon CloudWatch	Holds the utilization, invocation, and token metrics an optimization decision needs, along with the dashboards that display them and the alarms that react to them.
AWS X-Ray	Traces a request across the components that served it, which is how a latency problem is attributed to a component instead of being answered with more capacity everywhere.
Amazon Bedrock AgentCore Observability	Captures the inference calls, tool invocations, and memory operations inside an agent invocation, which is what makes an agent's resource consumption pattern visible per invocation.
AWS Compute Optimizer	Analyzes utilization history for the compute resource types it supports and recommends a configuration that matches the observed workload.
AWS Cost Explorer	Reports historical and forecast spend, and can group and filter it by tag, service, or usage type, which is how the driver behind a rising bill is identified.
AWS Budgets	Tracks spend or usage against an amount that was defined for it and sends an alert, so a threshold is noticed while there is still time to act on it.
AWS Trusted Advisor	Checks an account against AWS best practices across several categories, including cost checks that surface idle and underused resources.

Knowledge Check

A team needs to be alerted while a workload's spend is on track to pass an amount the team defined for the month. Which option is built for that? (Choose 1)

A. AWS Cost Explorer
B. AWS Budgets
C. AWS Trusted Advisor
D. AWS X-Ray

Answer & Explanation

Correct Answer: B. AWS Budgets

Explanation:
AWS Budgets allows you to set custom financial targets and usage thresholds for your AWS accounts and workloads. Crucially, it supports forecasted cost alerts—notifying team members via email or SNS topics as soon as spending is projected (on track) to exceed the defined monthly budget threshold before the overage actually occurs.

Why the other options are incorrect: * A. AWS Cost Explorer: AWS Cost Explorer visualizes, analyzes, and reports historical and forecasted spend data by tag, service, or usage type, but it does not generate automated threshold-driven alert notifications. * C. AWS Trusted Advisor: AWS Trusted Advisor inspects accounts against AWS best practices (including cost checks that surface idle or underused resources), but it does not monitor custom monthly spending caps or send forecasted spending alerts. * D. AWS X-Ray: AWS X-Ray is a distributed tracing service used to analyze performance bottlenecks, dependencies, and execution latency across microservices, having no role in cost management.

AWS skills overview

In this lesson, you review the skills behind Task 4.2 as a light refresher. Each task groups a handful of related skills into a few topics. Read through them and notice which ones already feel familiar, and which ones feel shaky enough that you want to come back to them before exam day.

Selecting inference compute and capacity

In practice, this group of skills lets the machine learning engineer match the compute behind inference to the traffic it actually serves.

Be comfortable doing the following:

Selecting an instance family for inference from the shape of the work, weighing accelerated compute against general - purpose compute, and sizing from measured utilization rather than from the size the endpoint launched with.
Letting capacity follow demand, so a workload with a daily peak and a quiet overnight period does not hold peak - capacity all night.
Choosing an inference option that suits the traffic pattern, including a real-time endpoint for steady interactive - traffic, an on-demand option for intermittent traffic, an asynchronous option for long-running or large requests, and a - shared endpoint for many small models.
Balancing cost against performance and reliability deliberately, and naming what was traded when capacity is reduced.

Two habits separate a real optimization from a hopeful one. The first is evidence: read utilization for the endpoint over a representative window before changing anything, because an endpoint that is busy for one hour a day and idle for the rest has a scaling problem rather than a sizing problem, and the two fixes are different. The second is naming the tradeoff out loud. Reducing capacity to the measured average removes the headroom that absorbed a spike, so the reliability requirement has to be restated rather than assumed. A shared endpoint hosting many models on one set of instances is a large saving when those models share a serving framework and are called intermittently, and it introduces load time for a model that is not resident when a request arrives.

Knowledge Check

An inference endpoint is busy during business hours and nearly idle overnight, and its instances are correctly sized for the daytime peak. Which change addresses the overnight cost? (Choose 1)

A. Scale endpoint capacity with demand so instances are released when traffic falls
B. Move the endpoint to a smaller instance family
C. Convert the workload to an asynchronous inference option
D. Host the model on a shared endpoint with unrelated models

Answer & Explanation

Correct Answer: A. Scale endpoint capacity with demand so instances are released when traffic falls

Explanation:
Configuring SageMaker Automatic Scaling (such as target tracking scaling policies based on InvocationsPerInstance or CPU/GPU utilization) allows the endpoint to dynamically scale instance count up and down. During peak business hours, it maintains enough capacity to handle incoming traffic, and as traffic drops overnight, it automatically scales down to release idle compute instances, reducing overnight hosting costs.

Why the other options are incorrect: * B. Move the endpoint to a smaller instance family: The current instances are already correctly sized for the daytime peak. Downsizing the instance family would lead to under-provisioning, causing high latency, throttling, or out-of-memory errors during business hours. * C. Convert the workload to an asynchronous inference option: Asynchronous inference is designed for large payload sizes (up to 1 GB) or long processing execution times (up to 15 minutes) with queued responses. Changing to an asynchronous model alters the client request/response paradigm unnecessarily when standard real-time auto-scaling resolves the diurnal pattern. * D. Host the model on a shared endpoint with unrelated models: Multi-model endpoints (MMEs) are designed to consolidate hundreds or thousands of infrequently invoked models onto a shared instance pool, but they introduce cold-start latency and complexity that are unsuited for a core, high-traffic daytime workload.

Instrumenting performance with dashboards and traces

In practice, this group of skills lets the machine learning engineer answer where the time and the money went, with a measure rather than an opinion.

Be comfortable doing the following:

Configuring the monitoring and tracing tools that troubleshoot a workload, and knowing which one answers which - question: an aggregate metric, a log line, or a trace across components.
Building a dashboard that carries the measures a decision needs, so latency, throughput, errors, utilization, and token - usage are read together instead of in four places.
Instrumenting an agent so its consumption is visible per invocation, and not only as a total for the account.

A dashboard earns its place when it shortens a decision, which means it holds the measures that move together. Utilization beside latency shows whether slowness is a capacity problem. Token usage beside invocation count shows whether a rising bill is more traffic or heavier prompts, and those two have different fixes. Traces then take over where aggregates stop: an average tells you the request was slow, and a trace tells you which component made it slow, which for an agent means separating model inference from a tool call, from retrieval, and from coordination between agents.

Knowledge Check

What type of data does Amazon CloudWatch collect from AWS resources? (Choose 1)

A. Metrics and logs
B. Source code repositories
C. Database schemas
D. Network packet captures

Answer & Explanation

Correct Answer: A. Metrics and logs

Explanation:
Amazon CloudWatch is the primary monitoring and observability service for AWS resources and applications. It natively collects, visualizes, and correlates operational telemetry in the form of metrics (time-series numerical measurements like CPU utilization, request latency, and token consumption) and logs (unstructured or structured text entries generated by applications, AWS services, and OS agents).

Why the other options are incorrect: * B. Source code repositories: Source code storage and version control are managed by services like AWS CodeCommit, GitHub, or GitLab. * C. Database schemas: Database schemas and tables are managed within database engines (e.g., Amazon RDS, Aurora, DynamoDB) or cataloged by data integration services like AWS Glue. * D. Network packet captures: Deep packet inspection and traffic capture are handled via VPC Traffic Mirroring or VPC Flow Logs rather than being natively collected as raw packet captures by CloudWatch.

Choosing purchasing options and cost management tools

In practice, this group of skills lets the machine learning engineer pay a lower rate for the usage that is already predictable, and see the rest before it becomes a surprise.

Be comfortable doing the following:

Selecting a purchasing option from the shape of the demand, committing where usage is steady and staying on demand - where it is variable or experimental.
Naming which commitment covers which service, rather than treating a commitment as a general discount.
Setting quotas and budgets so a workload has a ceiling, and tagging resources so the spend can be attributed to a team, - a model, or an environment.
Reading a cost report to find the driver behind a change, instead of applying a reduction everywhere.

Precision about coverage is the part that gets missed. SageMaker AI Savings Plans apply to Amazon SageMaker AI usage. Compute Savings Plans apply to usage across Amazon Elastic Compute Cloud (Amazon EC2), AWS Lambda, and AWS Fargate, and EC2 Instance Savings Plans apply to Amazon EC2 usage, so neither of those covers a SageMaker AI endpoint. Interruptible work such as a training job that can restart from a checkpoint is a fit for spare capacity at a reduced rate, and a serving baseline is not, because capacity that can be reclaimed is not a serving guarantee. Attribution is the quiet prerequisite for all of it: without cost allocation tags, a cost report shows a service total and no way to say which model or team produced it.

Knowledge Check

A steady, year-round inference baseline runs on Amazon SageMaker AI instances, and the team wants a commitment that reduces the rate charged for that usage. Which purchasing option applies? (Choose 1)

A. SageMaker AI Savings Plans
B. Compute Savings Plans
C. EC2 Instance Savings Plans
D. A budget with an alert on the account's monthly spend

Answer & Explanation

Correct Answer: A. SageMaker AI Savings Plans

Explanation:
SageMaker AI Savings Plans offer a flexible commitment-based pricing model designed specifically for Amazon SageMaker ML instance usage (including Real-Time Inference, Batch Transform, Training, and Notebooks). By committing to a consistent hourly spend ($/hour) over a 1-year or 3-year term, teams receive up to 64% savings on SageMaker instance rates compared to On-Demand pricing.

Why the other options are incorrect: * B. Compute Savings Plans: Compute Savings Plans apply automatically across Amazon EC2, AWS Fargate, and AWS Lambda, but they do not extend to SageMaker ML instance usage. * C. EC2 Instance Savings Plans: EC2 Instance Savings Plans apply exclusively to Amazon EC2 instances within a specific instance family and region, and cannot be applied to SageMaker endpoints (ml.* instance types). * D. A budget with an alert on the account's monthly spend: AWS Budgets tracks spend and sends alerts when thresholds are passed, but it is a financial tracking tool and provides no direct rate discounts.

Managing foundation model and agent inference costs

In practice, this group of skills lets the machine learning engineer control a bill that has no instances on it.

Be comfortable doing the following:

Evaluating what foundation model inference costs in production, starting from what the invocation is billed for rather - than from how much compute it appears to use.
Reducing that cost through the levers that exist: selecting a model no larger than the task requires, shortening the - instruction and the context that travel with every request, capping response length, caching a prompt prefix that - repeats, and moving work that tolerates delay to batch.
Monitoring the AI-specific cost patterns, including token usage, the cost of computing embeddings, and the storage a - vector index consumes.
Watching an agent's resource consumption per invocation, because one agent request produces several model calls and tool calls.

Two points are worth holding precisely. First, prompt caching is not a free discount. Cached tokens are billed at a reduced rate, tokens written into the cache can be billed higher than uncached input tokens depending on the model, a checkpoint has a minimum prefix length, and any dynamic content mixed into the prefix invalidates it, so the saving depends on a stable prefix that is reused often enough to repay the writes. Second, agent cost is amplification rather than a single charge. A single user request becomes a planning call, one or more tool calls, and a synthesis call, so cost per invocation is the measure to watch, and a retry loop that a human never notices is a bill that grows without any change in traffic.

Knowledge Check

A team enables prompt caching for an assistant whose prompt prefix includes a value that changes on every request. What is the likely result? (Choose 1)

A. The cache is invalidated on each request, so the saving does not materialize
B. The response length is reduced automatically
C. The request is routed to a smaller model
D. The workload is billed at the batch rate instead

Answer & Explanation

Correct Answer: A. The cache is invalidated on each request, so the saving does not materialize

Explanation:
Prompt Caching (e.g., on Amazon Bedrock or Anthropic Claude) relies on exact token sequence matching from the beginning of the prompt to reuse pre-computed Key-Value (KV) attention states. When a dynamic or volatile variable (such as a timestamp, request ID, or dynamic metadata) is placed inside or before the static prompt prefix, the sequence hash changes on every invocation. This causes a continuous cache miss/invalidation, preventing latency reductions and token cost savings from materializing. Best practice requires placing static content (system instructions, tool schemas) at the beginning and dynamic inputs at the end.

Why the other options are incorrect: * B. The response length is reduced automatically: Prompt caching optimizes input token pre-fill processing; it does not alter or truncate generated response lengths. * C. The request is routed to a smaller model: Caching mechanisms do not trigger automatic model routing or failovers; invocation remains on the target model. * D. The workload is billed at the batch rate instead: Cache misses default to standard real-time on-demand input token rates, rather than batch processing rates.

Optimizing ML and AI costs

Original video transcript

Cost optimization is where experienced people get overconfident. They've tuned endpoint costs for years, so they assume the same levers work everywhere. Right? And then a generative AI application appears on the bill, and none of those levers move it. The spending follows completely different rules, so the optimization has to as well. Exactly. So let's work both sides of that. I'll give you the traditional problem first. You optimize it, and then we'll see what transfers. All right. You're an ML engineer at a media company running recommendation models on Amazon Sagemaker AI real-time endpoints. The instances look oversized and traffic drops to almost nothing overnight. Dozens of small models each hold their own endpoint, and a steady baseline runs all year. Where do you start? I'd start with evidence, not instinct. Amazon Cloudwatch shows each endpoint's utilization. If the instances sit mostly idle, I'd move them to a smaller instance family. OK, that's right sizing. What about the overnight drop though? Well, that's a job for auto scaling. The endpoint adds instances for daytime traffic and releases them overnight, so capacity follows demand instead of waiting for a peak. Daytime traffic is heavy here, so the endpoint stays real time rather than moving the serverless, and the dozens of small models, dozens of endpoints sounds expensive. It is because every endpoint builds for its own instances. So since these models share a framework, a multi-model endpoint hosts them all on shared compute, and they load as they're called. So one set of instances serves dozens of models. OK. And the steady baseline that runs all year. Well, steady usage is what purchasing options are for. Committing to that baseline with Sagemaker AI savings plans costs less than paying the on-demand rate for capacity that you know you'll use. Notice the question sitting under every one of those moves. What are you actually paying for? Remember that question because the second scenario depends on it. So here it comes. Freddie. The same media company launches a generative AI assistant on Amazon Bedrock, and the bill starts climbing. Token spend is rising, the prompts are long, the model is bigger than the job needs, and the volume is steady and high. Let's optimize it. All right, well, I'll start with your question from before, Melissa. What am I actually paying for? Tokens, not ours. There's no instances the right size here because bedrock on Demand charges for the tokens coming in and going out, which means the right sizing instinct, the very first move from the endpoint world, does nothing here. Not directly. That's true, but it is a similar idea. Right sizing the instance becomes right sizing the model. If a smaller model meets the quality requirement for this assistant, selecting it cuts the cost of every single request. OK. And what about the long prompts? Well, every word in a prompt adds tokens, and those tokens are billed on every request. So trimming instructions and shortening context cut input tokens, and capping response length cuts output tokens. Cashing a prompt prefix that repeats lowers the cost of resending it, so on an end point nobody counts words, but here every token has a price. So how do you keep the token spend from surprising anyone again? Well, AWS budgets and cost allocation tags and the cost management tools with alerts before the spend passes the threshold, not after. Token usage becomes a dashboard metric which is watched like any other. Once the whole team can see the token spend, behavior changes before any threshold is reached, and the steady high volume in the first scenario, steady baseline meant savings plans. The same reasoning applies, but the mechanism changes. Predictable high volume inference on a supported foundation model is a case for provision throughput, which reserves model units at a fixed hourly rate. Variable or experimental traffic stays on demand. So the commitment idea survives, but the unit changed from instance hours to model throughput. Perfect. So let's review what transferred and what didn't. Well, the question transferred perfectly. What are you actually paying for? But the answers diverged with compute hours on one side and tokens on the other. Right sizing became model selection. Auto scaling became prompt discipline. And savings plans became provision throughput, and anyone who applies the endpoint levers to the bedrock bill or the other way around ends up optimizing nothing. So here's a useful self-check. If every translation between those two bills made sense, your cost reasoning is sound, right? And if any lever surprised you, especially prompt length as a cost control, that's your gap. Go back, understand what each bill is actually charging for, and then keep studying.

The core shift in machine learning cost optimization comes down to answering one fundamental question: What are you actually paying for? Traditional ML endpoints bill for provisioned compute hours, whereas Generative AI services like Amazon Bedrock bill primarily for input and output token volume. Applying instance-level optimization techniques to token-based workloads yields no savings.

1. Traditional ML Optimization (Amazon SageMaker AI)

Billing Driver: Dedicated instance hours and provisioned compute capacity.

Underutilized Capacity $\rightarrow$ Right-Sizing: Analyze CloudWatch utilization metrics and move idle or oversized endpoints to smaller instance families.
Diurnal Traffic Drops $\rightarrow$ Auto Scaling: Configure target-tracking auto scaling policies to add instance capacity for peak daytime traffic and release instances overnight.
Fragmented Small Models $\rightarrow$ Multi-Model Endpoints (MMEs): Consolidate dozens of small models sharing the same framework onto a single set of shared compute instances, loading models on demand.
Steady Year-Round Baseline $\rightarrow$ SageMaker AI Savings Plans: Commit to a steady baseline of compute usage in exchange for discounted hourly rates compared to On-Demand pricing.

2. Generative AI Optimization (Amazon Bedrock)

Billing Driver: Token throughput (input context length and generated output length).

Overpowered Models $\rightarrow$ Model Selection: Replace large, expensive foundation models with smaller, domain-appropriate models that satisfy the quality requirement at a lower cost-per-token.
Long Context & Volatile Prompts $\rightarrow$ Prompt Discipline & Caching:
Trim context windows and reduce unnecessary system instructions to lower input tokens.
Cap maximum output response lengths to control generated tokens.
Implement Prompt Caching on repeating static prompt prefixes to eliminate redundant token processing costs.
Predictable High Volume $\rightarrow$ Provisioned Throughput: Shift from On-Demand token pricing to Provisioned Throughput, reserving dedicated model units at a fixed hourly rate.
Cost Governance $\rightarrow$ Token Observability: Track token usage as a core operational metric via CloudWatch, using AWS Budgets and Cost Allocation Tags to trigger alerts before spending thresholds are breached.

Cost Optimization Translation Matrix

Optimization Lever	Traditional ML (SageMaker Endpoints)	Generative AI (Amazon Bedrock)
Billing Unit	Compute / Instance Hours	Token Volume (Input & Output Tokens)
Capacity Management	Instance Right-Sizing (Smaller instance types)	Model Selection (Smaller foundation models)
Demand Fluctuation	Auto Scaling (Adjust instance count dynamically)	Prompt Discipline & Caching (Reduce token length & reuse KV cache)
Baseline Commitment	SageMaker AI Savings Plans (Discounted compute rate)	Provisioned Throughput (Reserved model units)

Quick recall

Without rewatching, write the one question the conversation returns to: what are you actually paying for. Then name what each bill charges for, which the video gives as compute hours on a hosted endpoint and tokens on foundation model inference. Then, name the three translations between them: rightsizing becoming model selection, auto scaling becoming prompt discipline, and Savings Plans becoming provisioned throughput.

Find your gap

The video ends on a self-check. If every translation between the two bills made sense as it went, your cost reasoning is ready. If any lever surprised you, especially prompt length as a cost control, that is your gap. If necessary, return to the skills lesson above and work out what each bill is actually charging for.

Knowledge Check

A steady, year-round inference baseline runs on Amazon SageMaker AI instances, and the team wants a commitment that reduces the rate charged for that usage. Which purchasing option applies? (Choose 1)

A. SageMaker AI Savings Plans
B. Compute Savings Plans
C. EC2 Instance Savings Plans
D. A budget with an alert on the account's monthly spend

Answer & Explanation

Correct Answer: A. SageMaker AI Savings Plans

Explanation:
SageMaker AI Savings Plans provide a flexible pricing model specifically for Amazon SageMaker workloads. By committing to a consistent dollar-per-hour spend over a 1-year or 3-year term, you receive discounts (up to 64%) across SageMaker instance usage, including Real-Time Inference, Batch Transform, Training, Processing, and Studio Notebooks.

Why the other options are incorrect: * B. Compute Savings Plans: Compute Savings Plans apply automatically to Amazon EC2, AWS Fargate, and AWS Lambda usage, but they do not extend coverage to SageMaker ML instance types. * C. EC2 Instance Savings Plans: EC2 Instance Savings Plans apply exclusively to standard Amazon EC2 instances within a specific instance family and AWS Region, providing no discounts for SageMaker resources. * D. A budget with an alert on the account's monthly spend: AWS Budgets tracks spending and sends notifications when cost thresholds are approached or exceeded; it is an observability and tracking tool that does not lower billing rates.