Skill 4.4: Optimize capacity for cost, performance, and reliability
Task 4.2: Operating, Monitoring, and Securing ML and AI Solutions – Optimize and manage ML and AI infrastructure costs and performance
Core objective: Match capacity to actual demand; reduce cost without violating latency, throughput, or availability SLOs. Always ask: What is being billed?
Two billing worlds: | Workload | Billing driver | Primary levers | |---|---|---| | SageMaker hosted inference | Instance hours / provisioned compute | Right-sizing, auto scaling, MME, serverless, async, Savings Plans | | Bedrock FM inference | Input/output tokens | Model selection, prompt discipline, caching, batch, provisioned throughput |
Exam gotcha: Compute/EC2 Savings Plans do not cover SageMaker AI endpoints. Use SageMaker AI Savings Plans for
ml.*instance usage. Provisioned Throughput is the FM equivalent for predictable token volume.
Capacity Selection Matrix
| Pattern | Best option | Why / trade-off |
|---|---|---|
| Steady interactive, low latency | SageMaker Real-Time Endpoint | Always-on instances; predictable latency; cost of idle capacity |
| Intermittent / spiky, cold start OK | SageMaker Serverless Inference | No idle cost; cold-start latency; memory/time limits |
| Long-running or large payloads | SageMaker Asynchronous Inference | Queue + auto scale to zero; up to 15 min/1 GB; not for sub-second |
| Offline, large batch | SageMaker Batch Transform | No persistent endpoint; cost per job |
| Many small models, shared framework | Multi-Model Endpoint (MME) | Shared compute; model load latency on cold request |
| Predictable high-volume FM | Bedrock Provisioned Throughput | Reserved model units; fixed hourly rate; commit for steady traffic |
| Asynchronous FM jobs | Bedrock Batch Inference | Lower cost for delay-tolerant work; not real-time |
| Variable/experimental FM | Bedrock On-Demand | Pay per token; no commitment |
Right-Sizing, Scaling, and Reliability
- Evidence first: Read CloudWatch utilization over a representative window. Busy 1h/day + idle rest = scaling problem, not sizing problem.
- Right-sizing: Move to smaller instance family only when CPU/GPU/memory headroom is consistently excessive. Do not downsize to average if spikes must be absorbed.
- Auto scaling: Target tracking on
InvocationsPerInstance, CPU, GPU, or custom metrics. Scale out for peak, scale in overnight. Set min capacity for baseline reliability. - Multi-Model Endpoints: Consolidate many models on shared instances. Good for intermittent calls; adds load latency if model not resident.
- Serverless / Async: Scale to zero. Trade latency for cost. Async is not for real-time request/response.
- Reliability trade-off: Reducing capacity removes headroom. Restate SLO and test failover. Auto scaling helps, but cold starts and scale-in delays affect tail latency.
Gotcha: Do not use Spot/Interruptible capacity for a serving baseline. Spot is for checkpointable training or interruptible batch, not guaranteed inference.
Purchasing Options & Coverage
| Option | Covers | Use when |
|---|---|---|
| SageMaker AI Savings Plans | SageMaker AI instance usage (inference, training, notebooks, processing) | Steady year-round SageMaker baseline |
| Compute Savings Plans | EC2, Lambda, Fargate | Steady compute across those services |
| EC2 Instance Savings Plans | Specific EC2 family/region | Steady EC2 usage; not SageMaker |
| On-Demand | No commitment | Variable, experimental, unpredictable |
| Provisioned Throughput (Bedrock) | Reserved model units | Predictable high-volume FM inference |
| Spot / Interruptible | Spare EC2 capacity | Training with checkpoints, fault-tolerant batch |
Gotcha: A Budget is not a discount. It tracks spend and alerts; it does not lower rates. Use tags for attribution.
Monitoring & Observability
| Tool | Answers | Use for |
|---|---|---|
| Amazon CloudWatch | Metrics + logs | Utilization, latency, errors, invocations, token usage, dashboards, alarms |
| AWS X-Ray | Distributed traces | Attribute latency to model, tool, retrieval, or agent coordination |
| Bedrock AgentCore Observability | Agent invocation internals | Inference calls, tool calls, memory ops per invocation |
| AWS Compute Optimizer | Resource recommendations | Rightsizing supported compute from utilization history |
| AWS Cost Explorer | Historical/forecast spend | Find cost driver by tag, service, usage type |
| AWS Budgets | Threshold alerts | Forecasted/actual spend or usage alerts |
| AWS Trusted Advisor | Best-practice checks | Idle/underused resources, cost checks |
Dashboard rule: Put measures that move together on one dashboard: utilization + latency, invocations + token usage, cost + errors. A dashboard should shorten a decision.
FM, Embedding, and Vector Cost Levers
- Model selection: Use the smallest model that meets quality. This is the FM equivalent of instance right-sizing.
- Prompt discipline: Trim instructions/context; cap max output tokens. Every token is billed on every request.
- Prompt caching: Caches stable prefix. Requires exact prefix match. Dynamic content in prefix invalidates cache. Cached tokens cheaper, but cache writes may cost more; minimum prefix length applies.
- Batch inference: Move delay-tolerant work to Bedrock Batch for lower cost.
- Provisioned Throughput: For predictable high volume; on-demand for variable.
- Embeddings: Monitor embedding computation costs; cache/reuse embeddings where possible; batch embedding jobs.
- Vector DB storage: Optimize index size, dimensionality, and retention; delete stale vectors; choose storage tier appropriate to query patterns.
- Agent cost: One user request = planning + tool calls + synthesis. Watch cost per invocation, retry loops, and memory operations.
Gotcha: Prompt caching is not automatic savings. If the prefix changes every request, cache misses; you pay write costs without reuse benefit.
Security, Compliance, and Cost Attribution
- IAM: Least-privilege execution roles for SageMaker endpoints, Bedrock, agents. Separate roles per environment.
- Encryption: KMS at rest for model artifacts, EBS volumes, vector databases, CloudWatch Logs. TLS in transit.
- Network: VPC endpoints (PrivateLink) for SageMaker/Bedrock to keep traffic private and avoid NAT gateway data processing costs.
- Tagging: Cost allocation tags on endpoints, models, jobs, agents. Without tags, Cost Explorer shows service totals, not team/model/environment attribution.
- Quotas: Set service quotas and budgets as ceilings. Monitor usage against them.
Anti-Patterns / Exam Traps
- Applying EC2 right-sizing to Bedrock: Token bill has no instances; optimize model, prompt, and caching instead.
- Using Compute/EC2 Savings Plans for SageMaker: Wrong coverage. Use SageMaker AI Savings Plans.
- Auto scaling a steady baseline: If traffic is flat, Savings Plans/Provisioned Throughput are better than scaling.
- Serverless for sustained high traffic: Serverless is for intermittent; steady high volume is cheaper on real-time with Savings Plans.
- Async for real-time UX: Async adds queueing; use real-time for interactive.
- MME for high-traffic single model: MME is for many intermittently called models; can add cold-start load latency.
- Ignoring token observability: Token usage must be a CloudWatch metric/dashboard, not just a monthly bill surprise.
- No cost allocation tags: Cannot attribute spend; optimization becomes guesswork.