Skill 4.2.5: Optimize costs and set cost quotas by using appropriate cost management tools
Task 4.2: Operating, Monitoring, and Securing ML and AI Solutions - Optimize and manage ML and AI infrastructure costs and performance.
Core Objective & Billing Drivers
Fundamental question: What are you actually paying for?
SageMaker hosted inference → instance hours. Bedrock on-demand → tokens. Bedrock provisioned → model units/hour. Training → instance hours. Agents → amplified model + tool calls.
| Workload | Billing Driver | Primary Cost Levers | Anti-Pattern |
|---|---|---|---|
| SageMaker real-time endpoint | Instance hours | Right-sizing, Auto Scaling, MME, Savings Plans | Applying token levers |
| Bedrock on-demand | Input + output tokens | Model selection, prompt discipline, caching, batch | Right-sizing instances |
| Bedrock provisioned throughput | Model units/hour | Commitment for predictable high volume | Using for variable traffic |
| Training / processing | Instance hours | Spot + checkpointing, Savings Plans | Spot for real-time inference |
| Agents | Amplified calls | Per-invocation observability, retry caps | Ignoring retry loops |
Cost Management Tools — Selection Matrix
| Tool | Primary Use | Key Features | Exam Gotcha |
|---|---|---|---|
| AWS Cost Explorer | Visualize/analyze spend | Group/filter by tag, service, usage type; forecasts; RI/SP recommendations | Does not send alerts |
| AWS Budgets | Cost/usage quotas & alerts | Cost, usage, RI/SP budgets; forecasted alerts; Budget Actions | Not a rate discount |
| AWS Cost Anomaly Detection | ML-based anomaly detection | Automatic baselines; alerts on unusual spend | No manual thresholds |
| AWS Cost and Usage Report (CUR) | Most granular cost data | Line items to S3; query with Athena/Redshift | Not real-time |
| Cost Allocation Tags | Attribution | Team/model/env attribution | Must be activated |
| AWS Trusted Advisor | Best-practice checks | Idle/underused resources | Not custom budgets |
| AWS Compute Optimizer | Right-sizing recs | Supported compute resources | Not all services |
| AWS Service Quotas | Resource limits | Set/request quotas; monitor usage | Limits resources, not dollars |
| AWS Organizations | Governance | Consolidated billing, SCPs, volume discounts | SCPs can restrict instance types/regions |
Setting Cost Quotas & Guardrails
- AWS Budgets = primary cost quota tool. Set cost budget, usage budget, forecasted alerts. Use Budget Actions to trigger SNS, apply IAM/SCP, or stop EC2/RDS.
- Service Quotas = resource quotas. Cap SageMaker endpoint instance counts, Bedrock model units, tokens per minute, etc. Prevents runaway resource sprawl.
- SCPs = hard guardrails. Restrict instance families, regions, or services (e.g., block GPU instances in dev accounts).
- Tag Policies = enforce consistent cost allocation tags across accounts.
| Scenario | Tool |
|---|---|
| Alert when forecasted spend exceeds budget | AWS Budgets |
| Detect unusual spend without manual thresholds | AWS Cost Anomaly Detection |
| Visualize spend by tag/service | AWS Cost Explorer |
| Most granular cost data | CUR |
| Attribute spend to team/model | Cost Allocation Tags |
| Limit SageMaker endpoint instance count | Service Quotas |
| Restrict instance types/regions | SCPs |
| Identify idle/underused resources | Trusted Advisor |
| Right-size supported compute | Compute Optimizer |
Exam Gotcha: AWS Budgets is the tool for forecasted spend alerts and cost quotas. Service Quotas limit resources, not dollar spend. Cost Explorer visualizes but never alerts.
Purchasing Options & Coverage
| Option | Covers | Best For | Does NOT Cover |
|---|---|---|---|
| On-Demand | All | Variable/experimental | N/A |
| SageMaker AI Savings Plans | SageMaker ML instances (training, inference, processing, notebooks) | Steady year-round SageMaker baseline | Bedrock tokens, EC2, Lambda |
| Compute Savings Plans | EC2, Lambda, Fargate | Steady compute baseline | SageMaker AI |
| EC2 Instance Savings Plans | Specific EC2 family/region | Steady EC2 baseline | SageMaker AI |
| Reserved Instances | EC2, RDS, etc. | Steady EC2/RDS | SageMaker AI |
| Spot Instances | EC2, SageMaker training | Interruptible training/processing | Real-time inference |
| Bedrock Provisioned Throughput | Bedrock model units/hour | Predictable high-volume FM inference | Variable traffic |
| Bedrock Batch Inference | Bedrock async jobs | Large, delay-tolerant jobs (discounted) | Real-time interactive |
Critical Exam Gotcha: Compute Savings Plans and EC2 Instance Savings Plans do NOT cover SageMaker AI. Only SageMaker AI Savings Plans cover SageMaker ML instances.
Optimization Levers by Workload
SageMaker Hosted Inference
- Right-sizing: Read CloudWatch utilization (CPU, GPU, memory, InvocationsPerInstance) over a representative window. Move idle/oversized endpoints to smaller families.
- Auto Scaling: Target-tracking policies on InvocationsPerInstance or CPU/GPU. Add capacity for daytime peaks; release overnight.
- Multi-Model Endpoints (MME): Consolidate dozens of small models sharing a framework onto shared compute. Trade-off: cold-start load time for non-resident models.
- Serverless Inference: Intermittent traffic. No idle cost. Cold-start trade-off.
- Asynchronous Inference: Large payloads/long processing. Queued. Can scale to zero.
- Savings Plans: Commit steady baseline for discounted hourly rates.
Bedrock / Foundation Models
- Model selection: Smallest model meeting quality requirement. Cuts cost per request.
- Prompt discipline: Trim instructions, shorten context, cap max output tokens. Every token is billed.
- Prompt caching: Stable prefix reused often. Cached tokens billed at reduced rate; cache writes may cost more; minimum prefix length; dynamic content invalidates.
- Batch inference: Async, large jobs, discounted (no latency SLA).
- Provisioned Throughput: Predictable high volume. Fixed hourly rate for model units.
- Monitoring: Track token usage per invocation, per model via CloudWatch.
Agents
- Amplification: One user request → planning call + tool calls + synthesis call. Cost per invocation is the key metric.
- AgentCore Observability: Captures inference, tool, memory operations per invocation.
- Retry loops: Cap retries, use DLQs, alert on abnormal invocation counts.
Embeddings & Vector DB
- Embedding computation: Batch, cache, use smaller embedding models.
- Vector DB storage: Dimension reduction, quantization, tiered storage, prune stale vectors.
Monitoring, Attribution & Dashboards
- CloudWatch: Metrics, logs, dashboards, alarms. Custom metrics for token usage, cost per invocation.
- AWS X-Ray: Distributed tracing. Attribute latency to a component instead of adding capacity everywhere.
- AgentCore Observability: Per-invocation inference, tool, memory operations for agents.
- Cost Explorer: Filter/group by tag, service, usage type. Forecast spend.
- CUR: Most granular line items for custom analysis.
- Cost Allocation Tags: Prerequisite for attribution. Activate in Billing console. Use Tag Policies to enforce.
- Dashboards: Combine utilization + latency + token usage + invocation count. Utilization beside latency shows capacity issues; token usage beside invocation count shows traffic vs heavier prompts.
Anti-Patterns & Gotchas
Compute Savings Plans ≠ SageMaker. Use SageMaker AI Savings Plans for SageMaker. AWS Budgets alert; they do not discount. Do not confuse with Savings Plans. Cost Explorer does not alert. Use Budgets or Cost Anomaly Detection. Trusted Advisor is not a custom budget tool. Prompt caching is not free. Cache writes may cost more; dynamic prefix invalidates. Spot Instances never for real-time inference. Right-size from measured utilization over a representative window, not a single peak. Without activated cost allocation tags, cost reports show service totals only.
Quick Recall
- Question: What are you actually paying for?
- SageMaker: instance hours → right-size, auto-scale, MME, Savings Plans.
- Bedrock: tokens → model selection, prompt discipline, caching, batch, provisioned throughput.
- Cost quotas: Budgets (dollars/usage), Service Quotas (resources), SCPs (guardrails).
- Attribution: Cost Allocation Tags + CUR + Cost Explorer.