Skill 4.2.7: Evaluate cost implications of using FMs for inference in production
Task 4.2: Operating, Monitoring, and Securing ML and AI Solutions - Optimize and manage ML and AI infrastructure costs and performance
Core question: What are you actually paying for? FM inference is billed by tokens/model units/throughput, not instance-hours. Applying SageMaker endpoint right-sizing to Bedrock token spend yields no savings.
FM Inference Cost Drivers
| Billing Dimension | What Drives Cost | Production Action |
|---|---|---|
| Input tokens | System prompt, user prompt, history, retrieved context, tool schemas | Trim instructions, summarize history, re-rank/fewer RAG chunks, cache stable prefixes |
| Output tokens | Generated response length; often priced higher than input | Cap max output, use concise/structured responses, stop sequences |
| Model choice | Per-token rate varies by FM size/family | Use smallest model that meets quality; larger model only where needed |
| Capacity mode | On-demand, batch, provisioned throughput | Match mode to traffic pattern; see decision matrix |
| Embeddings | Tokens/requests to generate vectors | Compute once, reuse/cache, batch, smaller embedding model if quality allows |
| Vector store | Storage, index build, queries, OCUs/RCUs | Reduce dimensions, prune stale vectors, tier storage, right-size index |
| Agents | One user request = multiple model/tool/memory calls | Cap iterations, cache tool results, route simple steps to smaller models |
| Observability/security | Logs, traces, KMS, VPC endpoints, Guardrails | Sample logs, set retention, use managed keys where acceptable |
Capacity Mode Decision Matrix
| Mode | Billing | Best For | Avoid When | Trade-off |
|---|---|---|---|---|
| On-Demand | Per input/output token | Intermittent, variable, experimental, low-volume | Steady high volume | Highest per-token rate; no commitment |
| Batch | Per token, typically discounted vs on-demand | Async jobs, large payloads, no latency SLA | Interactive/real-time | Delay; S3 input/output; not instant |
| Provisioned Throughput | Hourly per model unit | Predictable high volume, latency-sensitive | Spiky/unpredictable/experimental | Pay for reserved capacity even if idle |
| Prompt Caching | Cached input tokens at reduced rate; cache writes may cost more | Repeated long static prefix | Dynamic prefix, low reuse | Min prefix length; invalidation risk |
Provisioned Throughput is not auto-scaling. It reserves model units for a term. If utilization is low, you overpaid; if throttled, you under-provisioned.
Cost Optimization Levers
- Model right-sizing: Smaller FM = lower cost per token. Trade-off: quality/reasoning depth. Distractor: instance right-sizing does not apply to Bedrock.
- Prompt discipline: Shorter system prompts, fewer few-shot examples, compressed history, fewer retrieved chunks. Trade-off: context/accuracy.
- Output capping: Set max output tokens. Trade-off: truncated answers.
- Prompt caching: Reuse stable prefix. Requires min prefix length, cache TTL, high reuse. Dynamic content in prefix invalidates cache.
- Batch inference: Move delay-tolerant work to batch. Trade-off: no real-time response.
- Provisioned throughput: Commit for predictable high volume. Trade-off: idle cost.
- Embedding reuse: Store embeddings; avoid recomputing unchanged content. Trade-off: staleness.
- Vector DB optimization: Lower dimensions, prune, tier, right-size index. Trade-off: recall/latency.
- Agent optimization: Cap loops, cache tool outputs, use smaller router model. Trade-off: capability.
Traditional ML vs FM Inference Cost Translation
| Lever | Traditional ML / SageMaker Endpoints | Generative AI / Bedrock |
|---|---|---|
| Billing unit | Instance/compute hours | Tokens/model units |
| Capacity management | Instance right-sizing | Model selection |
| Demand fluctuation | Auto Scaling | Prompt discipline + caching |
| Baseline commitment | SageMaker AI Savings Plans | Provisioned Throughput |
Monitoring, Governance & Attribution
| Tool | What to Watch | Cost Action |
|---|---|---|
| CloudWatch | Token usage, invocations, latency, errors, provisioned throughput utilization | Dashboard token spend beside invocations; alarm on anomalies |
| Bedrock AgentCore Observability | Per-invocation inference, tool, memory calls | Detect agent cost amplification/retry loops |
| AWS X-Ray | Latency across model/tool/retrieval | Attribute latency before adding capacity |
| Cost Explorer | Spend by tag/service/usage type; forecast | Find driver behind rising bill |
| AWS Budgets | Actual/forecasted spend vs threshold | Alert before overage |
| Cost Allocation Tags | Team/model/env attribution | Prerequisite for chargeback/showback |
| Trusted Advisor / Compute Optimizer | Idle/underused compute resources | Rightsize EC2/Lambda/EBS, not FM tokens |
| Service Quotas | Bedrock TPM/RPM, provisioned throughput limits | Prevent runaway throughput; may cause throttling |
CloudWatch alone does not alert on forecasted spend. AWS Budgets does. X-Ray traces latency, not cost. Trusted Advisor checks idle resources, not token spend.
Purchasing Options & Coverage
| Option | Covers | Does Not Cover | Use When |
|---|---|---|---|
| Bedrock On-Demand | FM token usage | Commitment discounts | Variable/experimental traffic |
| Bedrock Batch | Async FM inference | Real-time interactivity | Delay-tolerant jobs |
| Bedrock Provisioned Throughput | Reserved model units | Spiky/unpredictable traffic | Predictable high volume |
| SageMaker AI Savings Plans | SageMaker ML instances | Bedrock, EC2, Lambda, Fargate | Steady SageMaker baseline |
| Compute Savings Plans | EC2, Lambda, Fargate | Bedrock, SageMaker | Steady compute baseline |
| EC2 Instance Savings Plans | EC2 specific family/region | Bedrock, SageMaker | Steady EC2 baseline |
No Savings Plan covers Bedrock on-demand token spend. For Bedrock commitment, use Provisioned Throughput. SageMaker AI Savings Plans cover SageMaker only; Compute/EC2 Instance Savings Plans do not cover SageMaker or Bedrock.
Hidden Cost Adders
| Component | Cost Driver | Mitigation |
|---|---|---|
| Model invocation logging | CloudWatch Logs ingestion/storage | Sample, set retention, log metadata only if possible |
| VPC endpoints for Bedrock | Hourly + data processing | Use only where private connectivity required |
| KMS encryption | API calls for CMK | Use AWS-managed keys where compliance allows |
| Guardrails | Per text unit processed | Apply only to required flows |
| Batch S3 storage | Input/output object storage | Lifecycle policies, delete after processing |
| Data transfer | Cross-region/internet egress | Keep data and inference in same region |
Exam Gotchas & Anti-Patterns
Agent cost is amplification, not a single charge. One user request becomes planning + tool + synthesis calls. Monitor cost per invocation.
Prompt caching is not free. Cache writes may cost more than uncached input; cache reads are cheaper. Dynamic content in the prefix invalidates savings.
- Right-sizing instances for Bedrock: No effect; right-size the model instead.
- Using Compute Savings Plans for Bedrock/SageMaker: Not covered.
- Using on-demand for steady high volume: Evaluate Provisioned Throughput.
- Using Provisioned Throughput for spiky/experimental: You pay for idle reserved capacity.
- Ignoring output token cost: Output often costs more than input; cap response length.
- Monitoring only account total: Cannot attribute to model/team/environment without tags.
- Ignoring embeddings and vector DB: Total FM cost includes embedding generation and vector storage/query.
- Adding capacity to fix latency: Trace first; bottleneck may be retrieval or tool call.
- No max output cap: Runaway generation cost.
- Agent retry loops: Cost grows without traffic growth.
- Caching volatile prefix: Cache miss and write cost with no savings.