Skill 4.2.9: Manage FM Inference Costs with Usage Optimization

Task 4.2: Operating, Monitoring, and Securing ML and AI Solutions — Optimize and manage ML and AI infrastructure costs and performance

Objective: Cut cost per unit of FM work without breaking quality, latency, or reliability. Bedrock FM inference is billed on token volume (input + output), not provisioned compute — so instance-level levers (right-sizing, auto scaling) move this bill by zero.

The Billing Model Is the Whole Game

Workload	Billed For	Right First Lever
SageMaker hosted endpoint	Instance-hours held	Instance right-sizing, auto scaling, MME
Bedrock On-Demand	Input + output tokens	Model selection, prompt discipline
Bedrock Batch	Tokens at a reduced rate	Move delay-tolerant work
Bedrock Provisioned Throughput	Model units / hour	Predictable high steady volume

Exam trap: "The endpoint looks oversized" reasoning never transfers to a Bedrock bill — there are no instances to resize. Translate the intent, not the mechanism: right-sizing → model selection; auto scaling → prompt discipline; Savings Plans → Provisioned Throughput.

Usage Optimization Levers (by typical impact)

Right-size the model — smallest FM that meets the quality bar. Cost scales with model tier, so this cuts the price of every request. Validate against an eval set first.
Trim the input — system instruction, few-shot examples, retrieved context, and tool schemas are billed on every call.
Cap the output — max output length bounds the most expensive token class; uncapped responses are open-ended spend.
Cache the stable prefix — reuse KV state for a repeated prompt prefix.
Batch the delay-tolerant work — asynchronous jobs at a lower token rate.
Commit the steady baseline — Provisioned Throughput when volume is high and predictable.

Prompt Caching: Not a Free Discount

Rule	Consequence
Cached read tokens	Billed at a reduced rate
Cache write tokens	Often billed higher than uncached input
Minimum prefix length	Short prefixes are not cacheable
Exact prefix match required	Any dynamic content invalidates the cache

Design pattern: static content first (system prompt, tool definitions, reference docs), dynamic content last (user turn, timestamp, request ID, session state).
Payback test: the saving only materializes if the same prefix is reused often enough to repay the cache writes.

Gotcha: A timestamp, request ID, or user-specific value inside the prefix → cache miss on every call → you pay the write premium and get none of the reuse. Most-tested caching failure mode.

Capacity Mode Decision Matrix

Traffic Pattern	Mode	Why
Interactive, variable, low-to-moderate	On-Demand	Per-token, no commitment, no waste
Large payloads / long-running / delay-tolerant	Batch	Reduced token rate; results delivered asynchronously
High, steady, predictable volume	Provisioned Throughput	Reserved model units at a fixed hourly rate; predictable latency + throughput
Spiky, unpredictable, or experimental	On-Demand	Provisioned throughput on spiky traffic = paying for idle model units

Rule: Provisioned Throughput is a commitment, not a scaling tool. Under-utilized model units are wasted spend exactly like idle instances. Match it to a measured steady baseline, not a peak.

Agent Inference Is Amplified Cost

One user request fans out into a planning call → one or more tool calls → a synthesis call; multi-agent setups multiply further.
Watch cost per invocation, not just the account total. Rising spend with flat traffic = retry loops, longer tool chains, or growing context — not more users.
Attribute per-invocation consumption with Amazon Bedrock AgentCore Observability: inference vs. tool calls vs. memory operations.
Cap the loop — bound max iterations/tool calls and set timeouts; an unnoticed retry loop grows the bill with zero traffic change.

AI-Specific Cost Patterns to Monitor

Cost Driver	Optimization
Token usage	Model selection, prompt trimming, output caps, caching, batch
Embedding computation	Batch/backfill embeddings, avoid re-embedding unchanged content, cache results
Vector store storage	Prune stale vectors, right-size index/dimension, tier storage, dedupe

Observability & Cost Governance

CloudWatch — token usage and invocation counts as first-class dashboard metrics. Read tokens alongside invocations: rising tokens + flat invocations = heavier prompts; both rising = more traffic. Different fixes.
AWS Cost Explorer — group/filter by cost allocation tag to attribute spend to a model, team, or environment.
AWS Budgets — threshold and forecasted alerts before the overage, not after.
AWS Trusted Advisor — surfaces idle and underused resources.

Ordering gotcha: Budgets and Cost Explorer only attribute what you tagged. Tagging is a design-time decision, not a cleanup task — without cost allocation tags a cost report shows a service total and no way to say which model produced it.

Security & Compliance Touchpoints

IAM — scope invoke permissions per model and per application role; least privilege limits the blast radius of a runaway caller.
KMS — encrypt cached prompt content, batch input/output, and vector stores at rest; TLS in transit.
VPC endpoints (PrivateLink) — keep Bedrock and vector store traffic off the public internet; required in restricted networks.
Invocation logging / guardrails — logging every prompt and response carries its own storage cost; sample or scope it.

Anti-Patterns

Applying SageMaker levers to a Bedrock bill (or the reverse) — optimizes nothing.
Downsizing the model without an eval — a quality regression is not a saving.
Prompt caching over a dynamic prefix — pays the write premium, gets no reuse.
Provisioned Throughput for spiky or experimental traffic — idle model units.
Batch for interactive traffic — the discount does not survive a latency requirement.
Optimizing total spend without per-invocation visibility — hides retry loops and context growth.
Reducing cost without naming the tradeoff — state what was traded (quality, latency, throughput headroom) before committing.