Skill 2.1.7: Assess tradeoffs between AI model performance, latency, and cost
Task 2.1: ML Model and Foundation Model (FM) Development - Choose appropriate modeling approaches for ML and AI solutions
Core objective: Select the least-costly model/serving pattern that meets the required quality bar, p95 latency SLA, and throughput—while satisfying interpretability, security, and compliance.
Tradeoff Dimensions
| Dimension | Key Metrics | Primary Levers | AWS Controls |
|---|---|---|---|
| Performance | Accuracy, F1, AUC, RMSE, ROUGE, BERTScore, faithfulness, hallucination rate | Model size/architecture, fine-tuning, RAG, ensembles, feature quality | Bedrock model choice, SageMaker algorithms, JumpStart, Clarify |
| Latency | p50/p95/p99, TTFT, inter-token latency, cold start, end-to-end | Model size, quantization, distillation, hardware, caching, streaming, provisioned throughput | Bedrock Provisioned Throughput, SageMaker Real-time/Serverless/Async, Inferentia/GPU, Neo, Prompt Caching |
| Cost | Training cost, inference cost/request/token, hosting, storage, vector DB, data transfer | On-demand vs provisioned, batch, serverless, multi-model, quantization, RAG vs fine-tune | Bedrock On-Demand/Provisioned/Batch, SageMaker Batch Transform, Serverless, Multi-Model, Savings Plans |
Model & Approach Selection
| Approach | Performance | Latency | Cost | Best For |
|---|---|---|---|---|
| Fixed rules | Deterministic | Very low | Very low | Fixed business logic |
| Managed AI service | High for standard task | Low, API-bound | Pay-per-use | Textract, Rekognition, Comprehend, Transcribe |
| Traditional ML | High on tabular/structured | Low | Low–moderate | Forecasting, classification, regression with labeled history |
| Small FM | Moderate | Low | Low | Simple summarization, classification, low-latency GenAI |
| Large FM | High | High | High | Complex reasoning, creative, multi-step |
| Fine-tuned FM | Higher on domain style | Similar/higher | Training + hosting | Domain terminology, format, style |
| Distilled/quantized FM | Near-teacher | Lower | Lower | Latency/cost-sensitive production |
| RAG | Fresh, citable | Adds retrieval latency | Vector DB + tokens | Dynamic knowledge, citations |
Serving Pattern Tradeoffs
| Pattern | Latency | Cost | Throughput | Use Case |
|---|---|---|---|---|
| Real-time endpoint | Low, predictable | Highest | Moderate | Interactive apps, strict p95 |
| Serverless Inference | Cold starts | Pay-per-use | Variable | Intermittent/spiky traffic |
| Asynchronous Inference | Near-real-time | Lower | High | Large payloads, long processing |
| Batch Transform | Hours | Lowest | Very high | Offline scoring |
| Bedrock On-Demand | Variable | Per-token | Shared | Spiky, unpredictable |
| Bedrock Provisioned Throughput | Low, predictable | Fixed hourly | High | Steady high-volume, latency SLA |
| Multi-Model Endpoint | Cold start on load | Lower hosting | High | Many models, low individual traffic |
| Edge | Very low | Hardware + mgmt | Limited | Offline, ultra-low latency |
RAG vs Fine-Tuning vs Prompting
| Approach | Performance | Latency | Cost | Best For |
|---|---|---|---|---|
| Prompt engineering | Baseline | No added latency | Lowest | Instructions, few-shot, format |
| RAG | Fresh facts + citations | + retrieval latency | Vector DB + tokens | Changing knowledge, source attribution |
| Fine-tuning | Style/domain/format | Similar; may reduce prompt length | Training + versioning | Persistent behavior, terminology |
| Distillation | Near-teacher | Lower | Lower inference | Deploy smaller model |
Exam Gotcha: Fine-tuning does not reliably add new factual knowledge. Use RAG for fresh/citable facts. Use fine-tuning for style, format, and domain terminology.
Decision Flow
- Fixed logic? → Rules/code.
- Standard pre-defined task? → Managed AWS AI service.
- Patterns in proprietary tabular data? → Train traditional ML (SageMaker).
- General language/creative? → FM via Bedrock.
- Start with prompt engineering.
- Need fresh/citable facts? → RAG / Bedrock Knowledge Bases.
- Need style/format/terminology? → Fine-tuning.
- Need lower latency/cost? → Distillation/quantization.
- Need predictable high-volume? → Provisioned Throughput.
Evaluation & Optimization Tools
- Amazon Bedrock Model Evaluation: Compare FMs on quality, latency, cost; automatic and human eval.
- SageMaker Inference Recommender: Recommend instance type and endpoint config for latency/throughput/cost.
- SageMaker Clarify: Feature attribution, bias detection for interpretability.
- SageMaker Neo: Compile models for edge/target hardware; lower latency.
- AWS Inferentia/Trainium: Cost-effective inference/training accelerators.
- Prompt Caching (Bedrock): Reduce latency and cost for repeated prompts.
- Guardrails for Bedrock: Safety controls; adds minor latency.
Exam Gotchas
Tradeoff rule: Optimizing one of performance, latency, or cost usually degrades another. Identify the non-negotiable requirement first.
Latency for GenAI: Time-to-first-token (TTFT) and inter-token latency matter more than total model size. Streaming improves perceived latency.
Provisioned Throughput: For steady high-volume, low-latency FM inference. On-demand is for spiky/variable traffic and may throttle.
Batch vs real-time: Batch inference lowers cost per request but is not suitable for interactive latency SLAs.
Cold starts: Serverless Inference and Multi-Model Endpoints can spike p95 latency due to model loading. Provisioned/real-time avoids this.
Interpretability: For regulated/finance use cases, a simpler interpretable model often beats a more accurate black-box model.
Anti-Patterns
- Max accuracy at any cost: Violates latency/cost SLAs; always evaluate against business constraints.
- Fine-tuning for facts: Use RAG; fine-tuning is for behavior/style.
- LLM for tabular regression: Traditional ML (XGBoost, linear) is cheaper, faster, more interpretable.
- Real-time endpoint for batch: Use Batch Transform to reduce cost.
- Ignoring p95/p99: Average latency hides cold starts and queueing.
- Over-provisioning: Provisioned Throughput only for steady high volume.
- No baseline: Always measure prompt vs RAG vs fine-tune vs distillation.
Security & Compliance Constraints (Cost/Latency Impact)
- IAM roles: Least privilege for Bedrock/SageMaker; model access must be granted.
- KMS: Encrypt training data, model artifacts, vector stores, and inference logs at rest; TLS in transit.
- VPC endpoints (PrivateLink): Keep traffic private; may add slight latency but avoids NAT/data transfer costs.
- Data residency: Region choice affects model availability, latency, and compliance.