Skill 2.1.1: Evaluate and select appropriate FMs from Amazon Bedrock based on task requirements and performance criteria
Task 2.1: ML Model and Foundation Model (FM) Development - Choose appropriate modeling approaches for ML and AI solutions
Core objective: Choose the right Bedrock FM for a task by matching modality, quality, latency, cost, context, customization, and compliance—not by defaulting to the largest or newest model.
1. FM Selection Decision Flow
- Define task first: summarization, Q&A, chat, code, extraction, classification, image generation, embeddings, multimodal reasoning.
- Check modality: text-only vs multimodal (image/video/audio input) vs embedding vs image generation. Not all FMs support all modalities.
- Set performance criteria: quality/accuracy, latency SLA, throughput, cost per token, context window, customization support, safety, region availability.
- Shortlist models: use Bedrock Model Evaluation, provider benchmarks, and representative domain data.
- Decide customization path: prompt engineering → RAG → fine-tuning/continued pre-training/distillation only if evaluations prove a persistent gap.
- Validate operations: on-demand vs Provisioned Throughput vs batch; streaming for interactive UX.
- Apply guardrails: Bedrock Guardrails are model-agnostic and can enforce PII, harmful content, and denied topics across FMs.
2. Selection Criteria Matrix
| Criterion | What to check | Exam trade-off |
|---|---|---|
| Task fit | Text gen, summarization, Q&A, code, image, embeddings | Wrong modality = unusable model |
| Quality | Evaluation scores on domain-specific data | Larger models often higher quality but slower/pricier |
| Latency | Model size, streaming, Provisioned Throughput | Smaller models + streaming reduce perceived latency |
| Cost | Input/output token pricing, batch discounts, distillation | Batch and distillation lower cost; quality may drop |
| Context window | Max input+output tokens | Long docs need RAG/chunking or long-context FM |
| Customization | Fine-tuning, continued pre-training, distillation support | Not all FMs support all customization types |
| Safety | Guardrails, content filters, PII handling | Guardrails apply independently of model choice |
| Compliance | Region availability, data privacy, VPC/KMS | Model access is per Region; data stays in Region |
3. Bedrock Model Evaluation & Performance Criteria
| Method | Use when | Key metrics / notes |
|---|---|---|
| Automatic evaluation | Compare FMs on standard or custom datasets | Accuracy, robustness, toxicity, bias, coherence, fluency, relevance |
| Human evaluation | Subjective quality, tone, brand voice | Custom metrics, human reviewers |
| Custom prompt datasets | Domain-specific selection | Must represent production traffic |
| A/B or shadow testing | Pre-production validation | Compare latency, cost, quality in real conditions |
Model Evaluation is for selection, not just reporting. Use it to justify FM choice against business criteria before deployment.
4. Task-to-Model Fit
| Task | Recommended FM characteristics | Bedrock feature / note |
|---|---|---|
| Summarization | Strong language understanding, controllable length | Prompt + few-shot; fine-tune for style/terminology |
| Q&A over docs | Good synthesis, citation support | Knowledge Bases for RAG |
| Chatbot | Multi-turn context, streaming | Converse API, Guardrails, Agents for tools |
| Code generation | Code-capable FM | Select code-focused models |
| Image generation | Image FM | Stability AI, Amazon Titan Image |
| Embeddings | Embedding model, not generation model | Titan Embeddings, Cohere Embed |
| Classification | Zero-shot FM or trained ML model | Use FM if no labeled data; train ML if labels exist |
| Document extraction | Managed AI service first | Textract for forms/invoices; FM for unstructured text |
| Multimodal reasoning | Vision-capable FM | Check image/video input support |
5. Customization vs Retrieval vs Prompting
| Approach | Use when | Trade-off |
|---|---|---|
| Prompt engineering | Quick style/format, few-shot examples | No training, immediate, limited by context window |
| RAG / Knowledge Bases | Dynamic knowledge, citations, freshness | External vector store, retrieval latency, chunking required |
| Fine-tuning | Domain terminology, style, format, behavior | Training cost, versioning, catastrophic forgetting risk |
| Continued pre-training | Domain adaptation with unlabeled data | Expensive, needs large domain corpus |
| Distillation | Reduce cost/latency of a larger teacher | Smaller student may lose quality; needs prompt dataset |
Prompt first, fine-tune second. Fine-tuning is justified only when evaluations show a gap that prompting and RAG cannot close. RAG vs fine-tuning: RAG for fresh facts and citations; fine-tuning for behavior, style, and terminology. Never fine-tune to update dynamic facts.
6. Bedrock Operations & Throughput
| Mode | Use when | Exam note |
|---|---|---|
| On-demand | Variable/sporadic traffic | Pay per token; quotas per model per Region |
| Provisioned Throughput | Predictable high volume, guaranteed throughput | Hourly commitment; reduces latency variability |
| Batch inference | Large async jobs, not latency-sensitive | Cost-effective; async |
| Streaming | Chat/interactive UX | Reduces perceived latency; total cost similar |
| Agents | Multi-step tasks, tool use | Orchestrates FM + APIs + Knowledge Bases |
Provisioned Throughput ≠ on-demand. Use PT for sustained production load; on-demand for variable or dev/test.
7. Security, Compliance & Data Privacy
- IAM roles: Least privilege for Bedrock actions; separate roles for invoke, customize, evaluate.
- KMS: Encrypt custom models, Knowledge Bases, and Agents at rest with customer-managed keys.
- TLS: Encryption in transit for all Bedrock API calls.
- VPC endpoints (PrivateLink): Private connectivity without public internet.
- CloudTrail: Audit Bedrock API activity.
- Data privacy: Bedrock does not use customer prompts/completions to train base models; data is not shared with model providers.
- Guardrails: Model-agnostic safety layer for PII, harmful content, denied topics, and contextual grounding.
- Region availability: Model access and features vary by Region.
Model access is regional. You must request access to each FM in each AWS Region before invoking it. Guardrails are model-agnostic. Apply them to any FM without changing the model itself.
8. Anti-Patterns & Exam Gotchas
- Defaulting to the largest FM: Often slower, costlier, and unnecessary. Evaluate against task requirements.
- Fine-tuning before prompting/RAG: Wastes time and money; prompt engineering and RAG usually close the gap first.
- Using an FM for tabular regression/forecasting: Traditional ML on SageMaker is better for structured historical data.
- Ignoring latency/cost SLAs: Quality alone is not enough; non-functional criteria drive model choice.
- Fine-tuning for dynamic facts: Use RAG/Knowledge Bases instead.
- Forgetting context window limits: Chunk documents and use RAG when input exceeds model limits.
- Using a generation model for embeddings: Use a dedicated embedding model.
- Assuming all FMs support customization: Check fine-tuning, continued pre-training, and distillation support per model.
- Overlooking batch inference: For large, non-urgent workloads, batch can significantly reduce cost.
- Neglecting Guardrails: Safety and PII enforcement should not determine FM choice; Guardrails can be applied independently.
Exam rule of thumb: Match the model to the task, measure with Model Evaluation, optimize with RAG/fine-tuning only when needed, and enforce safety with Guardrails.