Skill 2.1.3: Compare and select appropriate ML models, GenAI models, algorithms, and solution templates to meet business needs (for example, interpretability, domain-specific performance, latency).
Task 2.1: ML Model and Foundation Model (FM) Development - Choose appropriate modeling approaches for ML and AI solutions.
Core exam rule: Match the problem to where the answer or capability lives: fixed logic → rules; solved task → managed AI service; proprietary patterns → trained ML; general language → FM with prompting, RAG, or fine-tuning only when justified.
1. 4-Tier Selection Hierarchy
| Problem signal | Best approach | AWS option | Why |
|---|---|---|---|
| Fixed deterministic logic | Rule/code | Lambda, Step Functions, simple conditionals | Cheaper, faster, auditable |
| Pre-defined standard task | Managed AI service | Textract, Rekognition, Comprehend, Transcribe | No training, fastest time-to-market |
| Patterns in proprietary data | Trained ML model | SageMaker AI built-in/custom algorithms | Answer is in your data; control interpretability |
| General language/creative/reasoning | Foundation model | Amazon Bedrock | Capability already in model; prompt first |
| Need pre-built model/template | Pre-trained model/solution template | SageMaker JumpStart | Start from working model instead of blank notebook |
2. Selection Criteria That Shape Model Choice
| Criterion | Key question | AWS lever / tradeoff |
|---|---|---|
| Interpretability | Must stakeholders explain/defend predictions? | Linear/tree models + SageMaker Clarify/SHAP vs deep learning/LLM black box |
| Domain-specific performance | Does it perform on your data? | Bedrock model evaluations, SageMaker HPO, custom training |
| Latency | Real-time or batch SLA? | Smaller/distilled models, provisioned throughput, caching, RAG retrieval latency |
| Cost | Per-token vs per-hour vs per-request? | On-demand vs provisioned throughput, smaller models, batch, distillation |
| Training time | Time-to-market pressure? | Managed AI service → pre-trained/JumpStart → custom training |
| Output quality | What metric proves success? | Automatic/human evaluation, A/B tests, business KPIs |
3. FM Selection & Customization Path
- Amazon Bedrock: Single API for multiple FMs. Compare by task fit, quality, latency, cost, context window, modality, region availability, compliance.
- Evaluation first: Use Bedrock model evaluation jobs (automatic/human) — select on measured evidence, not reputation.
- Prompt engineering first: Instructions + few-shot examples. Fastest, cheapest, reversible. Usually closes wording/format gaps.
- RAG: For fresh facts, citations, proprietary documents. Managed: Bedrock Knowledge Bases. Custom: embeddings + vector store + retriever + orchestration. Re-sync updates index.
- Fine-tuning: Supervised fine-tuning (labeled input/output pairs), reinforcement fine-tuning (reward functions), distillation (teacher behavior into smaller student; prepared as prompts, managed path generates teacher responses). Use only when prompting/RAG cannot close behavior/style/terminology gap.
- Catastrophic forgetting: Narrow fine-tuning can degrade general capability. Blend broader data into training mixture.
Exam order: Prompt engineering → RAG for knowledge → Fine-tuning for behavior/style/format. Fine-tuning is not the first move for terminology; it is justified only after evaluations show a persistent gap prompting cannot close.
4. RAG vs Fine-Tuning vs Prompting
| Need | Best fit | Why | Tradeoff |
|---|---|---|---|
| Fresh facts / cite sources / frequent policy edits | RAG | External index; near real-time updates; source citations | Weaker for whole-document synthesis |
| Style, format, tone, internal terminology | Prompting first, then fine-tuning | Writes behavior into weights | No citations; updates need retraining |
| General reasoning, summarization, creative | FM prompting | Capability already in model | May need prompt iteration |
| Both freshness and specialized behavior | RAG + fine-tuned model | Retrieval supplies facts; fine-tuning supplies behavior | Higher complexity/cost |
5. Managed AI Services: Task-to-Service Map
| Business task | AWS service | Conceptual API/feature |
|---|---|---|
| Extract text, forms, tables from scanned docs | Amazon Textract | AnalyzeDocument, AnalyzeExpense |
| Analyze images/video for objects, scenes, moderation | Amazon Rekognition | DetectLabels, DetectModerationLabels |
| Find entities, key phrases, sentiment, language in text | Amazon Comprehend | DetectEntities, DetectSentiment |
| Convert speech to text | Amazon Transcribe | StartTranscriptionJob |
| Chat/search over proprietary documents | Bedrock Knowledge Bases | RetrieveAndGenerate |
| Pre-trained models and solution templates | SageMaker JumpStart | Model/template deployment |
6. Tradeoff Matrix: Custom vs Managed vs Pre-trained vs FM
| Option | Best when | Effort/cost | Control | Interpretability | Latency |
|---|---|---|---|---|---|
| Custom trained model | Unique proprietary patterns; need full control | High | High | Can be high if simple algorithm | Varies |
| Managed AI service | Task is already solved generically | Low | Low | Varies | Low API latency |
| Pre-trained model / JumpStart | Need head start in common domain | Medium | Medium | Medium | Medium |
| FM / Bedrock | General language, summarization, Q&A, creative | Low–medium | Low | Low–medium | Can be high; optimize with model choice |
7. Anti-Patterns & Exam Traps
- Build when managed service exists: Custom OCR for invoices vs Textract AnalyzeExpense.
- Model when logic is fixed: Threshold reorder rule beats ML for deterministic inventory triggers.
- FM for precise tabular forecasting: LLMs are not for large-scale time-series across SKUs/stores; use SageMaker trained ML.
- Fine-tune before prompt: Prompt + few-shot first; fine-tuning adds data prep, training, versioning.
- RAG for style/format: RAG adds facts, not behavior. Fine-tune for tone/terminology.
- Ignore interpretability: Finance/regulated users may reject black-box. Choose interpretable algorithm + Clarify.
- Ignore latency/cost: Larger model may miss SLA. Use smaller/distilled, provisioned throughput, batch, caching.
- Forget citations/freshness: Fine-tuned weights cannot cite and update slowly. RAG can cite and reflect edits quickly.
8. Security, Compliance & Guardrails
- IAM least privilege for Bedrock, SageMaker, and data sources.
- KMS encryption at rest; TLS in transit for training data, model artifacts, RAG stores.
- VPC endpoints / PrivateLink for private access to Bedrock/SageMaker.
- Bedrock Guardrails for PII, harmful content, denied topics.
- SageMaker Clarify for bias detection and explainability.
- Region/model availability and data residency constraints can eliminate otherwise good model choices.
9. High-Yield Mnemonics
4-Tier: Rule → Service → Train → FM.
Adaptation order: Prompt → RAG → Fine-tune.
RAG for facts, fine-tuning for behavior.
Non-functional drivers: Interpretability, latency, cost — not accuracy alone.