Skill 2.1.4: Evaluate tradeoffs between custom solutions, managed services, pre-trained models, and FMs to meet business needs.

Task 2.1: ML Model and Foundation Model (FM) Development - Choose appropriate modeling approaches for ML and AI solutions.

Primary objective: Match the modeling approach to where the answer/capability resides: fixed logic → rules; solved task → managed AI; proprietary patterns → custom ML; general language → FM. Interpretability, latency, cost, and training time are selection criteria, not afterthoughts.

1. Four-Tier Selection Hierarchy

Signal	Best Approach	AWS Example	Why
Fixed, deterministic logic	Rules/code	Lambda, Step Functions	Cheaper, auditable, no ML overhead
Standard pre-defined task	Managed AI service	Textract, Rekognition, Comprehend, Transcribe	No training, fastest time-to-value
Patterns in proprietary/tabular data	Custom trained model	SageMaker AI built-in algorithms, Autopilot, JumpStart	Learns domain-specific patterns from history
General language/reasoning/generation	Foundation model	Amazon Bedrock	Capability already in model; prompt first
Fresh, citable, frequently changing knowledge	RAG	Bedrock Knowledge Bases, OpenSearch, pgvector	External index; updates without retraining

2. Tradeoff Matrix: Custom vs Managed vs Pre-trained vs FM

Dimension	Custom Model	Managed AI Service	Pre-trained Model (JumpStart)	FM (Bedrock)
Build effort	High	Low	Medium	Low–Medium
Data needed	Labeled domain data	None	Fine-tune data optional	Prompt/few-shot; fine-tune optional
Interpretability	Controllable (linear, tree, SHAP)	Limited	Varies	Low; use citations/guardrails
Latency	Tunable	Service-dependent	Tunable	Token/streaming dependent
Cost	Training + hosting	Per-API call	Hosting or endpoint	Per-token / provisioned throughput
Best for	Proprietary prediction	OCR, vision, speech, NLP	Domain-specific CV/NLP	GenAI, summarization, Q&A, agents
Main risk	Over-engineering, drift	Limited customization	Integration overhead	Hallucination, cost, latency

Performance/latency/cost trilemma: Larger FMs or higher accuracy → higher latency and cost. Custom training → higher upfront cost but lower marginal inference. Managed AI → lowest ops, per-call cost. RAG → adds retrieval latency but avoids retraining cost.

3. FM Customization Ladder: Prompt → RAG → Fine-tune → Distill

Step	When	AWS Mechanism	Key Tradeoff
Prompt engineering	First always	Bedrock prompts, few-shot	Fast, cheap, reversible; limited if knowledge changes
RAG	Need fresh, citable, external knowledge	Bedrock Knowledge Bases; custom vector store	Freshness + citations; not for whole-document synthesis
Fine-tuning (SFT)	Gap persists after prompting/RAG	Bedrock custom models; SageMaker	Behavior/terminology; needs labeled pairs; retraining for updates
Reinforcement fine-tuning	Need reward-based alignment	Bedrock RFT	Reward function required; complex
Distillation	Smaller/cheaper model mimics larger	Bedrock distillation	Uses prompts; managed teacher responses; cost/latency reduction

Exam rule: Prompt first. Fine-tune only when evaluations prove prompting/RAG cannot close the gap. Fine-tuning writes behavior into weights; RAG keeps knowledge external.

4. RAG Architecture: Managed vs Custom

Component	Managed (Bedrock Knowledge Bases)	Custom
Ingestion	S3 connectors, parsing, chunking	Lambda/Glue, Textract, custom chunking
Embeddings	Bedrock embedding model	SageMaker/OpenSearch embeddings
Vector store	Managed OpenSearch Serverless, Aurora, Neptune	OpenSearch, pgvector, Pinecone, etc.
Retrieval	Managed RetrieveAndGenerate	Custom retriever, hybrid search, reranking
Generation	Bedrock FM	Bedrock/SageMaker endpoint
Guardrails	Bedrock Guardrails	Custom filters
Orchestration	Managed	Step Functions, custom orchestration

Choose managed RAG for speed and low ops. Choose custom RAG for control over retriever, vector store, chunking, hybrid search, or compliance.

5. AWS AI Service Fit (Use Before Building)

Modality	Service	Use Case	Not For
Documents	Textract	OCR, forms, tables, invoices	Dynamic Q&A over policies
Images/Video	Rekognition	Objects, faces, moderation, labels	Custom defect detection
Text	Comprehend	Entities, key phrases, sentiment, PII	Generative summarization
Speech	Transcribe	Speech-to-text, timestamps, diarization	Audio generation
GenAI	Bedrock	Summarization, Q&A, agents, RAG	Precise tabular forecasting
ML lifecycle	SageMaker AI	Custom training, tuning, hosting, MLOps	Solved AI tasks

6. Security, Compliance & Governance

Area	Requirement
IAM	Least-privilege roles for Bedrock, SageMaker, S3, KMS; no long-lived keys
Encryption	KMS at rest for S3, EBS, vector stores, custom models; TLS in transit
Network	VPC endpoints (PrivateLink) for Bedrock, SageMaker, S3, KMS; keep traffic off public internet
Data privacy	Bedrock does not train base models on customer data; fine-tuning data stays in region/account
Guardrails	Bedrock Guardrails for PII, harmful content, denied topics; SageMaker Clarify for bias/explainability
Audit	CloudTrail, CloudWatch, Model Monitor, Bedrock model invocation logging

7. Exam Gotchas & Anti-Patterns

Gotcha: Bedrock model access is region-specific and must be enabled before use. Not all FMs available in all regions.

Gotcha: RAG does not retrain the model. Updating the vector index reflects new content in minutes; fine-tuning takes hours/days and cannot cite sources.

Gotcha: Interpretability is a first-class requirement. A simpler explainable model may beat a complex model if stakeholders must defend predictions.

Anti-pattern: Training a custom model for a solved task (OCR, speech-to-text, sentiment) — use managed AI.

Anti-pattern: Using an FM for precise tabular forecasting or fixed-logic decisions — use custom ML or rules.

Anti-pattern: Fine-tuning before prompt engineering and RAG — wastes data prep, GPU cost, and versioning overhead.

Anti-pattern: RAG for whole-document synthesis — retrieval returns passages; not ideal when entire document context is required.

Anti-pattern: Ignoring catastrophic forgetting — fine-tuning on narrow data can degrade general capability; blend broader data.