Skill 2.2.8: Apply customization techniques for AI solutions (for example, task-specific prompt engineering, fine-tuning)
Task 2.2: ML Model and Foundation Model (FM) Development - Train, fine-tune, and customize models for ML and AI solutions
Customization Decision Matrix
| Technique | What changes | Best for | Data needed | AWS service | Avoid when |
|---|---|---|---|---|---|
| Prompt engineering | Instructions, format, examples | Simple tasks, tone, output structure | Few examples | Bedrock Prompt Management | Complex domain adaptation or large knowledge injection |
| RAG | External retrieval context | Dynamic/proprietary knowledge, citations | Document corpus | Bedrock Knowledge Bases | Answer requires new reasoning/style not in docs |
| Supervised fine-tuning | Model weights | Style, format, jargon, task behavior | Labeled prompt-completion pairs | Bedrock, SageMaker JumpStart | Injecting facts; use RAG instead |
| Reinforcement fine-tuning | Model weights via reward | Alignment, preference optimization | Prompts + reward function | Bedrock | No clear reward signal |
| Distillation | Student model weights | Cost/latency reduction | Prompts + teacher responses | Bedrock | Accuracy loss unacceptable |
| Continued pre-training | Model weights on domain text | Specialized domains | Large unlabeled corpus | Bedrock, SageMaker | Small dataset |
| HPO / AMT | Hyperparameters | Improving model performance | Training/validation data | SageMaker AMT | Compute budget limited |
AWS Services & Key Configs
| Service | Customization role | Key concepts / gotchas |
|---|---|---|
| Amazon Bedrock | FM customization: fine-tuning, RFT, distillation | Custom models require Provisioned Throughput for inference. Training data in S3, IAM role, KMS. Prompt Management for versioned prompts. Knowledge Bases for RAG (chunking, embeddings, vector store). |
| SageMaker AI | Training jobs: built-in, script mode, BYOC | Script mode uses AWS-maintained DLCs. BYOC is heaviest. AMT: objective metric, max jobs, early stopping, warm start, Hyperband/Bayesian. Distributed: data vs model parallel. JumpStart for pre-trained fine-tuning. Multi-model endpoints for cost consolidation. |
| CloudWatch | Metrics/logs | Loss curves, training progress, alarms. |
Prompt Engineering & RAG Optimization
- Prompt engineering: explicit instructions, output format, few-shot examples, chain-of-thought. Version and track prompts (Bedrock Prompt Management). Use variables for reuse.
- RAG: optimize retrieval first. If retrieved passages lack answer, generator cannot fix. Fix embedding model, chunking, vector store, hybrid search, reranking before fine-tuning.
- Embedding models: domain alignment, dimensionality, cost, latency. Choose model matching content type and language.
- Chunking: too large dilutes signal; too small loses context. Use overlap. Semantic/hierarchical chunking for structured documents.
Fine-Tuning & Customization
- Supervised fine-tuning: labeled input-output pairs. Hyperparameters: epochs, batch size, learning rate. Monitor validation loss. Early stopping prevents overfitting.
- Reinforcement fine-tuning: reward function scores responses. Use for alignment and preference optimization.
- Distillation: teacher generates responses; train smaller student. Reduces cost/latency.
- PEFT/LoRA: parameter-efficient fine-tuning reduces compute, memory, and catastrophic forgetting.
- Catastrophic forgetting: narrow fine-tuning degrades general ability. Mitigate by blending general data, lower learning rate, PEFT, evaluate on general benchmarks.
Training & Tuning on SageMaker
- Script mode: entry point + requirements; AWS-maintained DLC. BYOC for custom dependencies.
- Built-in algorithms: XGBoost, Linear Learner, etc. for tabular; no custom training loop.
- AMT: search ranges, objective metric, direction (Minimize/Maximize), max jobs, max parallel jobs, early stopping, warm start. Hyperband for efficient search.
- Distributed training: data parallelism splits batch across replicas; model parallelism splits model across devices. Communication overhead can negate gains.
- Training time reduction: early stopping, distributed training, mixed precision, checkpointing, efficient data input (Pipe mode, FSx for Lustre).
Generalization & Model Combination
- Overfitting: high train, low validation. Fix: more data, regularization, dropout, early stopping, reduce capacity.
- Underfitting: low train and validation. Fix: more complex model, more features, longer training.
- Ensembling: bagging, boosting, stacking. Improves performance, increases cost/complexity.
- Multi-model endpoints: host many models on one endpoint, shared serving container, cost-efficient, cold start. Not for strict latency.
Security, Compliance & Networking
- IAM: SageMaker execution role, Bedrock service role. Least privilege for S3, KMS, CloudWatch, ECR.
- KMS: encrypt S3 training data, model artifacts, Bedrock custom models, SageMaker volumes. CMK for compliance.
- VPC: SageMaker training in VPC, network isolation, no internet. VPC endpoints (PrivateLink) for S3, Bedrock, SageMaker, CloudWatch.
- Data privacy: Bedrock doesn't store customer data or use it to train base models. Fine-tuned models are private, encrypted. Data residency via region.
- Compliance: CloudTrail for API audit, CloudWatch Logs for training logs, Model Monitor for drift.
Exam Gotchas
Bedrock custom models require Provisioned Throughput for inference; on-demand not supported. AMT objective metric direction must match (Minimize for loss, Maximize for accuracy/F1). RAG retrieval failures: fix embedding/chunking/ranking before fine-tuning generator. Catastrophic forgetting: blend general data or use PEFT when fine-tuning narrow domains. Data parallelism splits batch; model parallelism splits model. Use data parallel when model fits one device. Script mode uses AWS-maintained DLCs; BYOC is heaviest to maintain. Multi-model endpoints share memory/container; not for strict latency. KMS CMK required for encryption; IAM roles need kms:Decrypt/GenerateDataKey. VPC endpoints needed for private S3/Bedrock/SageMaker access.
Anti-Patterns
- Fine-tuning to inject facts → use RAG.
- Prompt engineering for complex domain adaptation → fine-tune.
- Ignoring retrieval quality → fix chunking/embedding first.
- Tuning without objective metric → set metric and direction.
- No early stopping → overfitting and wasted compute.
- Narrow fine-tuning without general data → catastrophic forgetting.
- Unversioned prompts → use Bedrock Prompt Management.
- Public internet for training data → use VPC endpoints.