Skip to content

Domain 2: ML Model and Foundation Model (FM) Development

Task 2.1: Choose appropriate modeling approaches

Trigger Keywords & Scenario Constraints Lock-In Architecture Immediate Rejection / Trap to Avoid Explanation
"No ML expertise / fastest time-to-value" + "standard business task" (OCR, sentiment, faces, speech) AWS AI managed service (Textract, Comprehend, Rekognition, Transcribe, Translate, Polly, Lex, Personalize) SageMaker custom training or Bedrock FM (overkill, higher cost/ops). Prefer fully managed AI APIs when the task is a well-defined AWS AI capability and customization depth is low.
"Extract text / forms / tables / queries from documents / IDs / invoices" Amazon Textract (AnalyzeDocument, AnalyzeID, AnalyzeExpense, Queries) Amazon Rekognition (images/faces, not structured docs) or custom CV OCR. Textract is the document-intelligence lock-in; Rekognition is pixels/faces/moderation.
"Image/video labels, faces, celebrities, PPE, content moderation" Amazon Rekognition SageMaker image classification (unless custom classes/architecture required). Use Rekognition for pre-trained CV APIs; move to Custom Labels / SageMaker only for proprietary classes.
"Custom image classes, few hundred labeled images, no ML team" Amazon Rekognition Custom Labels Full SageMaker object-detection training from scratch. Transfer-learning CV with minimal data/ops; SageMaker wins only if you need custom architecture, multi-GPU, or exotic losses.
"NLP sentiment, entities, PII, key phrases, language, topic" + "no training data / API" Amazon Comprehend (built-in) Bedrock LLM for simple extractive NLP (cost/latency). Built-in Comprehend APIs beat FMs for classic extractive NLP at scale.
"Domain-specific text classification / entity recognition" + "labeled docs, not generative" Amazon Comprehend Custom (classifier / entity) Fine-tune a Bedrock FM or BlazingText unless generation/reasoning is required. Custom Comprehend is the cheap, low-latency classifier/NER path; FMs are for generation and reasoning.
"PHI / medical entities / ICD / RxNA" Amazon Comprehend Medical Generic Comprehend or a general FM without medical NER. Medical/PHI entity extraction is a dedicated service; do not substitute generic NLP.
"Speech → text / call transcripts / speaker diarization / custom vocab" Amazon Transcribe (+ Custom Vocabulary / Custom Language Model) Amazon Polly (TTS, opposite direction) or generic FM audio unless multimodal FM is specified. Transcribe is STT; Polly is TTS; Translate is text-to-text.
"Text → speech / neural voice / SSML" Amazon Polly Amazon Transcribe. Opposite of Transcribe; SSML/neural voices are Polly triggers.
"Language translation + custom terminology / brand names" Amazon Translate (Custom Terminology / Active Custom Translation) Seq2Seq from scratch or a general FM unless free-form style/tone is the requirement. Dedicated MT service; custom terminology protects do-not-translate brand tokens.
"Chatbot / IVR intents & slots, deterministic dialog" Amazon Lex Bedrock Agents (tool-using reasoning agents) or a raw FM chat endpoint. Lex = intent/slot bots; Bedrock Agents = multi-step tool use / reasoning.
"Real-time / personalized recommendations, user-item interactions" Amazon Personalize Factorization Machines or custom recsys unless you must own the algorithm. Personalize is the managed recsys lock-in (HRNN, similar-items, ranking recipes).
"Tabular structured data, classification or regression" SageMaker XGBoost (built-in) Linear Learner (only if strictly linear / massive scale baseline) or an LLM. XGBoost is the default tabular lock-in; handles missing values, mixed features, non-linearity.
"Linear model, millions–billions of rows, need a scalable linear baseline" SageMaker Linear Learner XGBoost (heavier) or Factorization Machines (interaction/sparse). Linear Learner is the large-scale linear class/reg algorithm; recordIO-protobuf preferred.
"Sparse high-dimensional, CTR / click prediction, feature interactions" SageMaker Factorization Machines XGBoost (dense tabular) or Object2Vec (pair embeddings). FM is the sparse interaction / CTR lock-in.
"Unsupervised clustering" SageMaker K-Means K-NN (supervised/similarity) or PCA (dim reduction, not clusters). K-Means = clusters; PCA = axes; k-NN = neighbors.
"Dimensionality reduction / decorrelate features before training" SageMaker PCA LDA/NTM (topics, not general dim reduction). PCA compresses numeric features; topic models are for bag-of-words corpora.
"Unsupervised anomaly detection on metrics/events" SageMaker Random Cut Forest (RCF) IP Insights (IP-entity) or supervised XGBoost unless labels exist. RCF is the unlabeled anomaly lock-in; use supervised models only with labeled fraud.
"Suspicious IPs / account-takeover / IP–user association" SageMaker IP Insights Random Cut Forest (generic anomalies). IP Insights learns IP-entity affinity; RCF does not understand IP identity.
"Many related time series, demand/traffic forecast" SageMaker DeepAR XGBoost-on-lags (weak for related series) or a generic FM. DeepAR is the built-in probabilistic multi-series forecaster (JSON Lines).
"Word embeddings or fast text classification" SageMaker BlazingText (Word2Vec / supervised) Object2Vec (pair similarity) or Bedrock embeddings (semantic RAG). BlazingText = Word2Vec/text class; use Titan/Cohere embeddings for RAG vectors.
"Embeddings of pairs (user-item, title-description) / similarity" SageMaker Object2Vec BlazingText or Factorization Machines. Pair-encoder embeddings; not a CTR FM and not Word2Vec.
"Translation / summarization as a trained seq2seq model (not an FM)" SageMaker Sequence-to-Sequence Amazon Translate (managed MT) or Bedrock (if you want an FM, not a trained seq2seq). Built-in seq2seq when the exam wants a SageMaker algorithm, not a managed AI API.
"Topic modeling, large corpus" SageMaker Neural Topic Model (NTM) LDA (small/medium corpora) or NTM vs LDA swapped. NTM scales; LDA is the smaller-corpus topic model.
"Topic modeling, smaller dataset, classical LDA" SageMaker LDA NTM (neural, large data). Exam often contrasts LDA (smaller) vs NTM (larger).
"Pixel-level masks / scene parsing" SageMaker Semantic Segmentation Object Detection (boxes) or Image Classification (one label/image). Classification ≠ boxes ≠ masks. Match the annotation type.
"Bounding boxes / locate multiple objects" SageMaker Object Detection Image Classification or Semantic Segmentation. Boxes → Object Detection; image-level label → Image Classification.
"Single label per image, transfer learning OK" SageMaker Image Classification (built-in / JumpStart) Object Detection or Rekognition (if custom model training is required). Image-level labels; Rekognition Custom Labels if “no ML team + few images”.
"k-NN similarity search / non-parametric classifier" SageMaker k-Nearest Neighbors K-Means (clustering) or OpenSearch k-NN (RAG retrieval store). Algorithm vs vector DB: k-NN estimator trains/indexes for ML; OpenSearch serves RAG.
"Business analyst, no code, tabular AutoML" Amazon SageMaker Canvas SageMaker Autopilot (code/API AutoML) or Studio notebooks. Canvas = no-code UI; Autopilot = AutoML API/jobs.
"AutoML for tabular: automatic FE + model selection + ensembling" SageMaker Autopilot JumpStart (pretrained/FM) or built-in XGBoost with manual HPO. Autopilot is the AutoML lock-in; JumpStart is pretrained/FM catalog.
"Pretrained CV/NLP/FM catalog, one-click deploy or fine-tune, own the endpoint" SageMaker JumpStart Amazon Bedrock (fully managed FM, no instance ownership). JumpStart → you own SageMaker instances/VPC; Bedrock → API, pay-per-token.
"Need VPC isolation / instance-type control / model not in Bedrock" for an FM SageMaker JumpStart / SageMaker training + endpoint Amazon Bedrock on-demand (no VPC-hosted weights, limited instance control). Infra ownership and network isolation push you off Bedrock onto SageMaker.
"No infrastructure, pay-per-token, FM via API, rapid GenAI" Amazon Bedrock (on-demand / Converse) JumpStart endpoint (idle instance cost) or custom training. Bedrock is the managed FM platform; do not stand up GPUs for a simple API workload.
"Choose FM by task: reasoning / long context / coding" Amazon Bedrock – select a high-reasoning chat FM (e.g. Anthropic Claude family) Small Titan/Llama if the stem stresses complex reasoning, or embeddings models for chat. Match modality + reasoning depth; embeddings models are not chat FMs.
"Choose FM: AWS-native, embeddings, or cost-efficient generation" Amazon Titan (Text / Embeddings / Image as specified) Claude-class models for cheap embeddings or simple classify. Titan covers AWS-native text, embeddings, and image; don’t pick a chat FM for vectors.
"Need embeddings for RAG / semantic search" Bedrock embedding model (Titan Embeddings / Cohere Embed) Chat FM InvokeModel as a vectorizer, or BlazingText Word2Vec for RAG. RAG retrieval needs a dedicated embedding model, not Word2Vec and not a chat completion.
"Image generation from text" Bedrock image FM (Titan Image / Stable Diffusion via Bedrock) Amazon Rekognition (analysis, not generation) or SageMaker unless self-host is required. Generation ≠ recognition.
"Ground answers in private/changing enterprise knowledge, citations" RAG (Bedrock Knowledge Bases or retrieve-then-generate) Fine-tuning (stale parametric knowledge, expensive, no citations). RAG for knowledge; FT for style/behavior; prompt-only cannot add a private corpus.
"Model lacks domain facts / corpus updates often / must cite sources" RAG architecture Continued pre-training or full fine-tune as first choice. Updating a KB is cheaper and fresher than retraining weights for factual recall.
"Need consistent tone, output schema, or task behavior; knowledge already in-model" Fine-tuning / PEFT (Bedrock customization or JumpStart SFT) RAG (doesn’t reliably change style) or prompt-only if examples already failed. FT changes weights/behavior; RAG injects facts.
"Acceptable quality already; only need format/tone/persona tweaks; no training budget" Prompt engineering (system prompt, few-shot, Prompt Management) Fine-tune or RAG. Always try prompting first; it has zero training cost and instant iteration.
"Multi-step tool use, API calling, planning, agentic workflow" as the modeling approach Bedrock Agents / agentic FM pattern Single-shot RAG or a Lex intent bot. Agents orchestrate tools/reasoning; RAG only retrieves; Lex is dialog-state, not tool-planning.
"Fully managed RAG: S3/SharePoint/Confluence → embeddings → retrieve+generate" Amazon Bedrock Knowledge Bases DIY embeddings + custom OpenSearch unless custom vector-DB control is required. Knowledge Bases is the managed RAG lock-in; DIY only when the stem demands a specific vector store/tuning.
"Custom vector DB, own indexing/ANN, hybrid search control" Customer-managed RAG (embed with Bedrock + OpenSearch / RDS pgvector / Aurora pgvector) Bedrock Managed KB if the stem insists on self-managed index/schema. Managed KB hides the index; customer-managed is the control/lock-in for custom ANN.
"RAG + keyword AND semantic, product SKUs / error codes" Hybrid search (lexical + vector) Pure k-NN vector search only. Exact tokens (SKUs, IDs) fail pure embedding search; hybrid is the trap-avoider.
"RAG quality low because top-k chunks are noisy" Reranking stage (Bedrock reranker / Cohere Rerank) after retrieve Increase k only, or fine-tune the FM first. Retrieve broadly, then rerank; don’t jump to FT to fix retrieval ranking.
"Long documents, need both context and precise chunks" Parent-child / hierarchical chunking RAG Naive fixed-size chunks only, or stuffing the full doc into the prompt. Child chunks retrieve; parent chunks provide generation context.
"Graph relationships / multi-hop entities in RAG" Graph RAG (Amazon Neptune + vectors, or KB graph patterns) Flat vector RAG only. Multi-hop entity relations are a graph pattern, not cosine-k-NN.
"Tradeoff: custom unique architecture / proprietary loss" SageMaker custom training (script mode / framework estimator) Bedrock FT, Autopilot, or built-in algorithm. Unique nets/losses require script mode (or BYOC); managed FMs cannot train arbitrary architectures.
"Tradeoff: pretrained weights, modest data, standard CV/NLP" Transfer learning via JumpStart / built-in image/text algorithms Train from scratch (cost/time) or Bedrock if you need a hosted FM API. Transfer learning beats from-scratch whenever a pretrained backbone exists.
"Train from scratch" only when No suitable pretrained model + novel architecture/data modality JumpStart/Bedrock/built-in as first answer. From-scratch is almost always the distractor at Associate level.
"Assess FM quality vs latency vs cost before picking a model" Amazon Bedrock model evaluation / playground + token-pricing comparison Immediately fine-tune the largest model. Pick the smallest FM that meets quality; eval before customization.
"Need higher quality than a small FM but production latency/cost of a large FM is too high" Model distillation (Bedrock Model Distillation: teacher → student) Full fine-tune of the teacher for production serving. Distillation transfers teacher quality into a cheaper/faster student.
"Guaranteed throughput / consistent FM latency in prod" (selection implication) Bedrock Provisioned Throughput (model units) On-demand Bedrock (throttling/variable) or a always-on JumpStart GPU if you don’t need to own infra. PT buys reserved FM capacity; on-demand is pay-per-token with no reservation.
"Variable / spiky GenAI traffic, minimize idle cost" Bedrock on-demand inference Provisioned Throughput (idle MU cost) or SageMaker always-on endpoint. On-demand wins for spiky/low-duty-cycle FM inference.
"Offline scoring of millions of prompts, not user-facing" Bedrock Batch Inference Real-time InvokeModel loops or a SageMaker real-time endpoint. Batch is the cost/throughput path for offline FM eval or bulk generation.
"ML performance vs training time vs cost" + "interruptible, checkpointable" SageMaker Managed Spot Training + checkpoints On-demand training instances as default. Spot cuts training cost (~70–90%) when jobs can resume from checkpoints.
"Model fits on one GPU; dataset is huge; need faster training" Distributed data parallel (SageMaker DDP / SMDDP / FSDP data shard) Model parallel (for models that don’t fit). Data parallel shards data; model parallel shards weights.
"Model too large to fit on a single GPU/instance" SageMaker Model Parallel / FSDP / tensor-parallel / pipeline-parallel Data parallel only, or a bigger single instance that still OOM. OOM on one GPU → shard the model, not just the batch.
"Large-scale FM pre-training on resilient GPU/Trainium clusters" SageMaker HyperPod A single SageMaker training job without cluster resiliency. HyperPod is the multi-node FM pretrain/resiliency lock-in.
"Interpretability required, tabular, regulated" Linear Learner / shallow XGBoost + SHAP Large FM or deep net as first model. Exam: interpretability → linear/tree + SHAP, not a black-box FM.
"Latency budget in milliseconds, structured input" Classical ML (XGBoost/Linear Learner) on CPU Bedrock FM or large Transformer. Don’t use an LLM as a classifier when XGBoost meets the SLA.
"Need domain language/style; unlabeled domain corpus is large" Continued pre-training / domain-adaptive pre-training Instruction SFT (needs labeled pairs) or RAG (doesn’t rewrite the base distribution). CPT uses unlabeled domain text to adapt token statistics; SFT needs labels.
"Labeled prompt–response pairs for a task" Supervised fine-tuning (SFT / instruction tuning) DPO (needs preference pairs) or CPT (unlabeled). Match the data shape: pairs → SFT; chosen/rejected → DPO; unlabeled corpus → CPT.
"Human preference: chosen vs rejected responses" DPO / preference optimization (Bedrock/SageMaker customization) SFT on the chosen text only, or RLHF from scratch as first answer. Preference pairs lock DPO; RLHF is heavier (reward model + RL).
"Reward signals / verifiable correctness (code, math)" Reinforcement fine-tuning (RFT / RLVR) SFT only, or DPO without a reward. Verifiable rewards → RFT; pairwise prefs → DPO; labels → SFT.
"Keep base knowledge, cheap adapters, small labeled set" LoRA / PEFT Full fine-tuning (FFT) of all weights. LoRA trains adapters; FFT is costlier and more prone to catastrophic forgetting.
"Maximum task adaptation, large labeled set, can afford GPUs" Full fine-tuning (FFT) LoRA if the stem emphasizes deepest weight updates. FFT updates all weights; use only when data/compute justify it.
"Healthcare FHIR clinical data → ML" AWS HealthLake + SageMaker/Comprehend Medical Generic S3+Comprehend as the clinical store. HealthLake is the in-scope FHIR/clinical store feeding ML.

Task 2.2: Train, fine-tune, and customize

Trigger Keywords & Scenario Constraints Lock-In Architecture Immediate Rejection / Trap to Avoid Explanation
"No training code, just algorithm + hyperparameters + S3 channels" SageMaker built-in algorithm Script mode or BYOC. Built-in = container + HP; you do not bring train.py.
"Bring your own PyTorch/TF/SKLearn/HF script, use AWS framework container" SageMaker script mode (entry_point=train.py, framework estimator) BYOC Docker (unnecessary) or built-in algorithm (no custom script). Script mode is the default “my training code + managed container” answer.
"Custom CUDA/ops, unsupported framework, fully custom image" BYOC training image on SageMaker Script mode (won’t fit unsupported runtimes). BYOC only when framework containers cannot run the code.
"Hugging Face Transformers fine-tune on SageMaker" Hugging Face estimator / JumpStart NLP-FM Generic SKLearn estimator or Bedrock if the stem requires SageMaker training jobs. HF estimator is the script-mode path for Transformers on SageMaker.
"Need simplicity + performance with a supported framework" SageMaker script mode on official TF/PyTorch/MXNet/HF/SKLearn containers Hand-rolled EC2 training. Exam wants SageMaker training jobs, not DIY EC2.
"Search hyperparameters automatically" SageMaker Automatic Model Tuning (AMT) / HyperParameterTuner Manual grid in a notebook, or Autopilot (that’s full AutoML, not HP search on your estimator). AMT tunes your training job’s HPs; Autopilot builds the whole model.
"HPO, expensive jobs, sequential smart search" Bayesian AMT strategy Random (wastes sequential budget) or Hyperband (needs epoch-level metrics). Bayesian is the default sequential strategy.
"HPO, highly parallel, large search space, cheap trials" Random search AMT Bayesian (less parallel-efficient). Random parallelizes better when trials are cheap.
"HPO + iterative algorithm that reports metrics per epoch; reallocate budget" Hyperband AMT Bayesian/Random; and do not also set AMT early stopping AUTO with Hyperband. Hyperband is multi-fidelity; its internal early-stop conflicts with TrainingJobEarlyStoppingType=AUTO.
"Stop bad HPO trials early to save cost / avoid overfit" AMT early stopping AUTO (algorithm must emit per-epoch objective) Hyperband + AUTO together; or early stop with an algorithm that only emits a final metric. Early stop needs epoch metrics (XGBoost, Linear Learner objective_loss, image class, etc.).
"Continue HPO from a previous tuning job" Warm-start hyperparameter tuning Brand-new tuner from scratch (loses prior trials). Warm start reuses prior AMT knowledge.
"Custom algorithm in AMT" Define MetricDefinitions regex so SageMaker can scrape the objective from logs Expect built-in metric names to exist magically. Only built-ins auto-emit named metrics; custom scripts must print parseable metrics.
"Reduce training time: stop when validation stops improving" Early stopping (training job or AMT) More epochs “to be safe”. Early stopping is both a time-saver and an overfit control.
"Reduce training time: reuse initialized instances across short jobs" SageMaker training warm pools Cold-start a new cluster every experiment. Warm pools cut start-up time for iterative experimentation.
"Reduce training time: compile TF/PyTorch graphs" SageMaker Training Compiler Model parallel (wrong problem) or just a larger instance. Compiler speeds supported TF/PT workloads; it is not a distribution strategy.
"Reduce training time: stream S3, avoid full download" Pipe mode (or FastFile if POSIX/random access needed) File mode (full download) when I/O is the bottleneck. Pipe = stream once; FastFile = POSIX-like S3; File = download-all.
"POSIX-like access to S3 during training, no full copy" SageMaker FastFile mode Pipe mode (sequential stream, poor random access). FastFile is the “looks like a filesystem, backed by S3” training I/O mode.
"XGBoost/image class: continue from a prior model artifact" Incremental training (use_saved_model / checkpoint channel) Full retrain from scratch. Incremental training is the built-in warm-start for supported algorithms.
"Spot interruptions must not lose progress" Checkpointing to S3 + Managed Spot Training Spot without checkpoints. Spot + no checkpoints is the classic trap.
"CPU preprocessing + GPU training in one job" SageMaker heterogeneous clusters All-GPU cluster for heavy CPU FE, or a separate Glue job when the stem wants one training job. Heterogeneous clusters mix instance types in a single training job.
"Mixed precision / FP16 to speed GPU training" AMP / FP16 in script + GPU instance Drop to CPU, or change algorithm. Precision reduction speeds training; it is not the same as post-train quantization-for-size (removed as a C02 2.2 skill).
"Overfitting: train acc high, val acc low / val loss rises" Regularization (dropout, L1/L2, max_depth↓, min_child_weight↑) + early stopping + more data/augmentation Train more epochs or add parameters. Overfit → simplify/regularize/stop; do not train longer.
"Underfitting: train and val both poor" Increase capacity / epochs, reduce regularization, better features, higher model complexity More dropout or stronger L2. Underfit is the opposite of overfit; reducing capacity makes it worse.
"XGBoost overfit" Lower max_depth, raise min_child_weight/gamma, add subsample/colsample_bytree, early stop on validation:rmse/auc Raise eta and add epochs. Tree depth/regularization are the XGBoost overfit knobs.
"Vanishing/exploding gradients, dead ReLU, loss not decreasing" during training SageMaker Debugger built-in rules + LR schedule / gradient clipping / init fix SageMaker Model Monitor (production) or Clarify (bias/SHAP). Debugger watches tensors while training; Monitor is post-deploy drift.
"Catastrophic forgetting after FM fine-tune" (MMLU/general quality collapses) LoRA not FFT, lower LR, fewer epochs, mix replay/base data, eval a retention benchmark More epochs / higher LR / full FT. Forgetting is over-adaptation; PEFT + conservative HPs preserve the base.
"Combine models to raise accuracy" Ensemble — bagging (variance), boosting (bias), stacking (meta-learner) A single bigger FM as the only lever. Autopilot often ensembles; XGBoost is boosting.
"Reduce variance of unstable learners" Bagging / model averaging Boosting (that reduces bias, can increase variance). Bagging vs boosting is a common trap pair.
"Reduce bias of weak learners, tabular" Boosting (XGBoost / LightGBM / CatBoost) Bagging/RF-only when the stem says high bias. Boosting is the high-bias tabular answer.
"Meta-learner on top of several base models" Stacking Simple averaging if the stem wants a second-level model. Stacking = another model on base predictions.
"Epoch" too high on FM SFT Reduce epochs (often 1–3); watch val + retention metrics “More epochs always better”. Extra SFT epochs are a top forgetting/overfit trigger.
"Learning rate too high" (loss NaN / diverges / forgets) Lower LR, warmup, gradient clipping Raise batch size only, or add epochs. High LR is the divergence/forgetting knob.
"Learning rate too low" (no convergence, underfit) Raise LR or add warmup / cosine schedule More regularization. Low LR looks like underfitting.
"Batch size ↑" (smoother grads, may need LR scaling, can hurt generalization) Scale LR with batch (linear/sqrt); watch val Blindly max out batch to “go faster” as quality-neutral. Large batch is a speed/quality tradeoff, not a free lunch.
"Steps vs epochs" steps = epochs × (N / batch); set max_steps or epochs, not both blindly Treat steps and epochs as independent magic numbers. Know the relationship; don’t double-train.
"LoRA r / alpha / dropout" PEFT LoRA hyperparameters (lora_r, lora_alpha typically \(2\)–\(4\times r\), lora_dropout) FFT HPs only (full rank updates). LoRA HPs are the PEFT lock-in; they do not exist for classic XGBoost.
"instruction_tuned = True" Instruction / SFT-style JumpStart fine-tune Continued pre-training on raw corpus. Instruction flag expects prompt–response data.
"FSDP during FT" enable_fsdp=True / SageMaker sharded data parallel Model-parallel pipeline when the stem is FSDP. FSDP shards params/grads/opt states across GPUs.
"int8 training / QLoRA-style" Quantized PEFT load (int8_quantization) Full FP32 FFT on the largest instance. Quantized adapters cut GPU memory for FT.
"Prompt-only customization, versioned templates, A/B prompts" Amazon Bedrock Prompt Management + prompt engineering Fine-tuning a custom model for a wording change. Prompt Management is the versioned-prompt lock-in (also used in eval).
"Few-shot" In-prompt examples (no weight update) Fine-tune because “examples exist”. Few-shot is still prompt engineering if examples live in the prompt.
"Chain-of-thought / multi-step reasoning without tools" CoT prompting (and low-to-mid temperature) Agents (if no tools) or FT as first step. CoT is a prompt pattern; Agents add tools.
"Deterministic extraction / JSON / classification with an FM" Temperature \(= 0\) (and constrained/schema prompting) High temperature “for quality”. Temperature \(> 0\) is for diversity/creativity, not extraction.
"Creative generation" Higher temperature / top-p Temperature \(0\). Opposite of extraction.
"RAG retrieval quality poor: embedding mismatch" Swap/eval embedding models; embed queries and docs with the same model Fine-tune the generator FM first. Retrieval bugs are embedding/chunk/search problems, not generator FT.
"RAG: chunks too big (dilute) or too small (lose context)" Retune chunk size/overlap; consider hierarchical/parent-child Dump whole PDFs into the context window as the first fix. Chunking is the first retrieval lever.
"Need metadata filters (tenant, date, product) at retrieve time" Metadata filtering on the vector index / KB Prompt the FM to ignore wrong tenants. Filtering is retrieval-time, not a prompt hope.
"Query ≠ document wording (vocabulary gap)" Query rewriting / HyDE / multi-query retrieval Larger k only. Rewrite the query into document-like text before embed.
"Bedrock FT/distilled custom model for inference" Custom model + Provisioned Throughput or custom-model on-demand deployment (where supported) Invoke the base model ID after FT and expect new behavior. Custom models need a dedicated deployment/PT ARN; the base ID will not serve your weights.
"Teacher FM → smaller student, use invocation logs or prompt set" Amazon Bedrock Model Distillation Manual synthetic-data pipeline on EC2, or PT on the teacher. Distillation is the managed teacher-student workflow; student still needs PT/deploy to serve.
"Prevent forgetting while teaching a new task" PEFT/LoRA + replay mix + low LR + retention eval FFT with high LR on task-only data. Same as catastrophic-forgetting control, applied as a customization strategy.
"Local debug of a training script before paying for a cluster" SageMaker local mode Launch a multi-GPU training job to print a shape error. Local mode is the cheap script-debug path.
"Reproducible training" Fix random seeds, log params/data versions in MLflow, version datasets in S3 “Same instance type” as sufficient reproducibility. Seeds + data/model lineage, not just hardware.

Task 2.3: Analyze and evaluate performance

Trigger Keywords & Scenario Constraints Lock-In Architecture Immediate Rejection / Trap to Avoid Explanation
"Reproducible experiments, params, metrics, artifacts, lineage" MLflow on SageMaker AI Ad-hoc CloudWatch logs only, or Model Registry (that’s versioning for deploy, Domain 3). MLflow is the C02 experiment-tracking lock-in named in the guide.
"Compare prompt versions, not model weights" Amazon Bedrock Prompt Management + eval jobs Model Registry for prompts, or a new fine-tune to test wording. Prompts are first-class versioned artifacts in C02.
"Automatic + human FM evaluation, toxicity/accuracy/helpfulness" Amazon Bedrock Evaluations (automatic, human, LLM-as-judge) SageMaker Debugger or CloudWatch CPU metrics. Bedrock Evaluations is the FM eval suite; Debugger is training tensors.
"Create a performance baseline, then detect drift" SageMaker Model Monitor baselining (data quality, model quality, bias, feature attribution) Debugger (training-time) or a one-off notebook metric. Baselines + drift = Model Monitor; the skill sits in 2.3 even though monitors run in prod.
"Data distribution change will hurt the model" (pre-prod design) Model Monitor data-quality schedule + baseline statistics Retrain on a fixed calendar with no drift signal. Drift detection is the trigger for retrain, not a cron by itself.
"Compare a candidate model to prod without affecting users" SageMaker shadow variant / shadow testing A/B production variants (users do see B) or batch transform only. Shadow copies live traffic; users stay on the production variant.
"A/B test two live variants with user traffic split" Production variants (traffic-shadow is the wrong tool) Shadow testing (no user impact, cannot measure UX). 2.3.3 is specifically shadow; don’t confuse with Domain 4 A/B.
"Explain tabular predictions / global+local feature attribution" SHAP (SageMaker Clarify SHAP / Kernel SHAP baseline) Debugger feature-importance-only, or Bedrock Evaluations (FM-only). SHAP is the explain-outputs lock-in for classical ML.
"Bias metrics on a trained classifier (DPPL, DI, etc.)" Pre/post-training bias analysis (Clarify metrics or equivalent pandas/sklearn in-pipeline + MLflow) Accuracy-only reporting. Fairness is a 2.3 eval axis, not a training HP.
"FM/LLM evaluation, not tabular SHAP" Amazon Bedrock Evaluations Clarify FMEval as the C02-first answer (guide moved FM eval to Bedrock). Tabular explainability ≠ FM eval.
"Debug why training does not converge" SageMaker Debugger rules (loss_not_decreasing, vanishing_gradient, exploding_tensor, overfit, class_imbalance, dead_relu) + LR/init/data checks Model Monitor or Bedrock Guardrails. Convergence debugging is a training-job problem.
"Imbalanced classification metric" AUC-PR / F1 / recall@precision (and scale_pos_weight) Accuracy or AUC-ROC as the only score. Accuracy is the standard imbalanced-class trap.
"Balanced binary classifier, ranking quality" AUC-ROC RMSE (regression) or BLEU (NLP). Match metric family to task.
"Regression error" RMSE / MAE / \(R^2\) Accuracy/F1. Don’t use classification metrics on continuous targets.
"Ranking / recommendations quality" NDCG / MAP / precision@k Accuracy. Personalize/recsys eval uses ranking metrics.
"Machine translation quality" BLEU (n-gram precision) ROUGE (summarization) as the primary MT metric. BLEU = translation; ROUGE = summarization.
"Summarization quality" ROUGE-1 / ROUGE-2 / ROUGE-L BLEU as the primary summarization metric. Don’t swap BLEU/ROUGE.
"Semantic similarity of generated vs reference text (paraphrase-robust)" BERTScore (and/or embedding cosine) BLEU/ROUGE only (surface n-grams miss paraphrases). BERTScore catches meaning when n-gram overlap is low.
"LM intrinsic quality, no references" Perplexity BLEU. Perplexity is not a task metric for summarization/translation.
"Extractive QA vs gold span" Exact Match + token F1 (SQuAD-style) ROUGE-L as the only QA metric. EM/F1 are the QA lock-in.
"Need an LLM to score free-form quality (helpfulness, coherence)" LLM-as-a-judge (Bedrock Evaluations judge model) BLEU-only on open-ended chat. Use judge models for subjective quality; always pair with a task metric to avoid the BLEU↔judge paradox.
"Human-in-the-loop quality assessment of generations" Bedrock human evaluation workflows and/or SageMaker Ground Truth / A2I Fully automatic BLEU as sufficient for “subjective quality”. HITL is explicit in 2.3.7; automatic metrics cannot replace it when the stem says subjective.
"Labeling / annotation jobs for eval or training sets" SageMaker Ground Truth A2I (A2I is inference-time human review) or Mechanical Turk DIY. Ground Truth = labeling; A2I = human review of predictions.
"Human review of low-confidence predictions during development/eval" Amazon A2I Ground Truth (dataset labeling). Don’t swap GT and A2I.
"Content quality, safety, bias of FM outputs" Bedrock Evaluations (toxicity, robustness, quality) ± Guardrails only if the stem is a runtime filter Debugger or Model Monitor data-quality (those are not FM content judges). 2.3.9 is eval; Guardrails is primarily Domain 4 runtime.
"RAG monitoring: is retrieval actually finding the right chunks?" Retrieval metrics: recall@k, MRR, hit rate, context precision/relevance Generator BLEU only. If retrieval is wrong, generation metrics lie.
"RAG monitoring: does the answer stick to retrieved context?" Faithfulness / groundedness / citation accuracy Helpfulness-only or BLEU vs an ungrounded reference. Hallucination vs context is a RAG-specific eval axis.
"RAG monitoring: is the answer useful to the user?" Answer relevance (plus human or LLM-judge) Context relevance only. Retrieval can be perfect while the answer is still off-task.
"End-to-end RAG quality framework" Faithfulness + answer relevance + context relevance (RAGAS-style) on Bedrock KB logs Only generator token-loss. C02 2.3.10 wants retrieval and generation assessed.
"Offline eval of prompts at scale" Bedrock batch inference + Bedrock Evaluations Live InvokeModel in a for-loop on an endpoint. Batch is the cheap offline eval path.
"Compare several FMs on the same prompt set" Bedrock model evaluation job (built-in + custom datasets) Manual playground clicking as the architecture. Evaluation jobs are the repeatable comparison mechanism.
"Detect overfit during a training job" Debugger overfit rule (val loss / train loss ratio) + early stop Model Monitor model-quality drift (that’s post-deploy). Timing distinguishes Debugger vs Monitor.
"Feature attribution drift vs training SHAP baseline" Model Monitor feature-attribution / SHAP baseline Retrain immediately without a SHAP baseline. Explainability drift needs a SHAP baseline job first.
"Classification threshold / precision-recall tradeoff" PR curve, choose threshold for the business cost of FP vs FN Default \(0.5\) threshold, or accuracy. Thresholding is an eval decision, not a training algorithm.
"Multi-class metrics" Macro vs micro F1 (macro for rare classes) Binary AUC blindly. Macro treats classes equally; micro follows support.
"Time-series forecast eval" RMSE / MAPE / quantile loss (wQL) for probabilistic DeepAR Accuracy/F1. Quantile loss matches probabilistic forecasts.
"Need a champion/challenger record of runs" MLflow experiment + registered metrics; promote only after eval gates Overwrite the same S3 prefix with no run IDs. Reproducible promotion is an eval practice, even if Registry is Domain 3.
"Shadow variant looks better on latency but worse on quality" Do not promote; require quality gates (F1/faithfulness/judge) not just p99 Promote on infrastructure metrics alone. 2.3.3 is a performance comparison of variants, not just hardware.
"FM eval dataset" Held-out prompt–response pairs + safety cases; never train split Reuse SFT training prompts as the only eval. Data leakage is an eval trap for both ML and FM.
"Catastrophic forgetting check" Retention suite (e.g. general MMLU-style) + target-task metric Target-task BLEU/judge only. Always measure what the model used to know.
"BLEU up, judge down (or the reverse)" Multi-metric dashboard; do not optimize a single proxy Single-metric HPO on BLEU for open-ended tasks. Surface n-gram metrics can anti-correlate with semantic judges.
"Regression-to-the-mean / too-good train metric, random val" Fixed val split + seed; check leakage (target in features) Tune on test, or shuffle after split incorrectly. Classic convergence “bug” is leakage, not the optimizer.
"Class imbalance showing up as a Debugger alert" Fix data (weights, resampling) then re-eval F1/AUC-PR Change instance type. Imbalance is a data/eval issue, not a compute issue.