Skip to content

Skill 1.1.1: Extract data from data sources

Question 1 - Foundation Model Selection with Latency and Cost Constraints

A healthcare technology company is building a clinical documentation assistant that must summarize patient-doctor conversation transcripts into structured clinical notes. The application serves a hospital network with strict compliance requirements: inference must complete in under 500 milliseconds per request to avoid disrupting physician workflows, and the total monthly inference cost must not exceed $2,000 for an estimated 500,000 requests per month. The transcripts contain sensitive medical terminology that requires domain-specific accuracy.

Which approach BEST satisfies these constraints while maintaining clinical accuracy? (Select ONE)

  • A) Deploy a large 70B+ parameter foundation model on Amazon SageMaker real-time endpoints with GPU instances to maximize accuracy, regardless of latency.
  • B) Select a smaller, task-optimized foundation model from Amazon Bedrock that meets the latency and cost requirements, then evaluate its clinical summarization quality against benchmark tests.
  • C) Fine-tune a large foundation model on clinical transcripts using Amazon Bedrock Custom Model Import, then deploy it on high-throughput GPU clusters.
  • D) Train a custom BERT-based sequence-to-sequence model on proprietary transcripts using SageMaker Training for maximum domain control.
  • E) Use Amazon Comprehend Medical to extract medical entities, then concatenate them into a clinical note template without generative AI.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: Smaller foundation models on Bedrock (e.g., Claude Haiku, Llama 3 8B) are specifically designed for low-latency, cost-efficient inference. A 500ms latency requirement and \(2,000/month budget with 500K requests (\)0.004 per request) are achievable with task-optimized small models. The evaluation step ensures clinical accuracy is validated before deployment.

Distractor Analysis

  • A is Incorrect: Large 70B+ models deployed on GPU SageMaker endpoints typically incur latency of 2-10 seconds and cost $100K+ monthly for 500K requests—violating both constraints by orders of magnitude.
  • C is Incorrect: Fine-tuning a large model does not solve the underlying latency and cost problems; large models remain slow and expensive regardless of customization. Custom Model Import adds operational complexity.
  • D is Incorrect: Training a custom BERT model from scratch requires massive labeled clinical transcript data and significant engineering time. This violates the speed-to-market and operational overhead requirements, though it could address domain specificity.
  • E is Incorrect: Amazon Comprehend Medical extracts entities but cannot generate coherent, context-aware clinical narratives. Concatenating entities produces unusable summaries that fail the clinical accuracy requirement.

Key Takeaway: When latency and cost per request are primary constraints, select smaller task-optimized foundation models on Bedrock and validate quality through evaluation—don't default to the largest model available.

Question 2 - Interpretability vs. Model Complexity Tradeoff

A regional bank's fraud detection team needs to deploy a credit card fraud classifier. Risk compliance officers require that every flagged transaction include a clear, human-readable explanation of which features contributed most to the fraud prediction and why. The model must process 10,000 transactions per second with sub-50ms inference latency. The data science team wants to use a deep learning model for maximum accuracy.

Which modeling approach BEST balances the compliance requirements with accuracy? (Select ONE)

  • A) Deploy a deep neural network with hundreds of layers, accepting that explanations will be generated post-hoc using SHAP values after each prediction.
  • B) Train an ensemble of XGBoost decision trees, which provides built-in feature importance and SHAP-compatible explanations, while meeting latency and throughput requirements on SageMaker multi-model endpoints.
  • C) Use a transformer-based foundation model fine-tuned on transaction data, leveraging its attention weights as interpretability explanations for compliance officers.
  • D) Build a logistic regression model exclusively, sacrificing all predictive performance for maximum interpretability.
  • E) Deploy a random forest model with 1,000 trees and manually document each tree's decision path for regulators.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: XGBoost provides native feature importance metrics and integrates seamlessly with SHAP for local interpretability. On SageMaker, XGBoost models easily handle 10K requests/second at sub-50ms latency through multi-model endpoints or serverless inference, meeting all compliance and performance requirements. Tree ensembles often match neural network accuracy on tabular fraud data.

Distractor Analysis

  • A is Incorrect: Deep neural networks violate the interpretability requirement. Post-hoc SHAP explanations are computationally expensive at 10K TPS, adding significant latency and cost. DNNs also struggle with tabular data compared to gradient-boosted trees.
  • C is Incorrect: Foundation models are inappropriate for high-throughput tabular classification. Attention weights are not interpretable features for compliance officers, and LLM inference cannot meet the 50ms/10K TPS constraint.
  • D is Incorrect: While logistic regression is interpretable, it sacrifices too much predictive power on complex fraud patterns. This is an overcorrection that fails the accuracy requirement.
  • E is Incorrect: A 1,000-tree random forest introduces unnecessary complexity, longer inference times, and does not provide better interpretability than XGBoost. Manually documenting decision paths is operationally infeasible for real-time compliance.

Key Takeaway: For tabular data with strict interpretability requirements, tree-based ensembles (XGBoost, LightGBM) outperform neural networks and offer built-in explanation capabilities.

Question 3 - RAG vs. Fine-Tuning Decision

A multinational pharmaceutical company maintains a knowledge base of 50,000 internal research documents that is updated weekly with new clinical trial results. The legal team requires that every AI-generated response cite the exact source document and page number. The compliance department mandates that responses must reflect document updates within 24 hours of publication.

Which architectural pattern should the ML engineer implement? (Select ONE)

  • A) Fine-tune a foundation model on all 50,000 documents quarterly, regenerating the training set with each document update.
  • B) Implement Retrieval-Augmented Generation (RAG) using Amazon Bedrock Knowledge Bases, re-syncing the data source after each weekly content update.
  • C) Train a custom document-answering model from scratch on the corpus using SageMaker Training.
  • D) Use prompt engineering to paste relevant document excerpts manually into the model context for each user query.
  • E) Deploy a hybrid approach where the model is fine-tuned monthly and RAG retrieval is disabled for cost savings.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: RAG architecture with Bedrock Knowledge Bases satisfies all requirements: (1) each retrieved chunk carries source metadata enabling exact citations, (2) re-syncing after weekly updates ensures responses reflect new documents within minutes/hours, not weeks, and (3) the managed service eliminates custom ETL infrastructure for parsing, chunking, and embedding.

Distractor Analysis

  • A is Incorrect: Quarterly fine-tuning fails the 24-hour update requirement—new clinical trial results would be invisible to the model for up to 3 months. Fine-tuning also cannot provide page-level source citations from retrieved documents.
  • C is Incorrect: Training a model from scratch ignores the general language understanding already present in foundation models, requires massive computational resources, and cannot easily cite sources or reflect rapid updates.
  • D is Incorrect: Manually pasting document excerpts is not scalable for 50,000 documents and does not support automated retrieval of relevant contextual passages based on query semantics.
  • E is Incorrect: This hybrid unnecessarily combines both approaches when RAG alone solves the problem. Fine-tuning adds training cost, model versioning overhead, and risks catastrophic forgetting while providing no citation or update-speed benefit over pure RAG.

Key Takeaway: When requirements include source citations and rapid content update reflection (hours/days), RAG is structurally superior to fine-tuning because knowledge stays outside model weights in an updateable vector index.

Question 4 - Managed AI Service vs. Custom Model

An insurance claims processing department receives 100,000 scanned accident report PDFs per month. The documents include handwritten notes, checkboxes, tables with damage assessments, and driver's license photos. The ML team must extract structured data (policy numbers, damage estimates, dates) into a database for downstream analytics. The project deadline is 3 weeks.

Which approach delivers a working solution with the LEAST development effort and operational overhead? (Select ONE)

  • A) Fine-tune a multimodal foundation model on a labeled subset of the accident reports using Amazon Bedrock.
  • B) Train a custom computer vision model using Amazon Rekognition Custom Labels to detect and extract text from handwritten fields.
  • C) Call Amazon Textract APIs (including AnalyzeDocument for forms and tables) to extract structured text from the scanned PDFs.
  • D) Build a custom OCR pipeline using SageMaker with open-source Tesseract containers and custom post-processing scripts.
  • E) Use Amazon SageMaker Ground Truth to label 10,000 documents for a supervised text extraction model.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Answer Analysis

  • C is Correct: Amazon Textract is a fully managed AI service pre-trained on millions of documents, specifically designed to extract text, forms (key-value pairs), tables, and handwriting from scanned documents. The AnalyzeDocument API requires zero custom model training, zero data labeling, and can be integrated into production pipelines within days, satisfying the 3-week deadline.

Distractor Analysis

  • A is Incorrect: Fine-tuning a multimodal foundation model requires preparing labeled training data, designing the fine-tuning dataset, and running compute-intensive training jobs. This takes weeks and requires significant ML expertise—violating both the time and effort constraints.
  • B is Incorrect: Rekognition Custom Labels is designed for object/scene detection in images, not document text extraction. Using it for OCR is an inappropriate tool choice that would require extensive custom work.
  • D is Incorrect: Building a custom OCR pipeline with Tesseract requires handling image preprocessing, handwritten text recognition limitations, table detection, and ongoing maintenance. This is the highest-effort option.
  • E is Incorrect: Ground Truth labeling followed by model training introduces a multi-week data labeling and model development cycle, adding unnecessary complexity when Textract already solves the problem as a managed service.

Key Takeaway: For standard document extraction tasks (invoices, forms, scanned reports), always evaluate Amazon Textract before building custom OCR or fine-tuning models—the managed service likely solves it immediately.

Question 5 - Prompt Engineering vs. Fine-Tuning Threshold

A customer service AI assistant uses a foundation model on Amazon Bedrock to draft email responses. Initial evaluations showed that the model writes professional, well-structured emails but fails to consistently use the company's approved product names (e.g., it writes "WidgetX Premium" when it should write "WidgetX™ Professional Suite"). The team added explicit instructions and three few-shot examples demonstrating correct product naming, but evaluations still show a 15% error rate on product naming across test cases.

Which is the MOST appropriate next step according to AWS best practices? (Select ONE)

  • A) Continue adding more few-shot examples to the prompt until the error rate drops below 5%.
  • B) Switch to a larger foundation model in Bedrock, assuming that increased parameters will solve the naming consistency issue.
  • C) Fine-tune the foundation model on a labeled dataset of input-output pairs demonstrating correct product naming conventions.
  • D) Implement a post-processing rule that replaces any incorrect product names with the correct ones after model generation.
  • E) Use Amazon Bedrock Guardrails to enforce product naming conventions in model outputs.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Answer Analysis

  • C is Correct: The standard adaptation progression is: Prompt Engineering → Supervised Fine-Tuning → Reinforcement Fine-Tuning. After prompt instructions and few-shot examples fail to close the gap (15% error persists), fine-tuning on a labeled dataset of prompt-response pairs updates model weights to consistently adopt the target naming conventions. This is the documented AWS best-practice escalation.

Distractor Analysis

  • A is Incorrect: Adding more examples provides diminishing returns once few-shot prompting has already been tried. The 15% persistent error rate signals that the model cannot fully internalize the naming rules through in-context learning alone.
  • B is Incorrect: While a larger model may have better instruction-following capabilities, model size does not guarantee domain-specific naming consistency. Larger models also increase latency and cost without a proven benefit.
  • D is Incorrect: Post-processing rules are brittle and fail at scale—product names may appear in varied contexts (plural, possessive, abbreviated), making regex-based replacement error-prone. This is a band-aid, not a solution.
  • E is Incorrect: Bedrock Guardrails filters content for safety, PII, and topic boundaries but does not enforce stylistic or naming convention rules. Guardrails cannot perform string replacements or validate brand naming compliance.

Key Takeaway: Fine-tuning is warranted only after evaluation shows prompt engineering (instructions + few-shot examples) cannot close the quality gap—and that's exactly what a persistent 15% error rate demonstrates.

Question 6 - Training Time vs. Model Performance Optimization

A financial services firm needs to train a customer churn prediction model on 100 million historical transaction records. The model must be retrained weekly and achieve an AUC of at least 0.92 to satisfy business requirements. A gradient-boosted tree model currently achieves 0.90 AUC but takes 14 hours to train on the full dataset using CPU instances, exceeding the weekly retraining window. The team is evaluating optimization strategies.

Which combination of strategies is MOST effective for balancing training time and model performance? (Select TWO)

  • A) Switch to Amazon SageMaker's GPU-optimized XGBoost container to accelerate gradient-boosted tree training through GPU acceleration.
  • B) Reduce the training dataset to 1 million records to finish training faster, accepting potential degradation in model quality.
  • C) Use SageMaker Automatic Model Tuning (hyperparameter optimization) to find hyperparameters that achieve higher AUC within the same training time budget.
  • D) Switch to a neural network architecture, which will train faster on the same data using GPU instances.
  • E) Use Amazon SageMaker distributed training with data parallelism to split the gradient-boosted tree training across multiple GPU instances.
Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: XGBoost GPU acceleration can reduce training time by 5-10x compared to CPU training on large datasets. SageMaker's GPU-optimized containers (e.g., XGBoost with CUDA support) enable training 100M records within the weekly window.
  • C is Correct: SageMaker Automatic Model Tuning systematically searches hyperparameters (learning rate, max_depth, subsample ratio) to find configurations that deliver higher AUC. Combined with GPU acceleration, it optimizes both performance and time.

Distractor Analysis

  • B is Incorrect: Reducing dataset size from 100M to 1M records (99% reduction) risks losing critical minority-class patterns in churn prediction, likely dropping AUC below the 0.92 requirement. This trades performance for speed incorrectly.
  • D is Incorrect: Neural networks do not inherently train faster than gradient-boosted trees on tabular data. They typically require more data, more hyperparameter tuning, and longer training for comparable performance on structured datasets.
  • E is Incorrect: XGBoost does not support distributed data-parallel training across GPU instances in the same way deep learning frameworks do. Gradient-boosted trees are sequential by nature; parallelization happens through GPU acceleration and algorithmic optimizations (e.g., histogram-based splits), not multi-node data parallelism.

Key Takeaway: GPU acceleration and hyperparameter optimization are complementary strategies for meeting training time windows without sacrificing model quality—distributed training does not apply to tree-based algorithms.

Question 7 - Real-Time Inference Architecture Cost Optimization

A media streaming company has trained a recommendation model using SageMaker. The model must generate personalized recommendations for 2 million concurrent users. The current architecture uses a single real-time SageMaker endpoint with a ml.m5.xlarge instance, which incurs high costs during traffic spikes and suffers from cold-start latency when traffic is low. The team wants to optimize costs without sacrificing latency during peak hours.

Which architecture BEST optimizes costs while maintaining real-time latency requirements? (Select ONE)

  • A) Migrate to SageMaker Serverless Inference, which automatically scales to zero during idle periods and horizontally scales during traffic spikes.
  • B) Maintain the current real-time endpoint but add more instances manually during known peak hours.
  • C) Switch to SageMaker Batch Transform to process recommendations in batches during off-peak hours.
  • D) Deploy the model using Lambda functions with the model weights bundled in the Lambda package.
  • E) Use SageMaker Multi-Model Endpoints to serve multiple recommendation models on a single instance.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Answer Analysis

  • A is Correct: SageMaker Serverless Inference automatically scales compute capacity based on traffic, eliminating idle infrastructure costs during low-traffic periods. It handles burst scenarios with rapid scaling (warm pools), meeting real-time latency requirements without manual intervention. Serverless endpoints are ideal for intermittent or variable traffic patterns.

Distractor Analysis

  • B is Incorrect: Manual instance scaling during peak hours requires operational intervention, risks human error, and does not eliminate idle costs. This violates the "least operational overhead" principle.
  • C is Incorrect: Batch Transform processes predictions asynchronously and cannot serve real-time recommendations to 2 million concurrent users. This changes the architecture from online to offline, violating the real-time requirement.
  • D is Incorrect: Lambda functions have a 250MB package limit and 15-minute execution timeout—model weights for recommendation systems typically exceed these limits. Lambda is not designed for large model inference.
  • E is Incorrect: Multi-Model Endpoints optimize cost by sharing compute across multiple models on one endpoint, but do not solve the idle capacity problem for variable traffic. They still require provisioned instances that incur costs during low-traffic periods.

Key Takeaway: SageMaker Serverless Inference is the cost-optimal choice for variable traffic patterns, while provisioned real-time endpoints remain appropriate for predictable, sustained high-throughput workloads.

Question 8 - Foundation Model Evaluation for Production Deployment

An e-commerce company is evaluating two foundation models on Amazon Bedrock for a product description generation task. Model A (70B parameters) achieves 95% quality score but costs $0.012 per 1K tokens with average latency of 3 seconds. Model B (8B parameters) achieves 88% quality score but costs $0.0005 per 1K tokens with average latency of 0.4 seconds. The application needs to generate descriptions on product pages where users expect pages to load in under 2 seconds, and the projected volume is 50 million descriptions per month.

Which model should be deployed, and what additional approach can narrow the quality gap? (Select ONE)

  • A) Deploy Model A because quality is the only metric that matters; latency and cost are secondary concerns.
  • B) Deploy Model B and implement prompt engineering improvements (structured templates, example descriptions, style guides) to close the 7-point quality gap without adding significant latency or cost.
  • C) Deploy Model B and fine-tune it on 10,000 high-quality product descriptions to improve quality, accepting the additional training overhead.
  • D) Deploy neither; build a custom generation model from scratch using SageMaker.
  • E) Use a routing architecture that sends 30% of high-value product requests to Model A and 70% to Model B.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: Model B meets the latency requirement (0.4s < 2s) and cost constraints (50M × $0.0005 ≈ $2.5K/month vs. Model A's $600K/month). The 88% quality score is already strong; prompt engineering improvements (structured templates, few-shot examples, brand voice guidelines) can realistically close the 7-point gap, as prompt refinement often yields significant gains without model changes.

Distractor Analysis

  • A is Incorrect: Model A costs $600K/month (50M descriptions × $0.012 per 1K tokens × ~1K tokens/description) and misses the latency SLA. This is economically infeasible for high-volume content generation.
  • C is Incorrect: Fine-tuning Model B adds significant operational overhead (data preparation, training jobs, version management) and is premature when prompt engineering has not been attempted. The prompt-first principle applies here.
  • D is Incorrect: Building a custom model from scratch ignores the general language generation capability already present in foundation models, requires massive training data and compute, and adds months of development time.
  • E is Incorrect: A routing architecture adds operational complexity, requires a routing service, and still incurs Model A's high costs on 30% of requests. This is an over-engineered solution when prompt engineering can solve the quality gap on the cheaper model.

Key Takeaway: Evaluate foundation models against concrete business constraints (latency, cost, quality) and exhaust prompt engineering before considering fine-tuning or complex routing architectures.

Question 9 - Model Selection for Tabular Prediction with Interpretability

A hospital's readmission prevention program needs a model to predict which discharged patients are at risk of readmission within 30 days. The hospital has 5 years of structured EHR data (demographics, diagnoses, medications, lab results). The care coordination team must understand and act on each prediction, requiring that the model output includes the top 5 risk factors for each patient. The model will process 50,000 patient records weekly in batch mode.

Which modeling approach BEST satisfies the interpretability and prediction requirements? (Select ONE)

  • A) Train an XGBoost classifier on SageMaker and integrate SHAP explainability to identify top risk factors per patient.
  • B) Use a large language model with patient data provided in the prompt, asking for both prediction and explanation.
  • C) Train a deep neural network, which will capture complex non-linear interactions but requires post-hoc saliency analysis.
  • D) Use SageMaker Autopilot to automatically select the best model, ignoring interpretability requirements.
  • E) Implement a simple logistic regression model with only the top 3 most important features to ensure maximum interpretability.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Answer Analysis

  • A is Correct: XGBoost handles structured EHR data effectively, achieving strong predictive performance on tabular features. SHAP (SHapley Additive exPlanations) provides per-instance feature attribution, identifying the top 5 risk factors for each patient. SageMaker Clarify can also be used for offline bias and explainability analysis. Batch processing 50K records weekly is within XGBoost's capabilities.

Distractor Analysis

  • B is Incorrect: Foundation models are not appropriate for tabular risk prediction. LLMs cannot reliably process years of structured EHR features in a prompt, lack calibrated probability outputs, and cannot provide defensible risk factor attribution for clinical decisions.
  • C is Incorrect: Deep neural networks on tabular data often fail to outperform gradient-boosted trees and lack native interpretability. Post-hoc saliency analysis (gradient attribution) is less reliable and clinically defensible than tree-based methods with SHAP.
  • D is Incorrect: SageMaker Autopilot optimizes for accuracy without considering interpretability constraints. It may select a black-box model that fails the care coordination team's requirement for actionable risk factor explanations.
  • E is Incorrect: Limiting to a 3-feature logistic regression sacrifices too many relevant clinical predictors (lab values, medications, comorbidity indices), likely producing inadequate predictive power for the readmission task.

Key Takeaway: For clinical tabular predictions with interpretability requirements, XGBoost + SHAP is the industry-standard approach—it provides both strong predictive performance and clinically actionable feature-level explanations.

Question 10 - Bedrock Model Selection with Multi-Dimensional Constraints

A legal technology startup is building an AI contract review assistant that must flag risky clauses in contracts and suggest alternative language. The application serves enterprise law firms with sensitive client data. Requirements include: (1) responses must include a confidence score for each flagged risk, (2) inference latency must be under 1.5 seconds, (3) the model must handle documents up to 100,000 tokens long, and (4) total monthly inference cost must stay under $10,000 for 1 million queries. The team has not yet benchmarked available foundation models.

Which evaluation sequence BEST aligns with AWS best practices for foundation model selection? (Select ONE)

  • A) Select the largest available foundation model in Amazon Bedrock to ensure maximum quality, then optimize cost after deployment.
  • B) Build a evaluation framework using Amazon Bedrock Model Evaluation to benchmark multiple models against quality metrics, then evaluate latency and cost, selecting the model that satisfies all constraints.
  • C) Deploy the first model recommended on the AWS blog for legal use cases and monitor costs after launch.
  • D) Fine-tune a small model on contract data rather than evaluating base models, assuming domain-specific training will yield better results.
  • E) Use Amazon Comprehend to identify risky clauses with a rule-based system, avoiding foundation models entirely.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: AWS best practice for model selection is evidence-based evaluation. Amazon Bedrock Model Evaluation (automatic or human-based) allows benchmarking multiple foundation models against custom quality metrics. The evaluation sequence should: (1) define task-specific quality metrics, (2) benchmark models for quality, (3) evaluate latency and cost constraints, and (4) select the model that meets all constraints. This approach matches the measurement-not-opinion principle from the syllabus.

Distractor Analysis

  • A is Incorrect: Selecting the largest model on reputation violates the evidence-based selection principle. Large models may fail the latency (<1.5s), cost ($10K/month for 1M queries), or context window requirements. "Maximum size" does not equal "best fit."
  • C is Incorrect: Deploying without evaluation introduces unnecessary risk—the model may fail quality, latency, cost, or context requirements. Blog recommendations are not a substitute for systematic benchmarking.
  • D is Incorrect: Fine-tuning before evaluating base models violates the "prompt first, fine-tune second" principle. Base models on Bedrock may already meet quality requirements, making fine-tuning unnecessary overhead. Fine-tuning also requires labeled legal data, which introduces data preparation and training costs.
  • E is Incorrect: Amazon Comprehend performs entity extraction and sentiment analysis but cannot identify nuanced contract risk clauses or suggest alternative legal language. This task requires generative AI capabilities that Comprehend does not provide.

Key Takeaway: Foundation model selection must be evidence-driven: define quality metrics, use Bedrock Model Evaluation to benchmark options, then evaluate latency and cost constraints—never select models on reputation or size alone.