Skip to content

Skill 1.1.1: Extract data from data sources

Question 1 - Foundation Model Selection for Real-Time Customer Support Chatbot

A financial services company is deploying a customer support chatbot that must handle account-specific questions with sub-second response times. The engineering team is evaluating foundation models on Amazon Bedrock and has narrowed the selection to three candidate models. Requirements state the chatbot must maintain 99.9% availability, achieve p95 latency under 800ms, and support 50 concurrent requests per second during peak trading hours. The total cost per 1M tokens must not exceed \(15 for input** and **\)30 for output.

Which approach should the machine learning engineer use to make the final model selection? (Select ONE)

  • A) Select the largest available foundation model (e.g., Claude 3 Opus) because higher parameter counts guarantee better response quality for financial queries.
  • B) Run a structured evaluation harness using Amazon Bedrock model evaluation jobs with held-out financial queries, measuring quality metrics, latency percentiles, and token costs simultaneously before selecting a model.
  • C) Deploy all three candidate models behind an Application Load Balancer and route traffic randomly to gather live production metrics before making a permanent selection.
  • D) Query the Amazon Bedrock pricing page and select the model with the lowest cost per token, assuming all foundation models deliver equivalent output quality for financial terminology.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: The requirement explicitly demands simultaneous evaluation of quality (financial domain accuracy), latency (p95 < 800ms), and cost (per-token constraints). Amazon Bedrock's managed evaluation jobs (CreateEvaluationJob) support automated benchmarking against custom datasets with configurable metrics, enabling evidence-based model selection rather than assumption-driven choices.

Distractor Analysis

  • A is Incorrect: Larger models do not guarantee better domain-specific performance for financial queries. They typically have higher latency and cost, potentially violating the p95 latency and cost constraints. Model selection must be measurement-driven.
  • C is Incorrect: Randomized live traffic routing introduces production risk (inconsistent customer experience, potential data governance issues) and does not provide controlled, reproducible comparison conditions. Model evaluation should occur in pre-production environments.
  • D is Incorrect: Selecting purely on cost ignores quality and latency requirements. A cheaper model may produce inaccurate financial advice or exceed latency budgets, creating regulatory and customer satisfaction risks.

Key Takeaway: Foundation model selection must be treated as a structured measurement exercise across quality, latency, and cost dimensions—never assume parameter count or pricing alone determines fitness for purpose.

Question 2 - Choosing Between RAG and Fine-Tuning for Policy Updates

An insurance provider is building an AI assistant that must answer questions about its claims processing policies. The compliance department updates these policies at least twice weekly, and each response must include a direct citation to the specific policy document and section. The assistant must reflect policy changes within 10 minutes of publication. The development team has access to Amazon Bedrock and is deciding between fine-tuning a foundation model or implementing a Retrieval Augmented Generation (RAG) architecture.

Which approach most effectively satisfies these requirements? (Select ONE)

  • A) Fine-tune a foundation model on the complete policy corpus weekly, incorporating all policy revisions into the model weights, and redeploy the fine-tuned model after each training job.
  • B) Implement RAG using Amazon Bedrock Knowledge Bases with an Amazon OpenSearch Serverless vector store configured with scheduled ingestion jobs running every 10 minutes, enabling citation of source documents through metadata.
  • C) Use a hybrid approach: fine-tune monthly for policy writing style, and rely on prompt engineering with few-shot examples for individual policy references.
  • D) Store all policy documents in Amazon S3 and instruct the foundation model to generate citations from memory, refreshing the model's knowledge through system prompts updated bi-weekly.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: RAG architecture keeps knowledge external to model weights, allowing rapid content updates through re-ingestion. Amazon Bedrock Knowledge Bases automates document parsing, embedding, and vector store synchronization. The vector index can be synced within minutes (via StartIngestionJob or automated triggers), and source metadata enables direct citation generation. This satisfies both the rapid refresh (10-minute requirement) and citation requirements.

Distractor Analysis

  • A is Incorrect: Fine-tuning embeds information in model weights, requiring hours or days of training before policy updates take effect—far exceeding the 10-minute freshness requirement. Additionally, fine-tuned models cannot naturally cite source documents since knowledge is encoded in weights rather than retrieved from identifiable passages.
  • C is Incorrect: Fine-tuning monthly for writing style is reasonable, but relying on few-shot examples for policy references defeats the purpose—few-shot examples cannot keep pace with bi-weekly policy changes at scale, and LLMs hallucinate specific policy details when asked to recall from memory.
  • D is Incorrect: Foundation models do not have persistent memory of specific documents stored in S3. Asking models to cite from memory invites hallucinations and provides no mechanism for reflecting changes within 10 minutes. No secure retrieval path exists in this approach.

Key Takeaway: Choose RAG over fine-tuning when knowledge freshness requirements are measured in minutes (not days) and source-citation traceability is mandatory for compliance.

Question 3 - Model Selection for Fraud Detection with Interpretability Requirements

A multinational bank needs to deploy a fraud detection model for credit card transactions that flags suspicious activity in real time. The compliance team requires that every declined transaction be accompanied by an explanation of which factors contributed to the fraud score. The bank has 3 years of labeled transaction history (fraud/non-fraud) covering 50 million records. The model must process transactions with maximum 50ms inference latency. The model will need to be updated quarterly as new fraud patterns emerge.

Which modeling approach best satisfies the interpretability, latency, and data requirements? (Select ONE)

  • A) Select a large foundation model on Amazon Bedrock and use prompt engineering to analyze each transaction record, generating natural language explanations for fraud decisions.
  • B) Train a deep neural network using SageMaker AI with distributed training across multiple GPU instances, achieving maximum predictive accuracy for fraud detection.
  • C) Train a gradient-boosted tree model (e.g., XGBoost) using SageMaker AI, then use SageMaker Clarify to generate feature attributions for each prediction, achieving low-latency inference through SageMaker real-time endpoints or multi-model endpoints.
  • D) Use an ensemble of transformer-based foundation models fine-tuned on transaction data, accepting higher latency as a trade-off for superior fraud detection accuracy.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Options Analysis

  • C is Correct: Gradient-boosted tree algorithms (XGBoost, LightGBM) are standard for tabular fraud detection, providing competitive accuracy with rigorous interpretability via feature attributions. Amazon SageMaker Clarify integrates directly to generate SHAP (SHapley Additive exPlanations) values for any trained model, satisfying the regulatory requirement for explanation. Inference latency with tree models is sub-10ms in most cases, easily meeting the 50ms constraint. The proprietary labeled transaction data (50M records) makes this a classic supervised learning scenario where patterns exist in the organization's own data.

Distractor Analysis

  • A is Incorrect: Foundation models are not designed for numerical tabular fraud detection at scale. Sending each transaction record through a general-purpose LLM would result in excessive latency (typically hundreds of milliseconds or more), high per-request costs, and questionable fraud-detection performance compared to specialized ML models.
  • B is Incorrect: Deep neural networks are often "black box" models that fail the compliance requirement for explainable decisions unless significant additional effort is invested in explanation methods. While SageMaker Clarify supports DNNs, the added complexity and training cost provide marginal accuracy gains over gradient-boosted trees for tabular data.
  • D is Incorrect: Transformer models are overkill for tabular fraud detection, introduce unnecessary operational complexity and cost, and will likely exceed the 50ms latency constraint. Fine-tuning foundation models on 50M labeled tabular records is also not an optimal use of FM capabilities.

Key Takeaway: For tabular business problems with proprietary historical data and strict interpretability requirements, traditional ML algorithms (especially tree-based) with feature attribution tools (Clarify/SHAP) outperform foundation models on both latency and explainability.

Question 4 - Evaluating When to Use a Managed AI Service vs. Custom Model

A healthcare startup needs to process patient intake forms that arrive as scanned PDF documents and photographs taken on mobile devices. The forms contain handwritten physician notes, printed insurance information, and table-structured medication lists. The startup has no in-house ML expertise and must reach production within 4 weeks. The system must extract text with high accuracy and handle multiple document layouts from different hospitals.

Which approach most effectively meets the constraints while minimizing development effort? (Select ONE)

  • A) Train a custom OCR model using SageMaker AI on a dataset of the specific forms, investing 3 weeks in data labeling and model training across GPU instances.
  • B) Use Amazon Textract with both AnalyzeDocument (for forms and tables) and AnalyzeExpense (for insurance-related structured extraction), deploying through a simple serverless architecture with Lambda and S3 triggers.
  • C) Fine-tune a multimodal foundation model on Amazon Bedrock using examples of the scanned documents, preparing a custom dataset and running multiple training jobs.
  • D) Write a Python script using OpenCV and Tesseract OCR, deploy it to an EC2 instance, and implement custom regex rules for extracting specific fields from each document layout.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: Amazon Textract is a fully managed AWS AI service specifically designed for document text extraction, form field identification, and table parsing. It handles handwriting, printed text, and multiple document layouts natively. The startup's constraints (no ML expertise, 4-week deadline, multiple layouts) make a managed service the optimal choice—no training, labeling, or infrastructure management required. The serverless architecture (S3 -> Lambda -> Textract) provides automatic scaling and minimal operational overhead.

Distractor Analysis

  • A is Incorrect: Training a custom OCR model from scratch requires ML expertise the startup doesn't have, 3+ weeks of training effort, and will only perform well on the specific form layouts used in training—failing the requirement for handling multiple document layouts from different hospitals.
  • C is Incorrect: Fine-tuning a multimodal foundation model requires significant dataset preparation, specialized training infrastructure, and ML expertise. This approach is unnecessarily complex when a managed service already solves the exact task of document text extraction. The 4-week deadline would be at risk.
  • D is Incorrect: Custom OCR solutions using Tesseract and OpenCV require substantial engineering effort, handle handwriting poorly, and break when document layouts vary. The healthcare environment's accuracy requirements and the startup's lack of ML expertise make this approach risky and maintenance-heavy.

Key Takeaway: For pre-defined tasks solved by AWS managed AI services (text extraction, image analysis, speech-to-text), calling the service APIs always beats custom training when you lack domain-specific data advantages and need fast time-to-production.

Question 5 - Selecting Between Custom Vector Store and Managed RAG for Compliance

A pharmaceutical company is building an internal RAG application for researchers to query clinical trial documents. The legal team mandates that all document chunks and embeddings remain in the company's private VPC with no data leaving their AWS account. The system must support hybrid search (combining vector similarity with BM25 keyword matching) for precise retrieval of drug interaction data. The ML team wants control over the embedding model and chunking strategy. The company plans to scale to 50 million document chunks within 2 years.

Which architectural pattern best satisfies these requirements? (Select ONE)

  • A) Use Amazon Bedrock Knowledge Bases for fully managed ingestion, embedding generation, and retrieval, accepting the service defaults for embedding model and chunking strategy.
  • B) Build a custom retrieval architecture with SageMaker AI endpoints hosting custom embedding models, an OpenSearch Serverless vector collection with hybrid search enabled, and Step Functions orchestrating a custom document processing pipeline.
  • C) Store document embeddings in Amazon RDS for PostgreSQL with the pgvector extension, implementing a custom vector similarity search layer with a Python FastAPI service.
  • D) Use Amazon Kendra for document indexing and retrieval, leveraging its semantic search capabilities without building any custom infrastructure.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: The requirements demand granular control over embedding model selection, chunking strategy, and search methodology (hybrid search), all while keeping data within a private VPC. A custom architecture using SageMaker AI for embedding models and OpenSearch Serverless for hybrid vector search provides this control. Step Functions orchestrates the document processing pipeline, and the entire stack can operate within the customer's VPC using VPC endpoints and private subnets. At 50M chunks scale, OpenSearch Serverless scales elastically without managing clusters.

Distractor Analysis

  • A is Incorrect: Amazon Bedrock Knowledge Bases abstracts away embedding model selection and chunking strategy control. While it now supports some customization, the managed service doesn't provide the level of control the team explicitly requires for hybrid search optimization and custom embedding models. Additionally, Knowledge Bases may not guarantee all processing stays within the company's VPC (it uses AWS-managed resources).
  • C is Incorrect: PostgreSQL with pgvector is a viable vector store for small-to-medium scale applications, but 50 million document chunks push pgvector's scalability limits. Creating a custom vector search layer with FastAPI adds unnecessary engineering overhead when OpenSearch provides built-in vector similarity search, hybrid search (BM25 + kNN), and enterprise-grade scaling.
  • D is Incorrect: Amazon Kendra is a managed intelligent search service, but it does not support custom embedding models or fine-grained chunking strategy control. Kendra abstracts the retrieval mechanism entirely, which conflicts with the team's requirement for controlling the embedding and chunking approach.

Key Takeaway: Choose custom retrieval architectures over managed services (Bedrock Knowledge Bases) when you need control over embedding models, chunking logic, search methodology (hybrid), and strict VPC data residency—even though custom architectures trade away operational simplicity.

Question 6 - Cost-Latency Tradeoff Between Real-Time and Batch Inference

A credit scoring bureau processes loan applications received throughout business hours. Applications arrive at irregular intervals averaging 100 per hour but occasionally spiking to 500 per hour during promotional periods. The ML model (a logistic regression trained on SageMaker) produces a risk score used in synchronous approval workflows, requiring response within 200ms. The company wants to minimize infrastructure costs while maintaining the latency SLA during spikes. The model needs no updates between quarterly retraining cycles.

Which deployment strategy best balances latency requirements and cost optimization? (Select ONE)

  • A) Deploy the model to a SageMaker Serverless Inference endpoint with provisioned concurrency set to handle maximum expected peak load (500 requests/hour), ensuring consistent latency.
  • B) Use SageMaker Asynchronous Inference, submitting batches of loan applications every 15 minutes and returning risk scores through S3 result polling.
  • C) Deploy the model to a SageMaker Real-Time Inference endpoint with auto-scaling configured based on invocation counts and latency metrics, setting minimum instance count to 0 during off-hours.
  • D) Process all loan applications in a nightly batch job using SageMaker Batch Transform, combining all daily applications into a single inference job run after business hours.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Options Analysis

  • C is Correct: SageMaker Real-Time Inference provides synchronous ML predictions with sub-100ms latency, meeting the 200ms SLA. Application Auto Scaling can be configured to automatically adjust instance count based on demand metrics (invocations per instance, CPU utilization), handling the variable load profile from 100 to 500 requests/hour. While real-time endpoints have base infrastructure costs, auto-scaling ensures you only pay for additional instances during demand spikes. Setting minimum to 0 during known off-hours (e.g., midnight to 6 AM) eliminates costs when no requests occur. This approach is ideal for synchronous workflows requiring immediate responses.

Distractor Analysis

  • A is Incorrect: SageMaker Serverless Inference with provisioned concurrency designed for maximum peak load results in paying for idle capacity during normal hours (100 requests/hour) when provisioned for peak (500 requests/hour). Serverless inference is better suited for intermittent workloads where cold-start latency is acceptable; provisioned concurrency eliminates cold starts but charges for idle compute.
  • B is Incorrect: Asynchronous inference introduces 15-minute delays, which is unacceptable for synchronous approval workflows requiring 200ms responses. Credit decisions must be returned immediately to the applicant during the application process.
  • D is Incorrect: Nightly batch processing completely violates the synchronous requirement—loan applications require real-time risk scores during business hours. Batch Transform is designed for offline scoring, not latency-sensitive synchronous use cases.

Key Takeaway: For synchronous ML scoring with variable throughput, SageMaker Real-Time Inference with auto-scaling provides the best balance—meeting strict latency SLAs while scaling costs to match actual demand patterns rather than provisioning for theoretical peak load.

Question 7 - Prompt Engineering vs Fine-Tuning for Domain-Specific Summaries

A legal tech company uses an Amazon Bedrock foundation model to summarize court decisions for attorneys. The summaries must follow the firm's specific document structure (with headers like "Holding", "Background", "Reasoning") and use precise legal terminology conventions. Initial testing shows the model produces well-written summaries but frequently deviates from the required structure and uses inconsistent legal terminology despite being given a detailed system prompt describing the requirements.

The engineering team has 50,000 labeled examples of court decisions paired with correctly formatted summaries created by senior attorneys over the past decade.

Which approach should the team implement next? (Select ONE)

  • A) Increase the temperature parameter to 0.9 to encourage more diverse outputs that may coincidentally match the required structure more frequently.
  • B) Fine-tune the foundation model on the labeled court decision-summary pairs using Amazon Bedrock or SageMaker AI, incorporating the precise terminology and structure from the dataset.
  • C) Add more retrieved context passages into the prompt at request time to help the model understand the required structure better.
  • D) Switch to a smaller foundation model to reduce inference costs, assuming the smaller model will follow formatting instructions more consistently.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: The team has exhausted prompt engineering options (detailed system prompts describing requirements) and still observes consistent gaps in structure and terminology. The availability of 50,000 high-quality labeled examples (in-domain paired data) makes supervised fine-tuning the appropriate next step. Fine-tuning updates model weights to internalize the specific formatting conventions and legal terminology patterns. This is the classic "prompt first, fine-tune when prompting fails" progression. With configured datasets meeting Bedrock fine-tuning requirements, this approach directly addresses the behavior gaps that prompting cannot close.

Distractor Analysis

  • A is Incorrect: Increasing temperature from the default to 0.9 makes model outputs more random and less consistent, exacerbating the deviation from required structure and legal terminology constraints. The problem is not lack of diversity—it's lack of adherence to specific conventions; higher temperature makes adherence worse.
  • C is Incorrect: RAG context passages supply factual knowledge (like previous court cases, statutes), but the requirement is about structural formatting and language style—not factual accuracy. Injecting more retrieved content will not train the model to consistently follow specific summary structures or terminology conventions.
  • D is Incorrect: Smaller models are generally less capable at following complex multi-part instructions and maintaining consistent formatting. Switching to a smaller model to reduce cost sacrifices instruction-following capability, making the structure deviation problem worse, not better.

Key Takeaway: Follow the MLOps progression for FM adaptation: task-specific prompt engineering first; only when evaluations show prompting cannot close quality gaps (and labeled data exists) should you invest in supervised fine-tuning.

Question 8 - Managed Service Selection for Multi-Modal Content Moderation

A social media platform needs to detect and remove violent imagery, hate speech in text comments, and spoken profanity in video content uploaded by users. The platform currently handles 100,000 uploads per hour globally. The moderation team has a small ML engineering staff but needs a production-grade solution within 2 months. Compliance requirements state that flagged content must be reviewed by human moderators within 15 minutes of upload.

Which AWS service combination most effectively addresses all three content types with minimal custom development? (Select ONE)

  • A) Train three separate custom models (CNN for images, transformer for text, RNN for audio) using SageMaker AI, then orchestrate them with Step Functions for each upload.
  • B) Use Amazon Rekognition for image and video moderation, Amazon Comprehend for text-based hate speech detection, and Amazon Transcribe for spoken profanity detection, triggering human review through an Amazon Simple Notification Service (SNS) queue.
  • C) Use a single multimodal foundation model on Amazon Bedrock to analyze all three content types simultaneously, providing unified moderation capabilities.
  • D) Deploy an open-source moderation model from HuggingFace (e.g., BLIP-2 or LLaVA) on SageMaker AI endpoints, processing all uploads through custom inference code.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: AWS provides purpose-built managed AI services for each content moderation task: Amazon Rekognition Content Moderation detects unsafe images and videos (violent imagery), Amazon Comprehend performs natural language processing to identify hate speech in text comments, and Amazon Transcribe converts spoken audio in videos to text for profanity detection. The service combination requires no custom model training, scales automatically for 100K uploads per hour, and can be integrated with SNS/SQS for human review workflows within minutes. The 2-month deadline is realistic when using managed services since no training pipelines are needed.

Distractor Analysis

  • A is Incorrect: Training three separate custom models requires extensive labeled training data collection, model architecture design, training infrastructure management, and ongoing model maintenance. The 2-month deadline is likely infeasible for a team with small ML engineering resources, and the operational overhead post-deployment is substantial.
  • C is Incorrect: While multimodal foundation models can theoretically analyze images, text, and audio, this approach lacks production-grade content moderation reliability, may have higher per-inference latency at 100K uploads per hour, and compliance-sensitive moderation requires purpose-built detection rather than general-purpose FM outputs. Custom orchestration is still needed to parse results for human review.
  • D is Incorrect: Open-source models require model hosting infrastructure, custom inference code for each modality, ongoing model performance evaluation, and security patching of the hosting containers. The small ML engineering team would become fully occupied with model operations rather than solving the moderation business challenge. Custom orchestration adds more complexity.

Key Takeaway: When multiple content types need production-ready detection within aggressive deadlines, combine AWS purpose-built managed AI services (Rekognition + Comprehend + Transcribe) rather than building custom models or relying on single general-purpose FMs for compliance-critical moderation tasks.

Question 9 - Training Time and Data Size Tradeoffs for Recommendation System

A streaming service is building a movie recommendation engine using collaborative filtering on their proprietary 5 years of user interaction data (100 million rows: user IDs, movie IDs, ratings, timestamps). The system must generate personalized recommendations in real time when users browse the platform. The company wants the model frequently updated (daily) as new viewing data accumulates. The ML team is choosing between training a deep learning model versus a matrix factorization approach.

The company also requires that recommendations are diverse (not just popular items) and that new movies receive fair exposure within the first week of release.

Which modeling approach best balances performance, training frequency, and business requirements? (Select ONE)

  • A) Train a deep neural network (two-tower architecture) using SageMaker AI with distributed training across 8 GPU instances, achieving state-of-the-art recommendation accuracy with 20-hour training jobs.
  • B) Use Amazon Personalize, a managed recommendation service, with the User-Personalization recipe (HRNN-based) that automatically handles real-time generation, cold-start items, and daily retraining without infrastructure management.
  • C) Train a matrix factorization model (e.g., Alternating Least Squares) using SageMaker AI on a single CPU instance, with daily scheduled retraining jobs and deployment to a real-time endpoint.
  • D) Use Amazon Bedrock with a foundation model, providing user viewing history as prompt context to generate personalized movie recommendations through natural language reasoning.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: Amazon Personalize is a purpose-built managed service for personalized recommendations. The User-Personalization recipe automatically handles daily retraining as new interaction data arrives, real-time recommendation generation through the get-recommendations API, cold-start item handling (new movies can receive exposure through exploration mechanisms built into the recipe), and recommendation diversity through recency and popularity adjustments. Zero infrastructure management overhead means the ML team isn't consumed with training pipeline maintenance—they simply upload interaction data to S3 and call the API.

Distractor Analysis

  • A is Incorrect: Training a custom two-tower neural network with 20-hour training jobs cannot support daily retraining requirements without 24/7 GPU cluster management. Distributed training across 8 GPUs introduces massive operational complexity and costs. Cold-start item handling and diversity require additional custom implementation (e.g., exploration policies, popularity penalties). The team would spend most time on infrastructure rather than recommendation quality.
  • C is Incorrect: Matrix factorization is a reasonable traditional approach, but it requires custom engineering for cold-start item handling and diversity mechanisms. The model would only learn from historical interactions, meaning new movies with no interaction history would be excluded from recommendations. Diversity in recommendations is not an inherent property of matrix factorization; the model naturally favors popular items. Custom infrastructure for daily training pipelines, endpoint updates, and feature engineering adds complexity compared to the managed service.
  • D is Incorrect: Foundation models are inappropriate for large-scale recommendation systems. They cannot efficiently process 100 million rows of interaction data in context windows, cannot be frequently retrained on the data size, and generate recommendations through natural language reasoning—which is not optimized for ranking accuracy compared to dedicated recommendation algorithms. Cost per recommendation would be prohibitive for real-time browsing.

Key Takeaway: For recommendation systems with daily update requirements and cold-start/diversity needs, Amazon Personalize eliminates common ML engineering challenges (training pipeline management, cold-start strategies, diversity tuning) while providing real-time inference through a managed API.

Question 10 - Precision vs. Recall Tradeoff in Medical Detection

A radiology AI company is deploying a model on SageMaker that detects pneumonia from chest X-rays. The system has two deployment options: a screening assistant for radiologists (flags suspicious images for review) and an autonomous triage system (directly notifies urgent care when critical cases detected). The company's medical advisory board states that missing a pneumonia case is unacceptable (a false negative could be life-threatening), while false positives can be tolerated (radiologists can dismiss non-pneumonia flags during review).

The company must choose an evaluation metric to guide model selection between two candidate models:

  • Model A: 95% precision, 70% recall
  • Model B: 85% precision, 95% recall

Which metric and model should the company prioritize for this use case? (Select ONE)

  • A) Prioritize F1-score for balanced optimization, selecting whichever model achieves a higher F1-score since both precision and recall are equally important metrics.
  • B) Prioritize precision, selecting Model A because higher precision means fewer false positive alerts, reducing radiologist workload and system fatigue.
  • C) Prioritize recall (sensitivity), selecting Model B because higher recall means fewer missed pneumonia cases (false negatives), aligning with the medical directive that missing cases is unacceptable.
  • D) Prioritize accuracy (overall correctness), selecting whichever model achieves higher overall accuracy regardless of precision-recall tradeoff.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Options Analysis

  • C is Correct: The medical advisory board's directive explicitly states that missing pneumonia cases (false negatives) is unacceptable because it could be life-threatening, while false positives are tolerable. Recall (sensitivity) measures the proportion of actual pneumonia cases correctly identified. Model B with 95% recall identifies 95% of true pneumonia cases (only 5% missed), versus Model A which would miss 30% of cases. This directly aligns with the stated constraint that sensitivity is the critical metric. The 85% precision (15% false positives) is acceptable because radiologists can review and dismiss false flags.

Distractor Analysis

  • A is Incorrect: F1-score assumes precision and recall are equally important. The medical directive establishes clear asymmetry: recall is prioritized over precision. Using F1-score would bias selection toward Model A if its mathematical balance appeared better, potentially missing 30% of pneumonia cases—violating the core business constraint.
  • B is Incorrect: Prioritizing precision would select Model A, which misses 30% of actual pneumonia cases. Despite reducing radiologist workload from false positives, this optimization would fail the primary medical safety requirement and potentially harm patients.
  • D is Incorrect: Overall accuracy is misleading with imbalanced datasets and doesn't capture the precision-recall tradeoff. A model with high accuracy could have poor recall on the minority class (pneumonia cases) while correctly classifying the majority of normal X-rays. The metric fails to express the cost asymmetry between false negatives and false positives in this medical setting.

Key Takeaway: Evaluation metric selection must reflect the business cost of different error types—when the cost of missing a positive case (false negative) exceeds the cost of false alarms (false positive), optimize for recall; when the cost of false alarms exceeds the cost of misses, optimize for precision; use F1 only when errors have symmetric costs.