Skill 2.1.4: Evaluate tradeoffs between custom solutions, managed services, pre-trained models, and FMs to meet business needs

Question 1 - Managed AI Service vs Custom Model

A healthcare insurance company receives thousands of scanned medical claim forms daily in PDF format. The claims department needs to extract patient names, dates of service, procedure codes, and billed amounts from these forms to populate their claims processing system. The operations team has no ML expertise, requires a solution within two weeks, and must comply with HIPAA regulations where all processing must occur within their VPC.

Which approach provides the FASTEST path to production with minimal ongoing maintenance? (Select ONE)

A) Train a custom OCR model using Amazon SageMaker with a labeled dataset of historical claim forms
B) Use Amazon Textract with VPC endpoints to extract structured form data from the scanned documents
C) Deploy a computer vision model from Amazon SageMaker JumpStart and fine-tune it on claim form layouts
D) Use Amazon Bedrock with a multimodal foundation model to extract key-value pairs from each document
E) Build a document processing pipeline using AWS Lambda functions that parse PDF text with open-source libraries

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: Amazon Textract is a fully managed AI service specifically designed for extracting text, forms, and tables from scanned documents. It requires zero model training, zero infrastructure management, and supports VPC endpoints for HIPAA-compliant processing. The AnalyzeDocument API with FORMS feature extracts key-value pairs from structured documents like claim forms, delivering production-ready results within days.

Distractor Analysis

A is Incorrect: Training a custom OCR model requires thousands of labeled examples, ML expertise for model selection and tuning, and ongoing model maintenance—this adds months of development time and violates the two-week deadline and no-ML-expertise constraint.
C is Incorrect: JumpStart provides pre-trained models as starting points, but fine-tuning still requires labeled data preparation, training jobs, hyperparameter tuning, and deployment management—overkill when a managed service already solves the problem.
D is Incorrect: While Bedrock multimodal models can extract information from documents, they are general-purpose and may hallucinate or miss structured fields. Textract is purpose-built for document key-value extraction with deterministic bounding box outputs.
E is Incorrect: Building custom PDF parsing code with open-source libraries requires significant engineering effort, fails on complex table structures and rotated text, and creates a codebase that requires ongoing maintenance as document formats evolve.

Key Takeaway: When a managed AWS AI service (Textract, Rekognition, Transcribe, Comprehend) already solves the task, calling that API is always faster and cheaper to maintain than building, training, or fine-tuning any model.

Question 2 - Foundation Model Selection Criteria

A financial services firm is building a real-time customer support chatbot that must answer questions about investment products. The solution requires:

Response latency under 2 seconds
Ability to handle complex financial reasoning
Support for 100,000 requests per day
Compliance with SOC 2 and data residency in the EU

The ML engineer is evaluating foundation models in Amazon Bedrock. Which model characteristic is MOST critical for meeting the latency and cost requirements while maintaining quality? (Select ONE)

A) The model's benchmark score on the MMLU (Massive Multitask Language Understanding) evaluation
B) The model's support for fine-tuning on proprietary financial data
C) The model's inference throughput, price per 1,000 tokens, and maximum context window
D) The model's training data cutoff date and inclusion of recent financial news
E) The model's ability to generate code that can be executed for financial calculations

Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

C is Correct: For production deployments with strict latency SLAs (under 2 seconds) and cost constraints (100K requests/day), the primary selection criteria are inference speed (tokens/second), pricing per request, and context window efficiency. A smaller model with lower latency and cost may satisfy the business need better than a larger model with marginally higher quality but excessive cost or delay. These operational characteristics directly determine whether the solution can meet real-time SLAs and remain financially viable.

Distractor Analysis

A is Incorrect: While benchmark scores indicate general capability, they do not reflect real-world latency, cost, or deployment efficiency. A high-ML benchmark score does not guarantee the model can respond within 2 seconds under production load.
B is Incorrect: Fine-tuning support is a secondary consideration. The question asks about initial model selection to meet latency and cost; fine-tuning may add capability but does not create latency improvements.
D is Incorrect: Training data cutoff is important for topical freshness, but it ranks below operational constraints (latency/cost) because the chatbot can use RAG to supplement recent financial information.
E is Incorrect: Code generation is a specialized capability that may be useful but does not directly answer the core requirement of real-time conversational support with strict latency and cost constraints.

Key Takeaway: Foundation model selection must prioritize operational constraints (latency, cost, throughput) alongside quality metrics—never benchmark scores alone.

Question 3 - Prompt Engineering vs Fine-Tuning Decision

A legal tech company deploys an Amazon Bedrock foundation model to summarize legal contracts. The model produces grammatically correct summaries but consistently fails to use the firm's preferred legal terminology (e.g., using "party" instead of "counterparty," or "termination" instead of "rescission"). The ML team has tried adding a system prompt instruction: "Use standard legal terminology." Evaluations still show 35% of summaries use non-standard terms.

The team is deciding between enhanced prompt engineering and supervised fine-tuning. Which action is the MOST appropriate next step? (Select ONE)

A) Immediately initiate supervised fine-tuning with a labeled dataset of 50,000 contract-summary pairs
B) Add few-shot examples to the prompt showing three examples of contracts summarized with correct legal terminology
C) Switch to a larger, more expensive foundation model in Amazon Bedrock
D) Use Amazon Bedrock Knowledge Bases to retrieve legal terminology definitions at inference time
E) Create a post-processing Lambda function that replaces non-standard terms with preferred terminology using a mapping dictionary

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: The progression path is: basic instructions → instructions + few-shot examples → fine-tuning. The scenario states only a basic system prompt was tried. Adding 3-5 worked examples (few-shot learning) directly inside the prompt demonstrates the exact terminology and style expected. This approach is immediate, costs nothing additional, and often closes terminology gaps without any training overhead.

Distractor Analysis

A is Incorrect: Jumping directly to fine-tuning before exhausting prompt engineering options violates the "prompt first, fine-tune second" principle. Fine-tuning requires dataset preparation, GPU training costs, version management, and weeks of iteration—all unnecessary if few-shot prompting suffices.
C is Incorrect: A larger model may not solve terminology consistency—the issue is task-specific adaptation, not general capability. This approach also increases cost and latency per inference.
D is Incorrect: RAG retrieves external facts, not stylistic preferences. Retrieving terminology definitions adds context but the model may still ignore them; few-shot examples in the prompt directly condition the model's behavior more effectively.
E is Incorrect: Post-processing with a dictionary is brittle—it cannot handle grammatical agreement, context-sensitive term usage, or new terminology requirements. It also creates an entirely separate codebase to maintain.

Key Takeaway: Always exhaust prompt engineering (instructions → few-shot examples) with formal evaluations before considering fine-tuning; few-shot examples are free, immediate, and often sufficient.

Question 4 - Custom ML vs Fixed Rules

A manufacturing plant wants to implement predictive quality control on its assembly line. Sensors monitor temperature, pressure, and vibration every second. The quality team reports that defects occur through complex, non-linear interactions between these variables—and a defect typically shows as a 0.3°C temperature anomaly combined with a specific vibration frequency shift. The operations manager asks for a system where technicians can understand WHY a defective part was flagged.

Which TWO characteristics make this best suited to interpretable ML rather than fixed rules or a black-box foundation model? (Select TWO)

A) The defect patterns involve hidden, non-obvious interactions between multiple sensor readings
B) The system must provide human-interpretable explanations for each prediction
C) The plant has no historical sensor data available
D) The anomaly detection must respond in under 10 milliseconds
E) The operations team has extensive cloud computing expertise

Click to reveal Answer & Explanation

Correct Answer: A and B

Correct Option Analysis

A is Correct: ML models (e.g., decision trees, linear models with feature attributions, or anomaly detection algorithms like Random Cut Forest) can learn subtle non-linear patterns and multi-variate interactions from historical sensor data that would be impossible to encode as static threshold rules.
B is Correct: The requirement for technician interpretability directly selects for interpretable ML models (or models paired with explainability tools like SageMaker Clarify). Fixed rules are explainable but insufficient for multivariate anomaly detection; deep learning models capture patterns but are black-boxes. Interpretable ML or explanation layers bridge this gap.

Distractor Analysis

C is Incorrect: This would argue AGAINST custom ML training. Supervised or unsupervised anomaly detection requires historical data to learn normal operational patterns; without data, fixed rules or external expertise would be preferable.
D is Incorrect: Sub-10ms latency constrains toward edge inference, lightweight models, or fixed rules—not away from interpretable ML. While important, it does not distinguish between the modeling approaches in the question.
E is Incorrect: Cloud expertise is an operational consideration, not a modeling constraint. The core decision factors are pattern complexity (favoring ML) and interpretability (favoring interpretable algorithms).

Key Takeaway: ML is worth building when patterns are too complex for rules AND stakeholder interpretability can be satisfied with interpretable algorithms or explainability tools.

Question 5 - RAG vs Fine-Tuning for Dynamic Content

An e-commerce company wants to build an AI assistant that answers customer questions about product return policies, shipping options, and warranty terms. The legal team updates these policies weekly, and the assistant must always cite the exact policy document and section number in every answer. The company has thousands of policy pages.

Which architectural approach BEST satisfies the requirements for always-current content and mandatory source citations? (Select ONE)

A) Fine-tune an Amazon Bedrock foundation model on the complete policy corpus so it internalizes all policies
B) Use Amazon Bedrock Knowledge Bases with an OpenSearch Serverless vector store to retrieve passages at query time
C) Write a deterministic rule-based system that searches for policy keywords and returns fixed responses
D) Train a custom question-answering model using Amazon SageMaker with labeled policy Q&A pairs
E) Use Amazon Comprehend to extract policy entities and route fixed answer templates based on the detected policy type

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: RAG (Retrieval-Augmented Generation) via Amazon Bedrock Knowledge Bases is specifically designed for this scenario:
1. Always-current content: When policies are updated, re-syncing the knowledge base updates the vector index within minutes—no model retraining needed.
2. Mandatory citations: Retrieval systems carry source metadata (document title, section number, page) with each chunk, enabling the model to explicitly cite where it found each fact.
3. Fully managed: Bedrock Knowledge Bases handles parsing, chunking, embedding, and vector store syncing, eliminating custom pipeline maintenance.

Distractor Analysis

A is Incorrect: Fine-tuning stores policy knowledge directly in model weights. Weekly policy updates would require a new fine-tuning job every week (data prep + GPU training + deployment + evaluation), and the model would NOT be able to cite specific document sections because that provenance is lost in weight optimization.
C is Incorrect: Keyword-based rules cannot handle natural language questions, synonyms, or multi-intent queries, and fixed responses will not adapt to nuanced policy questions. This limits user experience and fails to leverage modern AI capabilities.
D is Incorrect: Training a custom Q&A model requires a massive labeled dataset, and retraining on weekly policy changes is operationally prohibitive. This approach fails the always-current requirement.
E is Incorrect: Comprehend extracts entities but does not answer questions. Pairing entity extraction with fixed templates is a rules-based approach that fails on unseen queries and offers no generative response capability.

Key Takeaway: RAG is the correct architecture when content changes frequently OR answers must cite sources; fine-tuning is correct when behavior/style needs to change and content is relatively static.

Question 6 - Model Selection Tradeoffs: Performance vs Latency vs Cost

A video streaming platform wants to add AI-powered content recommendations. The system must:

Generate recommendations within 500ms for 10 million daily active users
Operate within a budget of $50,000 per month for inference
Achieve acceptable click-through rate improvement of at least 10%

The ML team is comparing two approaches:

Option A: A large transformer-based recommendation model in Amazon SageMaker with 800 million parameters
Option B: A lightweight gradient-boosted tree model (XGBoost) with feature engineering

Given that both models achieve similar click-through improvements in offline testing, which approach should the team select and why? (Select ONE)

A) Option A because transformer models are state-of-the-art and will perform better in production
B) Option A because larger models capture more complex user behavior patterns
C) Option B because it has significantly lower inference latency and cost while meeting the performance requirement
D) Option B because XGBoost models are more interpretable than transformer models
E) Either option because they achieve similar performance, so the choice is arbitrary

Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

1. Can serve predictions in milliseconds (meeting 500ms SLA)
2. Run efficiently on relatively modest compute instances (meeting $50K/month budget)
3. Are easy to deploy and scale horizontally for 10M users
C is Correct: When two models achieve equivalent business performance (similar CTR improvement), the deciding factors become operational constraints—latency and cost. XGBoost models:

The 800M parameter transformer model would require GPU inference endpoints that cost significantly more and introduce higher latency, potentially violating both constraints for no business benefit.

Distractor Analysis

A is Incorrect: "State-of-the-art" is not a business requirement. If offline testing shows both models achieve the 10% CTR improvement threshold, deploying a more expensive model buys nothing.
B is Incorrect: The scenario states both models achieve similar click-through improvement, so complexity does not equal superiority in this case. Model complexity only matters if it produces statistically significant performance gains.
D is Incorrect: Interpretability is not a stated requirement in this scenario. While XGBoost's interpretability (via SHAP) could be a bonus, the primary constraint driving selection is cost and latency.
E is Incorrect: The choice is not arbitrary—operational constraints (latency SLA, cost budget) decisively favor the lighter model.

Key Takeaway: Model selection should be driven by whether business requirements are met (performance) AND whether operational constraints are satisfied (latency, cost); don't pay for complexity you don't need.

Question 7 - AWS AI Service Selection

A media and entertainment company receives thousands of hours of customer service call recordings each month. The analytics team wants to:

Convert speech to text for all recordings
Identify customer sentiment (positive, negative, neutral)
Extract frequently mentioned product names and complaint categories
Generate weekly summary reports

The team consists of a data analyst and a business intelligence developer with no ML background. Which combination of AWS services provides the LOWEST operational overhead? (Select ONE)

A) Build a custom pipeline with SageMaker: train a speech-to-text model, a sentiment classifier, and a named entity recognition model
B) Use Amazon Transcribe for speech-to-text, then Bedrock foundation model with prompting for sentiment and entity extraction
C) Use Amazon Transcribe (speech-to-text) + Amazon Comprehend (sentiment and entity extraction) + Amazon QuickSight (dashboards)
D) Use Amazon Bedrock multimodal model to process audio directly, extracting sentiment and entities from the raw recordings
E) Host open-source Whisper models on EC2 for transcription paired with spaCy for NLP analysis

Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

1. Amazon Transcribe: Purpose-built for speech-to-text with speaker diarization and custom vocabulary support—no training required.
2. Amazon Comprehend: Native sentiment analysis API and entity extraction API that work directly on text from Transcribe—fully managed NLP.
3. Amazon QuickSight: Managed BI tool for weekly dashboards and reports.
C is Correct: This solution leverages fully managed AWS AI services for every component:

Zero model training, zero infrastructure management, zero custom pipeline code. The analyst and BI developer can orchestrate this with Step Functions or EventBridge without ML expertise.

Distractor Analysis

A is Incorrect: Training three custom ML models requires massive labeled datasets, ML expertise, infrastructure management, and ongoing maintenance—completely impractical for a team with no ML background.
B is Incorrect: This is viable but introduces more complexity than necessary. Bedrock requires prompt engineering, evaluation, and token cost management for sentiment and entity extraction tasks that Comprehend already performs deterministically for a fraction of the cost.
D is Incorrect: Multimodal models in Bedrock are not optimized for long-form audio transcription (context window limitations) and would be prohibitively expensive for thousands of hours of audio. Transcribe is purpose-built and more cost-effective.
E is Incorrect: Self-hosting open-source models on EC2 requires container management, auto-scaling, GPU instances, orchestration, and custom integration code—the highest operational overhead among all options.

Key Takeaway: When standard NLP tasks (transcription, sentiment, entity extraction) are needed, managed AI services like Transcribe and Comprehend deliver production results with zero ML infrastructure burden.

Question 8 - Distillation vs Fine-Tuning vs Prompting

A mobile gaming company wants to deploy an in-game chat moderation assistant that must:

Detect toxic language, harassment, and inappropriate content in real-time
Operate at sub-50ms latency on edge devices with limited connectivity
Achieve 98% accuracy on the company's specific game context (e.g., distinguishing trash talk from actual toxicity)
Minimize per-request cost since millions of messages flow daily

The company has evaluated foundation models via API but found them too slow and expensive for per-message moderation at scale. What is the MOST appropriate approach? (Select ONE)

A) Use prompt engineering with a Bedrock foundation model, optimizing with caching for common scenarios
B) Distill a large foundation model into a compact classifier model for on-device deployment
C) Fine-tune a mid-size foundation model in Bedrock and deploy it behind an API Gateway with auto-scaling
D) Use Amazon Comprehend's toxicity detection API for all messages
E) Write rule-based filters targeting known toxic words and phrases in multiple languages

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

1. Latency requirements (sub-50ms) exceed cloud API capabilities: Edge deployment requires a compact, efficient model.
2. Cost constraints demand local inference: No per-request API charges.
3. Domain-specific accuracy is required: The distill model can be trained on game-specific examples with labels distinguishing trash talk from toxicity.
B is Correct: Distillation is the appropriate approach when:

Distillation transfers the general language understanding of a large FM into a smaller, faster model (e.g., a distilled BERT-classifier) that can run locally on clients or edge servers.

Distractor Analysis

A is Incorrect: Cloud API latency cannot meet sub-50ms for real-time chat moderation, and caching only works for repeated messages, not novel content.
C is Incorrect: Even fine-tuned, a mid-size FM is still too large for edge deployment and cloud API calls introduce network latency exceeding 50ms.
D is Incorrect: Comprehend toxicity detection is a cloud API that still introduces network latency and may not capture game-specific nuance (trash talk vs. toxicity). It is also cost-prohibitive for millions of daily messages.
E is Incorrect: Rule-based filters are extremely brittle, fail on obfuscated terms ("sh1t", Unicode tricks), and cannot distinguish context-dependent trash talk from true toxicity.

Key Takeaway: Distillation is the right choice when you need the intelligence of a large model but the latency and cost profile of a small, edge-deployable model; it transfers capability rather than just patterns.

Question 9 - Zero-Shot vs Few-Shot vs Fine-Tuning Evaluation

A healthcare technology company is building a clinical note summarization tool using Amazon Bedrock. The ML team is evaluating whether to use zero-shot prompting, few-shot prompting, or supervised fine-tuning. They have:

Access to 100,000 historical clinical notes with expert-written summaries
A requirement for 95% medical terminology accuracy
A constraint that summaries must follow a specific SOAP format (Subjective, Objective, Assessment, Plan)
A two-week deployment deadline

They run an experiment with three configurations and measure format compliance:

Zero-shot: 62% format compliance
Few-shot (3 examples): 78% format compliance
Few-shot (10 examples): 84% format compliance

Based on this data, which approach should they choose? (Select ONE)

A) Zero-shot prompting with an exhaustive system prompt describing the SOAP format
B) Few-shot prompting with more curated examples (20-50) targeting format failures
C) Supervised fine-tuning on the full 100,000-example dataset
D) Supervised fine-tuning on a smaller subset of 5,000 high-quality examples
E) Retrieval Augmented Generation that retrieves exemplary summaries at inference time

Click to reveal Answer & Explanation

Correct Answer: D

Correct Option Analysis

D is Correct: The trend shows few-shot examples improve performance (62% → 78% → 84%), but diminishing returns suggest few-shot alone may not reach 95% compliance. Supervised fine-tuning with 5,000 high-quality, well-distributed examples provides enough signal to learn the SOAP format and medical terminology patterns. Using a subset (rather than all 100K) balances training time/cost against quality—fine-tuning can be completed within the two-week deadline while achieving high accuracy.

Distractor Analysis

A is Incorrect: Zero-shot achieved only 62% compliance—far below the 95% requirement. More verbose prompts are unlikely to bridge a 33-point gap.
B is Incorrect: The diminishing returns from 3→10 examples (78%→84%) suggest even 50 examples may not achieve 95% compliance. Prompt context windows would also become unwieldy.
C is Incorrect: Fine-tuning on 100K examples introduces unnecessary training time and GPU cost when 5K examples would suffice. Additionally, the dataset may contain noise or low-quality summaries that degrade training.
E is Incorrect: RAG for style/format is inefficient—the model still needs to learn how to apply the retrieved examples. This introduces retrieval latency and does not address the fundamental format-following deficiency.

Key Takeaway: When few-shot prompting shows improvement but plateaus below requirements, fine-tuning on a curated subset of high-quality labeled data is the appropriate escalation path.

Question 10 - RAG Architecture Pattern Selection

A global consulting firm wants to deploy an internal AI assistant that answers employee questions about HR policies, IT support procedures, and project management guidelines. The content spans:

50,000 documents across multiple internal systems (SharePoint, Confluence, ServiceNow)
Documents in 5 languages including English, Spanish, and Japanese
Frequent updates (multiple documents edited daily)
Strict requirement that answers cite specific source documents

The IT team must choose between Amazon Bedrock Knowledge Bases (managed RAG) and a custom RAG pipeline built on SageMaker components. Which TWO factors most strongly favor the managed Bedrock Knowledge Bases approach? (Select TWO)

A) The requirement for custom reranking algorithms to optimize retrieval quality
B) The need to reduce operational overhead for ongoing maintenance
C) The requirement to cite specific source documents and sections
D) The need to support multiple document formats and connectors
E) The requirement for a custom vector database with specialized indexing strategies

Click to reveal Answer & Explanation

Correct Answer: B and D

Correct Option Analysis

B is Correct: Bedrock Knowledge Bases eliminates the operational burden of building and maintaining parsing pipelines, embedding jobs, vector store syncing, and retrieval infrastructure. For a global firm with frequent document updates and no dedicated ML ops team, managed RAG dramatically reduces engineering effort and ongoing maintenance.
D is Correct: Bedrock Knowledge Bases provides native connectors for SharePoint, Confluence, ServiceNow, and S3—handling document ingestion, format normalization, and multi-language parsing automatically. Building custom connectors for multiple internal systems is significant engineering effort that the managed service eliminates.

Distractor Analysis

A is Incorrect: Custom reranking algorithms specifically require breaking away from managed retrieval. Bedrock Knowledge Bases uses its own retrieval and ranking mechanisms; the requirement for custom reranking would drive the team toward a custom pipeline.
C is Incorrect: Both managed and custom RAG can provide source citations. This requirement is neutral—it does not favor either approach strongly enough to be a deciding factor.
E is Incorrect: The need for a custom vector database with specialized indexing is directly incompatible with the managed RAG approach (which defines and manages the vector store). This factor would favor a custom pipeline.

Key Takeaway: Managed RAG (Bedrock Knowledge Bases) wins when operational simplicity and broad source connectivity matter more than fine-grained control over retrieval, indexing, or ranking algorithms; custom RAG wins when specialized control beats maintenance reduction.