Skill 2.1.6: Assess tradeoffs between ML model performance, training time, and cost
| Question | Answer & Explanation |
|---|---|
| ### Question 1 - Prompt Engineering vs. Fine-Tuning Tradeoffs A startup wants to implement a GenAI assistant to summarize internal IT support tickets. The assistant must use the company's specific jargon and adhere to a strict output format. The engineering team wants to minimize operational overhead, avoid unnecessary compute costs, and deploy the solution within a week. Which approach is the MOST cost-effective and fastest to implement? Select ONE that applies. - A) Fine-tune an Amazon Bedrock foundation model using labeled examples of past tickets. - B) Use task-specific prompt engineering with an Amazon Bedrock foundation model, including instructions and few-shot examples. - C) Train a custom sequence-to-sequence model from scratch using Amazon SageMaker AI. - D) Implement a Retrieval-Augmented Generation (RAG) pipeline using Amazon Kendra. |
Correct Answer: B Correct Answer Analysis: B is Correct: Task-specific prompt engineering (instructions + few-shot examples) is the fastest and least costly approach. It allows the team to adapt the foundation model's behavior to specific jargon and formats immediately without the overhead of preparing training data, running GPU training jobs, or managing model versions. Distractor Analysis: - A is Incorrect: Fine-tuning requires preparing a labeled dataset, running a training job, and versioning. This violates the "least operational overhead" and "within a week" constraints when prompting hasn't been tried yet. - C is Incorrect: Training a custom model from scratch incurs massive training time, cost, and operational overhead, vastly exceeding the 1-week deployment constraint. - D is Incorrect: While RAG is useful for fetching external knowledge, simply formatting output and applying jargon is a behavioral task best handled by prompt engineering. RAG adds unnecessary infrastructure overhead (vector stores, ingestion pipelines) for this specific formatting requirement. > Exam Tip: Always start with prompt engineering. Only escalate to fine-tuning if evaluations prove prompting cannot close the performance gap. |
| ### Question 2 - Managed AI Services vs. Custom ML for Document Extraction A logistics company receives thousands of scanned, unstructured PDF delivery receipts daily. They need to extract the sender, recipient, and package weight into a structured database. The company has no in-house ML expertise and wants to implement this with the LEAST development effort and operational overhead. Which approach meets these requirements? Select ONE that applies. - A) Train a custom Optical Character Recognition (OCR) model using Amazon SageMaker AI. - B) Use Amazon Textract to extract text and form data from the scanned receipts. - C) Use Amazon Bedrock to process the images and extract the required fields via prompt engineering. - D) Write fixed coordinate-based rules to parse the PDFs. |
Correct Answer: B Correct Answer Analysis: B is Correct: Amazon Textract is a managed AWS AI service pre-trained to extract text, forms, and tables from documents. It requires zero custom model training, no ML expertise, and no infrastructure management, satisfying the "least development effort" and "least operational overhead" constraints perfectly. Distractor Analysis: - A is Incorrect: Training a custom OCR model requires extensive ML expertise, labeled data, and significant training time/cost, violating the operational overhead and expertise constraints. - C is Incorrect: While Bedrock can process images, using an LLM to extract structured key-value pairs from standard receipts is more expensive, slower, and less reliable than a purpose-built service like Textract. - D is Incorrect: Fixed coordinate rules are extremely brittle and fail when scanned PDFs have slightly different layouts, margins, or scan alignments. > Exam Tip: If a task is already "solved" (like OCR, sentiment, image labeling), use a managed AWS AI service instead of building or training your own. |
| ### Question 3 - Model Distillation for Inference Cost Reduction An AI team is using a large foundation model via Amazon Bedrock for a high-volume text classification task. The model's accuracy is excellent, but the inference costs are exceeding the monthly budget. The team wants to maintain high accuracy while significantly reducing per-request costs and keeping additional training time to a minimum. Which strategy BEST balances these tradeoffs? Select ONE that applies. - A) Distill the knowledge of the large foundation model into a smaller, faster model using Amazon SageMaker AI. - B) Switch from On-Demand capacity to Provisioned Throughput on the large foundation model. - C) Implement a Retrieval-Augmented Generation (RAG) architecture using Amazon OpenSearch. - D) Train a custom transformer model from scratch on the company's labeled data. |
Correct Answer: A Correct Answer Analysis: A is Correct: Model distillation involves training a smaller "student" model to mimic the behavior of the larger "teacher" model. This preserves a high degree of accuracy while drastically reducing inference costs and latency, fitting the constraints of cost reduction with minimal training overhead compared to training from scratch. Distractor Analysis: - B is Incorrect: Provisioned Throughput is used for predictable, high-volume workloads to guarantee capacity, but it does not inherently reduce the per-request compute cost of a large model; it may actually increase costs if utilization isn't 100%. - C is Incorrect: RAG helps inject external knowledge, but it does not reduce the inference cost of the large model itself; in fact, it adds vector database retrieval costs. - D is Incorrect: Training from scratch requires massive amounts of data, extreme training time, and high compute costs, violating the "minimal training time" constraint. > Exam Tip: Distillation is the go-to strategy for reducing inference costs when a large model's performance must be ported to a cheaper, smaller model. |
| ### Question 4 - Traditional ML vs. FM for Tabular Data A retail chain needs to forecast weekly inventory demand for 5,000 stores. They have years of historical tabular sales data. The finance department requires full interpretability of the predictions to comply with internal auditing standards, and inference latency must be in the low milliseconds. Which modeling approach is the MOST appropriate? Select ONE that applies. - A) Train an interpretable model, such as XGBoost, using Amazon SageMaker AI built-in algorithms. - B) Use a large language model via Amazon Bedrock and pass the historical sales data in the prompt. - C) Train a complex deep neural network using Amazon SageMaker AI for maximum accuracy. - D) Use Amazon Comprehend to analyze the sentiment of historical sales reports. |
Correct Answer: A Correct Answer Analysis: A is Correct: For tabular time-series/forecasting data, traditional ML models like XGBoost are highly performant, operate in low milliseconds, and are inherently more interpretable than deep neural networks or foundation models. This fits the interpretability, latency, and data-type constraints. Distractor Analysis: - B is Incorrect: LLMs are not designed for precise, large-scale tabular forecasting. Passing years of tabular data in a prompt is inefficient, costly, suffers from context limits, and lacks the required auditability/interpretability. - C is Incorrect: While deep neural networks can be accurate, they act as "black boxes," making it difficult for finance stakeholders to audit predictions, violating the interpretability constraint. - D is Incorrect: Amazon Comprehend is for NLP tasks (entity extraction, sentiment), not numerical demand forecasting on tabular data. > Exam Tip: When the answer is in your tabular data and interpretability is required, use traditional ML (like SageMaker XGBoost), not foundation models. |
| ### Question 5 - Foundation Model Selection for Latency Constraints A gaming company wants to add a real-time Non-Player Character (NPC) dialogue system. The strict constraint is that responses must be generated in under 200 milliseconds to maintain gameplay fluidity, and per-request inference costs must be kept as low as possible. Which Amazon Bedrock strategy is the MOST appropriate? Select ONE that applies. - A) Select a smaller, latency-optimized foundation model (e.g., Amazon Titan Text Lite) and use prompt engineering. - B) Select a larger, highly capable foundation model (e.g., Anthropic Claude Opus) and use few-shot prompting. - C) Use a large foundation model and increase the max_tokens parameter to ensure faster generation. - D) Fine-tune a large foundation model to reduce the need for long prompts. |
Correct Answer: A Correct Answer Analysis: A is Correct: Smaller foundation models like Titan Text Lite have lower latency and lower per-token costs. For a real-time gaming NPC where <200ms latency is the strict constraint, choosing the smallest capable model satisfies the latency and cost tradeoffs without requiring any training overhead. Distractor Analysis: - B is Incorrect: Large models like Claude Opus have higher latency and higher costs, making them unsuitable for a sub-200ms strict latency constraint. - C is Incorrect: Increasing max_tokens allows for longer outputs but does not decrease latency; larger models still have higher time-to-first-token and overall processing times. - D is Incorrect: Fine-tuning a large model still leaves the fundamental latency bottleneck of the large model's architecture and size. It introduces unnecessary overhead. > Exam Tip: For strict low-latency and low-cost requirements, select the smallest foundation model that meets the baseline quality bar. |
| ### Question 6 - RAG vs. Fine-Tuning for Dynamic Knowledge An enterprise legal team needs a Q&A system that queries a repository of 10,000 legal contracts. The contracts are updated frequently, and the team must be able to cite the exact source document and clause for every answer. They want to minimize the time spent updating the model's knowledge base. Which approach meets these requirements with the LEAST operational overhead? Select ONE that applies. - A) Use Amazon Bedrock Knowledge Bases to ingest the contracts and retrieve relevant passages at query time. - B) Fine-tune an Amazon Bedrock foundation model on the text of the 10,000 legal contracts. - C) Train a custom NLP model on Amazon SageMaker AI using the legal documents. - D) Use Amazon Textract to extract the text and manually search using AWS Lambda. |
Correct Answer: A Correct Answer Analysis: A is Correct: Bedrock Knowledge Bases (RAG) keeps content outside the model weights. When a contract is updated, only the vector index needs re-ingesting (minutes), allowing immediate access to new info and source citations. This provides the lowest operational overhead for dynamic data requiring citations. Distractor Analysis: - B is Incorrect: Fine-tuning writes knowledge into the model weights. When contracts change, a new fine-tuning job (hours/days) is required, and fine-tuned models cannot natively cite exact source documents. - C is Incorrect: Custom training incurs massive overhead, cannot be updated quickly, and lacks native citation capabilities. - D is Incorrect: Building a custom manual search with Textract and Lambda introduces high development and operational overhead compared to a fully managed RAG service like Bedrock Knowledge Bases. > Exam Tip: If data changes frequently or requires source citations, use RAG. Fine-tuning is for behavior/formatting, not dynamic knowledge retrieval. |
| ### Question 7 - Model Interpretability vs. Complexity A healthcare provider is developing an ML model to predict patient readmission risk based on tabular electronic health records (EHR). Strict regulatory compliance requires that the model's predictions be fully explainable to medical staff. The model must also be highly cost-effective to train and deploy. Which approach is the BEST choice? Select ONE that applies. - A) Train an interpretable model like Logistic Regression or Decision Tree using Amazon SageMaker AI. - B) Fine-tune a foundation model on Amazon Bedrock using the EHR tabular data. - C) Deploy a pre-trained large language model and use prompt engineering to query patient data. - D) Use Amazon Rekognition to analyze patient charts and predict readmission. |
Correct Answer: A Correct Answer Analysis: A is Correct: Logistic Regression and Decision Trees are inherently interpretable. Medical staff can easily trace how input features map to a prediction. This meets the strict regulatory explainability constraint and is highly cost-effective for tabular data. Distractor Analysis: - B is Incorrect: Foundation models are massive "black boxes" and lack inherent interpretability. They are also inefficient and costly for purely tabular EHR data. - C is Incorrect: Same as B; LLMs are not interpretable and are not designed for high-accuracy tabular prediction. - D is Incorrect: Rekognition is for image and video analysis, not tabular health record prediction. > Exam Tip: Regulatory requirements for explainability almost always dictate traditional, interpretable ML models over deep learning or foundation models. |
| ### Question 8 - SageMaker JumpStart for Custom LLM Deployment A data science team wants to deploy an open-source Large Language Model (LLM) for a specialized text-generation task. They need full control over the underlying infrastructure and model artifacts to apply custom optimizations, but they want to minimize the initial training and setup time. Which AWS service provides the BEST balance of control and reduced setup time? Select ONE that applies. - A) Use Amazon SageMaker JumpStart to deploy a pre-trained open-source LLM. - B) Train a custom LLM from scratch using Amazon SageMaker AI training jobs. - C) Use Amazon Bedrock to access a managed foundation model. - D) Use Amazon Comprehend for text generation. |
Correct Answer: A Correct Answer Analysis: A is Correct: SageMaker JumpStart provides pre-trained open-source models that can be deployed with a few clicks. It gives the team full control over the model artifacts and underlying EC2 instances while bypassing the massive training time required to build an LLM from scratch. Distractor Analysis: - B is Incorrect: Training an LLM from scratch takes weeks or months and immense compute resources, violating the "minimize training time" constraint. - C is Incorrect: While Bedrock minimizes setup time, it is a fully managed service and does not provide the requested "full control over the underlying infrastructure and model artifacts." - D is Incorrect: Amazon Comprehend is for NLP analysis (sentiment, entities), not generative text generation. > Exam Tip: Use SageMaker JumpStart when you need to deploy pre-trained models quickly but still require infrastructure/model control that managed services (like Bedrock) don't offer. |
| ### Question 9 - Cost-Effective Batch Processing for Media A media company has an archive of 50,000 hours of video content that needs to be tagged with metadata (objects, scenes, and celebrities) for a search portal. There is no real-time requirement; the goal is to process the archive at the lowest possible cost. Which approach is the MOST cost-effective? Select ONE that applies. - A) Use Amazon Rekognition Video batch processing to analyze the video archive. - B) Stream the video through Amazon Rekognition Video real-time APIs. - C) Train a custom computer vision model on Amazon SageMaker AI. - D) Use Amazon Bedrock Knowledge Bases to index video transcripts. |
Correct Answer: A Correct Answer Analysis: A is Correct: Amazon Rekognition Video batch processing is designed for analyzing stored video libraries asynchronously. It avoids the premium costs of real-time processing infrastructure, making it the most cost-effective solution for a non-time-sensitive archive. Distractor Analysis: - B is Incorrect: Real-time APIs are optimized for low-latency live streams and carry a higher operational cost footprint, which is unnecessary since there is no real-time requirement. - C is Incorrect: Training a custom CV model from scratch to detect objects, scenes, and celebrities requires massive data labeling, training time, and compute costs. Rekognition already solves this task out-of-the-box. - D is Incorrect: Bedrock Knowledge Bases process text, not raw video files for visual metadata (objects, scenes). > Exam Tip: For non-urgent, large-scale processing, use asynchronous batch processing (e.g., Rekognition Video batch) to save costs over real-time APIs. |
| ### Question 10 - Bedrock Provisioned Throughput for Cost Control An enterprise is running a high-volume text summarization workload on Amazon Bedrock. The workload is highly predictable, running 24/7 with a constant rate of requests. The current On-Demand pricing is resulting in high costs. The team needs to reduce inference costs while maintaining the same model performance and throughput. Which action will BEST meet these requirements? Select ONE that applies. - A) Purchase Provisioned Throughput for the foundation model. - B) Increase the batch size of the summarization requests. - C) Switch to a smaller foundation model without evaluating accuracy. - D) Implement a RAG architecture with Amazon OpenSearch Serverless. |
Correct Answer: A Correct Answer Analysis: A is Correct: Provisioned Throughput in Amazon Bedrock allows you to purchase model capacity at a fixed rate. For predictable, high-volume 24/7 workloads, this is significantly more cost-effective than paying per-token On-Demand pricing, while maintaining the same model performance. Distractor Analysis: - B is Incorrect: Batch size optimization is an infrastructure tuning parameter, but On-Demand Bedrock pricing is based on tokens, not how you batch the API calls. - C is Incorrect: Switching to a smaller model without evaluation directly violates the constraint to "maintain the same model performance." - D is Incorrect: RAG adds retrieval infrastructure costs and does not reduce the inference cost of the summarization model itself. > Exam Tip: For predictable, high-volume GenAI workloads, use Bedrock Provisioned Throughput to lower costs compared to On-Demand. |