Skill 2.3.6: Apply comprehensive model evaluation techniques for traditional ML and GenAI models
| Question | Answer & Explanation |
|---|---|
| ### Question 1 - Reproducible Experiments with MLflow A data science team is training a custom XGBoost model on Amazon SageMaker AI. They notice that two runs of the same training configuration yield different evaluation metrics. They need a solution to record each run's parameters, metrics, and artifacts to compare them later with the least operational overhead. Which solution meets these requirements? - A) Use SageMaker Model Monitor to track the model's performance over time. - B) Use Amazon SageMaker AI with MLflow to track the training runs. - C) Write custom logging logic to push metrics to Amazon CloudWatch Logs. - D) Use Amazon SageMaker Shadow Tests to compare the two training runs. |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon SageMaker AI with MLflow is a fully managed service that automatically tracks experiments, recording parameters, metrics, and artifacts for each run. This allows teams to easily diff and compare runs to identify sources of variance with minimal operational overhead. Distractor Analysis: - A is Incorrect: SageMaker Model Monitor is used for monitoring models in production for data drift, not for tracking training experiment runs. - C is Incorrect: Writing custom logging logic to CloudWatch introduces unnecessary operational overhead and does not natively version artifacts or parameters in an experiment tracker format. - D is Incorrect: SageMaker Shadow Tests are used to compare a candidate model against a production model using live traffic, not for comparing offline training runs. > Exam Tip / AWS Best Practice: Use SageMaker MLflow to ensure reproducibility by tracking parameters, metrics, and artifacts across all experiment runs. |
| ### Question 2 - Evaluating GenAI Models with Bedrock An ML engineer is developing a text summarization application using an Amazon Bedrock foundation model. They need to evaluate the model's output for factual consistency and relevance against a golden dataset without provisioning any infrastructure. The solution must support automated scoring. Which approach should the engineer take? - A) Deploy a shadow variant in SageMaker AI and compare latency metrics. - B) Configure SageMaker Model Monitor to detect data drift on the inputs. - C) Create an Amazon Bedrock Evaluation job using programmatic metrics. - D) Implement a custom AWS Lambda function to compute exact-match accuracy. |
Correct Answer: C Correct Answer Analysis: - C is Correct: Amazon Bedrock Evaluations supports running evaluation jobs with programmatic metrics (like ROUGE or BERTScore) automatically. This requires no infrastructure provisioning and is specifically designed for evaluating GenAI models against custom datasets. Distractor Analysis: - A is Incorrect: Shadow variants in SageMaker are for comparing models on live traffic, not for evaluating summarization quality against a golden dataset. - B is Incorrect: Model Monitor tracks drift in production data, not the semantic quality of text generation. - D is Incorrect: Exact-match accuracy is not suitable for text summarization, as valid summaries can use different wording. Writing custom Lambda code also violates the least operational overhead constraint. > Exam Tip / AWS Best Practice: Use Amazon Bedrock Evaluations with programmatic metrics for automated, infrastructure-free assessment of LLM outputs. |
| ### Question 3 - SageMaker Shadow Tests A team has trained a new recommendation model and needs to compare its performance against the currently deployed production model. The evaluation must use live inference requests, but end users must only receive responses from the production model to minimize risk. The solution must require the least operational overhead. Which approach meets these requirements? - A) Configure an A/B test with a 50/50 traffic split between the models. - B) Deploy the new model as a shadow variant on the existing SageMaker endpoint. - C) Route 10% of traffic to the new model using SageMaker Inference Recommender. - D) Export live requests to S3 and run a batch transform job offline. |
Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker Shadow Testing deploys the new model alongside the production model on the same endpoint. It routes a copy of live traffic to the shadow variant, but only the production variant returns responses to the caller. This allows testing on live requests with zero risk to the end-user experience. Distractor Analysis: - A is Incorrect: An A/B test splits traffic, meaning 50% of users will receive responses from the unvalidated candidate model, violating the risk constraint. - C is Incorrect: Inference Recommender is used to find the optimal instance type and configuration for a model, not for shadow testing against live traffic. - D is Incorrect: Exporting to S3 and running Batch Transform is an offline evaluation method, not an evaluation using live requests. > Exam Tip / AWS Best Practice: Use SageMaker Shadow Tests to validate candidate models against live traffic copies without impacting the production user experience. |
| ### Question 4 - NLP Evaluation Metrics A company is building a translation service using a generative AI model. The engineering team needs to evaluate the quality of the generated translations against human reference translations. They want to use an N-gram overlap metric that is commonly used for translation tasks. Which metric is most appropriate for this use case? - A) BERTScore - B) ROUGE - C) BLEU - D) Perplexity |
Correct Answer: C Correct Answer Analysis: - C is Correct: BLEU (Bilingual Evaluation Understudy) is a precision-focused N-gram overlap metric standard for evaluating machine translation tasks against human reference translations. Distractor Analysis: - A is Incorrect: BERTScore uses embeddings to measure semantic similarity, not exact N-gram overlap. - B is Incorrect: ROUGE is a recall-focused N-gram overlap metric primarily used for summarization tasks, not translation. - D is Incorrect: Perplexity measures how well a language model predicts a sequence of words, used for evaluating language models intrinsically, not translation outputs. > Exam Tip / AWS Best Practice: Match the NLP metric to the task: BLEU for Translation, ROUGE for Summarization, and BERTScore for semantic similarity. |
| ### Question 5 - RAG System Evaluation An enterprise is using Amazon Bedrock Knowledge Bases for a RAG application. Users report that the system is generating highly fluent but factually incorrect answers that contradict the source documents. The team needs to identify whether the issue is with the retrieval step or the generation step with the least operational overhead. Which action will isolate the cause? - A) Monitor the token count of the generated answers using CloudWatch. - B) Evaluate retrieval accuracy (Context Recall) and generation faithfulness separately using Bedrock Evaluations. - C) Replace the foundation model with a larger one and check if the issue persists. - D) Use SageMaker Model Monitor to detect data drift on the input prompts. |
Correct Answer: B Correct Answer Analysis: - B is Correct: Evaluating retrieval metrics (e.g., Context Recall) and generation metrics (e.g., Faithfulness) separately isolates whether the root cause is poor vector search retrieval or the LLM ignoring the retrieved context. Bedrock Evaluations supports this natively. Distractor Analysis: - A is Incorrect: Token count measures output length, not factual accuracy or retrieval precision. - C is Incorrect: Replacing the model is a guess-and-check approach that doesn't diagnose the specific failing component (retrieval vs. generation). - D is Incorrect: Model Monitor is for traditional ML data drift, not for RAG pipeline evaluation metrics like faithfulness or context recall. > Exam Tip / AWS Best Practice: In RAG systems, always split evaluation into Retrieval Metrics (Context Precision/Recall) and Generation Metrics (Faithfulness/Answer Relevance) to isolate the root cause of failures. |
| ### Question 6 - Debugging Convergence A machine learning engineer is training a deep neural network on Amazon SageMaker AI. The training logs show that the loss function value oscillates wildly across epochs and never settles to a minimum. The engineer needs to adjust a hyperparameter to resolve this convergence issue. Which hyperparameter adjustment is the FIRST step to take? - A) Decrease the learning rate. - B) Increase the number of layers in the network. - C) Change the batch size to 1. - D) Increase the L2 regularization parameter. |
Correct Answer: A Correct Answer Analysis: - A is Correct: Oscillating loss curves during gradient descent typically indicate that the learning rate is set too high. The weight updates repeatedly overshoot the local minimum. Decreasing the learning rate helps stabilize the optimization process. Distractor Analysis: - B is Incorrect: Increasing network capacity (layers) would not fix an unstable loss curve caused by an aggressive learning rate. - C is Incorrect: While a batch size of 1 (stochastic gradient descent) introduces noise, the primary cause of wild oscillation across epochs is an excessively high learning rate. - D is Incorrect: L2 regularization is used to prevent overfitting (when training loss improves but validation worsens), not to fix oscillating training loss. > Exam Tip / AWS Best Practice: If loss oscillates and won't settle, the learning rate is too high. If loss barely moves, the learning rate is too low. If training improves but validation worsens, the model is overfitting. |
| ### Question 7 - Baselines and Drift Detection An e-commerce company has deployed a fraud detection model on SageMaker AI. The data science team needs to establish a baseline of known-good behavior and monitor the model in production for data drift (changes in input data distribution) with the least operational overhead. Which AWS service should the team use? - A) Use SageMaker Model Monitor to capture baseline statistics and schedule monitoring jobs. - B) Manually export inference data to S3 and run statistical tests using a SageMaker Processing job. - C) Deploy a shadow variant to track prediction drift on live traffic. - D) Use AWS CloudTrail to monitor API call distributions for the SageMaker endpoint. |
Correct Answer: A Correct Answer Analysis: - A is Correct: SageMaker Model Monitor automatically captures baseline statistics from the training data and continuously monitors live inference data against this baseline to detect data drift and prediction drift, sending alerts to CloudWatch. Distractor Analysis: - B is Incorrect: Manually exporting data and running custom Processing jobs introduces unnecessary operational overhead compared to the automated Model Monitor. - C is Incorrect: Shadow variants are used for comparing a new model against a production model, not for monitoring data drift. - D is Incorrect: CloudTrail logs API management events, not the payload distributions of inference requests. > Exam Tip / AWS Best Practice: Use SageMaker Model Monitor to establish baselines and continuously monitor for data and model quality drift in production. |
| ### Question 8 - Prompt Management A prompt engineer is iterating on different wording for a system prompt used in an Amazon Bedrock application. They need a way to store prompts with variables, version them, and compare the performance of different prompt variants side-by-side with the least operational overhead. Which AWS service fulfills these requirements? - A) Use Amazon SageMaker AI with MLflow. - B) Use Amazon Bedrock Prompt Management. - C) Create an AWS Lambda function to store prompts in Amazon DynamoDB. - D) Use Amazon Bedrock Knowledge Bases. |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock Prompt Management provides native capabilities to create, save, and version prompts. It allows engineers to inject variables and dynamically compare the outputs of different prompt variants to measure performance. Distractor Analysis: - A is Incorrect: MLflow is designed for tracking traditional ML experiments (hyperparameters, metrics) and generative AI traces, but it is not a dedicated prompt authoring and versioning tool. - C is Incorrect: Building a custom Lambda and DynamoDB solution violates the least operational overhead constraint and lacks native prompt comparison features. - D is Incorrect: Bedrock Knowledge Bases is used for RAG implementations, not for prompt versioning and comparison. > Exam Tip / AWS Best Practice: Use Amazon Bedrock Prompt Management to version control prompts, inject variables, and compare prompt variants side-by-side. |
| ### Question 9 - LLM-as-a-Judge Evaluation An AI team wants to evaluate the quality of responses generated by their LLM application. They need a scalable and automated way to score the outputs for helpfulness and safety without requiring manual human review for every response. The solution must use AWS native services. Which evaluation method is most appropriate? - A) Use Amazon Bedrock Evaluations with a judge model. - B) Implement a human-in-the-loop workflow using Amazon Mechanical Turk. - C) Use SageMaker Model Monitor with a custom metric for text similarity. - D) Deploy an A/B test to measure user click-through rates. |
Correct Answer: A Correct Answer Analysis: - A is Correct: Amazon Bedrock Evaluations supports using an LLM as a judge (a separate foundation model) to automatically score outputs based on criteria like helpfulness, safety, and factual accuracy, without human intervention. Distractor Analysis: - B is Incorrect: A human-in-the-loop workflow does not meet the requirement for a scalable, automated solution without manual review. - C is Incorrect: SageMaker Model Monitor is not designed for evaluating semantic text quality or safety of LLM outputs. - D is Incorrect: A/B testing measures user behavior (like click-through rates), not the objective quality, helpfulness, or safety of the generated text. > Exam Tip / AWS Best Practice: Use the "LLM-as-a-judge" mode in Amazon Bedrock Evaluations to automatically scale the assessment of subjective text qualities like helpfulness and safety. |
| ### Question 10 - Integrated Human Evaluation Frameworks A healthcare startup is developing a GenAI application that generates patient summaries from medical records. Because of the high-stakes nature of the domain, every generated summary must be reviewed by a medical expert before being sent to the physician. The team needs to integrate this human-in-the-loop workflow efficiently. Which approach meets these requirements? - A) Use Amazon Bedrock Evaluations with human workers to rate the summaries. - B) Deploy the model as a shadow variant and manually review outputs. - C) Use SageMaker AI with MLflow to track the summary generation runs. - D) Set up an A/B test between the GenAI model and a human writer. |
Correct Answer: A Correct Answer Analysis: - A is Correct: Amazon Bedrock Evaluations supports human evaluation workflows, where human workers (subject matter experts) can review and rate model outputs. This integrates human judgment into the evaluation pipeline for high-stakes content validation. Distractor Analysis: - B is Incorrect: Shadow variants are for comparing models on live traffic; they do not facilitate a human review gatekeeping workflow for generated text. - C is Incorrect: MLflow tracks parameters and execution traces, but it does not provide a UI for human workers to review and rate generated summaries. - D is Incorrect: An A/B test exposes users to different outputs, violating the constraint that every summary must be reviewed before reaching the physician. > Exam Tip / AWS Best Practice: For high-stakes GenAI applications requiring nuanced judgment, use Amazon Bedrock Evaluations' human worker integration to validate model outputs before deployment or delivery. |