Skill 2.3.1: Perform reproducible experiments (for example, by using MLflow on SageMaker AI, Amazon Bedrock evaluations, Amazon Bedrock Prompt Management)
| Question | Answer & Explanation |
|---|---|
| ### Question 1 - SageMaker AI MLflow for Reproducible Training Runs A machine learning engineering team is training a custom XGBoost model on Amazon SageMaker AI. They need to ensure that every training run is fully reproducible, tracking all hyperparameters, metrics, and model artifacts with the least operational overhead. Compliance rules mandate that experiment data must be stored securely within their AWS account without managing underlying tracking servers. Which solution meets these requirements? (Select ONE) - A) Manually log parameters and metrics to Amazon CloudWatch Logs and save artifacts to an S3 bucket. - B) Configure Amazon SageMaker AI Managed MLflow to automatically capture run metadata, parameters, metrics, and artifacts. - C) Deploy an open-source MLflow tracking server on an Amazon EC2 instance and point SageMaker training jobs to it. - D) Use the SageMaker Model Registry to track only the model artifacts after the training job completes. |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon SageMaker AI provides a fully managed MLflow tracking server. It automatically captures and logs parameters, metrics, and artifacts during training runs, ensuring reproducibility without the operational overhead of managing the tracking infrastructure. Distractor Analysis: - A is Incorrect: Manually logging to CloudWatch and S3 introduces unnecessary custom code and lacks the unified experiment tracking and visual comparison capabilities of MLflow. - C is Incorrect: Deploying an open-source MLflow server on EC2 violates the "least operational overhead" constraint as it requires the team to provision, patch, and manage the server. - D is Incorrect: The Model Registry only tracks model artifacts for deployment, not the full lineage of hyperparameters, metrics, and code versions required for experiment reproducibility. > Exam Tip: SageMaker AI Managed MLflow provides the open-source MLflow API without the server management overhead, automatically integrating with SageMaker training jobs. |
| ### Question 2 - Amazon Bedrock Evaluations for Foundation Models An AI team is using Amazon Bedrock to evaluate two different Foundation Models (for example, Anthropic Claude and Meta Llama) for summarizing complex financial reports. They need an automated evaluation that can explain the reasoning behind the scores assigned to each response with minimal human intervention. Which evaluation method should they use? (Select ONE) - A) Programmatic evaluation using ROUGE and BLEU metrics. - B) Human evaluation using a team of internal financial domain experts. - C) LLM-as-a-judge evaluation using a more powerful model (e.g., Claude 3 Opus) to score and explain the responses. - D) Shadow deployment comparing the two models on live traffic. |
Correct Answer: C Correct Answer Analysis: - C is Correct: The LLM-as-a-judge evaluation method in Amazon Bedrock Evaluations uses a capable model to automatically score responses and provide a reasoning chain for the score. This provides qualitative assessment with minimal human intervention. Distractor Analysis: - A is Incorrect: Programmatic metrics like ROUGE and BLEU calculate n-gram overlap mathematically but do not "explain" the semantic quality of the summary. - B is Incorrect: Human evaluation requires significant manual effort and intervention, violating the "minimal human intervention" constraint. - D is Incorrect: Shadow deployment evaluates operational metrics (latency, errors) on live traffic, not the content quality of the summaries. > Exam Tip: Use LLM-as-a-judge in Bedrock Evaluations when you need automated, semantic scoring with an explanation of the reasoning, reducing the need for human raters. |
| ### Question 3 - Amazon Bedrock Prompt Management Versioning A prompt engineer is iterating on system prompts for a customer service chatbot built on Amazon Bedrock. They need to systematically test changes to the prompt wording, compare the outputs side-by-side, and maintain a history of previous versions to ensure reproducibility of prompt behavior. Which AWS service provides this capability natively? (Select ONE) - A) AWS Systems Manager Parameter Store - B) Amazon SageMaker Experiments - C) Amazon Bedrock Prompt Management - D) Amazon DynamoDB |
Correct Answer: C Correct Answer Analysis: - C is Correct: Amazon Bedrock Prompt Management natively allows users to create, save, version, and compare prompt variants. This makes prompt iterations reproducible and measurable. Distractor Analysis: - A is Incorrect: Parameter Store is for configuration data management, not specifically for iterating and evaluating GenAI prompts. - B is Incorrect: SageMaker Experiments is designed for traditional ML training runs, not for managing GenAI prompt templates. - D is Incorrect: Using DynamoDB would require building a custom application to manage and compare prompts, violating the native capability requirement. > Exam Tip: Bedrock Prompt Management turns unstructured prompt engineering into a tracked, versioned, and reproducible process, allowing side-by-side variant comparison. |
| ### Question 4 - MLflow Tracing for GenAI Workflows A GenAI application built on Amazon SageMaker AI uses a multi-step chain: retrieving context from an Amazon OpenSearch Serverless vector database, generating a response using an LLM, and formatting the output. The team needs to debug an unexpected hallucinated response by examining the exact inputs and outputs of each step in the chain. Which feature of MLflow on SageMaker AI should they implement? (Select ONE) - A) MLflow Model Registry - B) MLflow Tracing - C) MLflow Artifacts - D) MLflow Projects |
Correct Answer: B Correct Answer Analysis: - B is Correct: MLflow Tracing is specifically designed for GenAI applications to record the inputs, outputs, and metadata at each step of an execution chain, allowing engineers to trace an unexpected output back to its source step. Distractor Analysis: - A is Incorrect: Model Registry is for versioning and managing deployed model artifacts, not tracing execution steps. - C is Incorrect: MLflow Artifacts are used for storing files (like models, plots, or raw data) associated with a run, not step-by-step execution tracing. - D is Incorrect: MLflow Projects are used for packaging code to ensure reproducible runs, not for debugging individual step I/O in a GenAI chain. > Exam Tip: Use MLflow Tracing to debug GenAI applications by capturing the specific input and output payloads at each step of an orchestration chain. |
| ### Question 5 - Evaluating RAG System Components A company has built a Retrieval-Augmented Generation (RAG) application using Amazon Bedrock Knowledge Bases. They are experiencing instances where the generated answers are fluent but omit key facts. They need to use a fully managed evaluation service to measure the retrieval accuracy and the generation quality separately. Which approach should they take? (Select ONE) - A) Deploy a SageMaker shadow variant to test the RAG endpoint against live traffic. - B) Use Amazon Bedrock Evaluations to create an evaluation job that assesses both the retrieval source (Knowledge Base) and the model's response. - C) Use Amazon CloudWatch Logs to parse the input and output payloads manually. - D) Train a custom text classification model in SageMaker to detect hallucinations. |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock Evaluations can run jobs against retrieval sources (Knowledge Bases) and LLMs. It evaluates retrieval metrics (e.g., context relevance) separately from generation metrics (e.g., faithfulness), isolating the cause of omissions. Distractor Analysis: - A is Incorrect: Shadow testing measures operational performance on live traffic, not the semantic accuracy of retrieval and generation steps. - C is Incorrect: Manually parsing CloudWatch logs is operationally heavy and does not provide structured metric scoring for RAG components. - D is Incorrect: Training a custom model to detect hallucinations is a high-overhead solution when Bedrock provides native LLM-as-a-judge capabilities for faithfulness. > Exam Tip: In RAG systems, fluent but factually incorrect answers can be caused by poor retrieval or poor generation. Bedrock Evaluations helps by measuring retrieval and generation quality separately. |
| ### Question 6 - Combining Prompt Management and Evaluations A marketing team wants to test three different prompt templates for generating ad copy using Amazon Bedrock. They need to compare the outputs side-by-side and objectively select the best-performing prompt variant with the least operational overhead. Which AWS services should they combine to achieve this? (Select TWO) - A) Amazon Bedrock Prompt Management - B) Amazon Bedrock Evaluations - C) Amazon SageMaker Ground Truth - D) Amazon EMR - E) AWS Lambda |
Correct Answer: A and B Correct Answer Analysis: - A is Correct: Bedrock Prompt Management is used to create, save, and version the three distinct prompt variants. - B is Correct: Bedrock Evaluations can be configured to run an automated evaluation job (e.g., LLM-as-a-judge) against the outputs of the three prompt variants to objectively score and compare them. Distractor Analysis: - C is Incorrect: SageMaker Ground Truth is a data labeling service for creating ML datasets, introducing unnecessary overhead compared to automated evaluation. - D is Incorrect: Amazon EMR is for big data processing and is completely irrelevant to prompt testing. - E is Incorrect: Using Lambda would require writing custom orchestration and evaluation logic, violating the least operational overhead constraint. > Exam Tip: For low-overhead prompt optimization, use Bedrock Prompt Management to version templates and Bedrock Evaluations to automatically score their outputs. |
| ### Question 7 - Reproducible GenAI Experiment Tracking A data science team is building a generative AI application using a SageMaker Training Job to fine-tune a Foundation Model. They must ensure that the exact environment, hyperparameters, and data paths are captured for every run so that any model can be retrained identically. They want to avoid managing their own tracking server infrastructure. What is the most operationally efficient way to achieve this? (Select ONE) - A) Deploy an open-source MLflow tracking server on an Amazon EC2 instance. - B) Use Amazon SageMaker AI Managed MLflow, which provisions the tracking server automatically. - C) Deploy MLflow using AWS Elastic Beanstalk. - D) Store tracking metadata in Amazon RDS and use a custom Flask app to view it. |
Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker AI provides a fully managed MLflow tracking server. It automatically captures environment dependencies, hyperparameters, and data paths, allowing for identical reproducibility without server management overhead. Distractor Analysis: - A is Incorrect: Deploying on EC2 requires the team to manage OS patching, scaling, and MLflow updates, violating the operational efficiency constraint. - C is Incorrect: Elastic Beanstalk still requires managing the application layer and MLflow configuration, adding overhead. - D is Incorrect: Building a custom Flask app and using RDS requires significant development effort and infrastructure management. > Exam Tip: SageMaker AI Managed MLflow is the AWS-native, serverless-compatible way to use the open-source MLflow API for experiment tracking. |
| ### Question 8 - Human Evaluation in Bedrock An enterprise is evaluating an LLM for generating highly sensitive legal summaries. Due to the critical nature of the output, automated metrics are deemed insufficient. They require a cohort of legal experts to rate the responses based on specific compliance and safety criteria, integrated directly into their AWS GenAI workflow. Which AWS feature is best suited for this evaluation workflow? (Select ONE) - A) Amazon Bedrock Evaluations with Human workers - B) Amazon SageMaker Ground Truth Plus - C) Amazon Bedrock Prompt Management - D) Amazon Comprehend Custom Classification |
Correct Answer: A Correct Answer Analysis: - A is Correct: Amazon Bedrock Evaluations supports human evaluation workflows where domain experts can be assigned to rate LLM responses based on custom criteria (like safety or compliance). Distractor Analysis: - B is Incorrect: While Ground Truth can be used for labeling, Bedrock Evaluations is specifically designed to evaluate FM outputs within the Bedrock ecosystem, providing better native integration for GenAI workflows. - C is Incorrect: Prompt Management is for storing and versioning prompts, not for assigning human raters to evaluate outputs. - D is Incorrect: Amazon Comprehend is for NLP tasks like entity recognition or classification, not for managing human evaluation of LLM generations. > Exam Tip: When automated or LLM-as-a-judge evaluations are insufficient for sensitive domains, use Amazon Bedrock Evaluations configured for Human workers. |
| ### Question 9 - Debugging ML Model Convergence using MLflow A team is using Amazon SageMaker AI to train a deep learning model. They notice that the validation loss has been worsening over the last 20 epochs while the training loss continues to improve. They are using SageMaker AI Managed MLflow to track experiments. Based on the tracked metrics, what is the most likely issue they are experiencing? - A) The learning rate is set too high, causing the loss to oscillate. - B) The model is overfitting to the training data. - C) The training data storage format is incorrect. - D) The model lacks the capacity to learn the signal. |
Correct Answer: B Correct Answer Analysis: - B is Correct: When training loss decreases but validation loss increases (worsens), it is the classic signature of overfitting. The model is memorizing the training data rather than generalizing to unseen data. Distractor Analysis: - A is Incorrect: A learning rate that is too high typically causes the loss to oscillate wildly and never settle, rather than showing a steady decrease in training loss. - C is Incorrect: Data storage format affects I/O speed, not the divergence of training and validation loss curves. - D is Incorrect: If the model lacked capacity, the training loss would barely move or fail to decrease in the first place. > Exam Tip: Read the curves: training loss down + validation loss up = overfitting. Loss not moving = model capacity or learning rate too low. Oscillating loss = learning rate too high. |
| ### Question 10 - Selecting GenAI Evaluation Metrics A team is using Amazon Bedrock Evaluations to assess a text summarization model. They need an automated, programmatic metric that evaluates the lexical overlap of n-grams between the generated summary and a reference summary. They want to avoid using an LLM-as-a-judge to save on compute costs. Which metric is most appropriate for this programmatic evaluation? (Select ONE) - A) BERTScore - B) ROUGE - C) Semantic Similarity - D) Perplexity |
Correct Answer: B Correct Answer Analysis: - B is Correct: ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a programmatic, n-gram overlap-based metric specifically designed for evaluating text summarization by comparing the generated text to a reference. Distractor Analysis: - A is Incorrect: BERTScore uses embeddings to measure semantic similarity, not lexical n-gram overlap, and is computationally heavier. - C is Incorrect: Semantic Similarity is an embedding-based measure that compares meaning, not exact word overlap. - D is Incorrect: Perplexity measures how well a probability model predicts a sample, typically used for evaluating language modeling itself, not summarization overlap. > Exam Tip: Choose ROUGE or BLEU for cheap, fast, lexical (n-gram) overlap evaluation. Choose BERTScore or LLM-as-a-judge for semantic meaning evaluation. |