2.3.7

Question	Answer & Explanation
### Question 1 - Choosing the Right Human Evaluation Service A medical research company is building a generative AI application that summarizes patient clinical trial reports. Compliance requires that a human expert reviews the generated summaries before they are finalized and stored in the database. The team needs a service that seamlessly integrates human reviewers into the machine learning inference workflow with the least operational overhead. Which AWS service is the most appropriate to implement this human-in-the-loop (HITL) workflow? - A) Amazon SageMaker Ground Truth - B) Amazon Augmented AI (A2I) - C) Amazon Bedrock Model Evaluation - D) AWS Step Functions with Amazon Mechanical Turk	Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Augmented AI (A2I) is specifically designed to bring human review to machine learning predictions and generative AI outputs. It provides built-in workflows and integrates directly with services like SageMaker, Textract, and Comprehend, allowing you to configure trigger conditions (e.g., low confidence scores) and route the output to human reviewers with minimal operational overhead. Distractor Analysis: - A is Incorrect: SageMaker Ground Truth is primarily used for creating labeled training datasets by humans. While it can evaluate outputs, A2I is the designated service for integrating human review into active inference/prediction pipelines. - C is Incorrect: Bedrock Model Evaluation (specifically human evaluation jobs) is used for offline batch assessment of model quality using prompt datasets, not for real-time, per-request human-in-the-loop gating in an application workflow. - D is Incorrect: Building a custom Step Functions state machine with Mechanical Turk introduces significant operational overhead and custom development compared to the managed capabilities of A2I. > Key Takeaway: Use Amazon Augmented AI (A2I) when you need to insert human judgment into an active ML inference pipeline, especially when specific trigger conditions must be met before routing to a human.
### Question 2 - Bedrock Human Evaluation Requirements An AI engineer is using Amazon Bedrock Evaluations to assess the quality of a foundation model's text generation capabilities. The team decides that subjective measures, such as tone, helpfulness, and safety, require human raters rather than automated programmatic metrics. The evaluation must be set up with the least amount of custom code. When configuring a human evaluation job in Amazon Bedrock, which of the following is a mandatory requirement? - A) The evaluation dataset must contain at least 10,000 prompts to ensure statistical significance. - B) The human workers must be sourced exclusively from a private SageMaker Ground Truth workforce. - C) A browser-facing display is required because prompts and responses must be presented visually to the human raters. - D) The foundation model must be fine-tuned using SageMaker JumpStart before it can be evaluated by humans. (Select ONE)	Correct Answer: C Correct Answer Analysis: - C is Correct: When using Amazon Bedrock Evaluations with human workers, the console provides a managed UI. However, a browser-facing display configuration is inherently required because human raters must visually read the prompt, the generated response, and any reference outputs to provide subjective scores on quality, tone, or safety. Distractor Analysis: - A is Incorrect: There is no strict minimum of 10,000 prompts. Evaluation datasets can be much smaller, depending on the evaluation goals and budget for human review. - B is Incorrect: Bedrock supports both public and private workforces. You are not restricted exclusively to a private SageMaker Ground Truth workforce. - D is Incorrect: Human evaluation can be performed on base foundation models provided by Bedrock; fine-tuning via SageMaker JumpStart is not a prerequisite for evaluation. > Key Takeaway: Human evaluation jobs in Amazon Bedrock require presenting prompts and responses in a browser-facing interface, whereas automated (programmatic) or LLM-as-a-judge evaluations run completely headless in the background.
### Question 3 - SageMaker Ground Truth Plus for GenAI A startup is building a domain-specific Large Language Model (LLM) and needs to evaluate the model's responses for instruction-following accuracy and safety. The engineering team lacks the in-house expertise to build a robust human evaluation pipeline and wants to completely offload the workforce management, task UI creation, and evaluation workflow to AWS. The solution must require zero infrastructure management. Which AWS offering meets these requirements? - A) Amazon Augmented AI (A2I) with a custom Worker Task Template - B) Amazon SageMaker Ground Truth Plus - C) Amazon Bedrock Knowledge Bases with a human-in-the-loop fallback - D) AWS Lambda functions integrated with Amazon Mechanical Turk (Select ONE)	Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon SageMaker Ground Truth Plus is a fully managed turnkey data labeling and evaluation service. AWS experts manage the entire workflow, including workforce provisioning (both public and expert workforce), UI setup, and task execution, requiring zero infrastructure management from the customer. Distractor Analysis: - A is Incorrect: While A2I handles the routing of human review, the customer still has to design the Worker Task Template, set up the flow definition, and manage the workforce (private or vendor). This violates the "zero infrastructure management" and "offload workforce management" constraint. - C is Incorrect: Bedrock Knowledge Bases provides RAG capabilities but does not natively offer a managed human evaluation workforce pipeline. - D is Incorrect: Building a custom Lambda and Mechanical Turk integration requires significant custom code and infrastructure management, directly violating the constraints. > Key Takeaway: SageMaker Ground Truth Plus is the turnkey solution for teams that want AWS to manage the entire human evaluation/labeling process end-to-end, from workforce to workflow.
### Question 4 - Trigger Conditions in A2I An e-commerce company uses an ML model to classify customer support tickets by severity (Low, Medium, High). Compliance policies mandate that any ticket classified as "High" severity must be reviewed by a human agent before being routed to the engineering team. The team wants to implement this with minimal custom routing logic. How should the ML engineer configure the Amazon Augmented AI (A2I) workflow to meet this requirement? - A) Create an A2I human review workflow and define a trigger condition using the ConditionType to activate human review only when the model's output severity equals "High". - B) Use SageMaker Model Monitor to detect data drift, and trigger an AWS Lambda function to route "High" severity tickets to an A2I human loop. - C) Route all customer tickets through an A2I human loop, and use the worker UI to let the human agent filter out the "Low" and "Medium" severity tickets. - D) Train a secondary model to evaluate the output of the first model, and if the secondary model predicts a mismatch, trigger the A2I human loop. (Select ONE)	Correct Answer: A Correct Answer Analysis: - A is Correct: A2I allows you to define specific trigger conditions within the human review workflow configuration. You can set up a ConditionType (e.g., Equals or GreaterThan) that automatically triggers a human loop only when the ML model's output meets specific criteria—in this case, when the severity is "High". Distractor Analysis: - B is Incorrect: Model Monitor is used for detecting drift in production models over time, not for per-request routing based on output values. Introducing Lambda for routing adds unnecessary custom logic. - C is Incorrect: Routing every single ticket to a human loop defeats the purpose of automation and creates massive operational overhead, violating the "minimal custom routing logic" constraint. - D is Incorrect: Training and maintaining a secondary model just to evaluate the first model is over-engineered for simple condition matching and adds unnecessary latency and cost. > Key Takeaway: A2I supports declarative trigger conditions (e.g., confidence score thresholds, specific output values) that automatically initiate human review without the need for custom routing code.
### Question 5 - A2I Component Architecture A data scientist is setting up an Amazon Augmented AI (A2I) human review workflow for the first time. They need to ensure that when the ML model's confidence score falls below 80%, the output is sent to an internal team of reviewers using a custom web interface. Which TWO components must the data scientist create to successfully establish this human-in-the-loop pipeline? (Select TWO) - A) A Flow Definition that specifies the reviewer workforce and the trigger conditions. - B) A Worker Task Template that defines the HTML/JavaScript UI presented to the human reviewers. - C) A SageMaker Processing Job to handle the aggregation of the human responses. - D) A SageMaker Model Monitor baseline to track the 80% confidence threshold. - E) An Amazon Bedrock Guardrail to intercept low-confidence scores.	Correct Answer: A and B Correct Answer Analysis: - A is Correct: The Flow Definition (created via CreateFlowDefinition API) is the core configuration in A2I. It links the workforce (e.g., private team), the human task UI, and the trigger conditions that determine when a human loop is initiated. - B is Correct: The Worker Task Template (created via CreateHumanTaskUi API) contains the HTML/CSS/JS code that renders the interface the human worker sees when they are assigned a review task. Distractor Analysis: - C is Incorrect: A2I automatically stores the human-reviewed results in Amazon S3. A separate SageMaker Processing Job is not required for basic aggregation of human responses. - D is Incorrect: SageMaker Model Monitor is used for ongoing model quality and drift monitoring, not for defining the A2I confidence threshold trigger. - E is Incorrect: Bedrock Guardrails are used to filter harmful content in generative AI applications, not to manage A2I human review workflows for confidence thresholds. > Key Takeaway: An A2I workflow requires at minimum a Flow Definition (to manage routing and workers) and a Worker Task Template (to render the UI for the human raters).
### Question 6 - Evaluating Text Generation Quality An ML team is evaluating an LLM's performance on a summarization task. They have thousands of generated summaries and want a comprehensive evaluation that balances automated scoring with human judgment. Automated metrics like ROUGE will be used, but the team also needs human raters to assess subjective qualities like fluency and coherence. The solution must scale efficiently and provide a unified evaluation environment. Which AWS service should the team use? - A) AWS Lambda functions computing ROUGE scores, combined with Amazon SES to email summaries to domain experts. - B) Amazon Bedrock Evaluations, configuring a single evaluation job with both programmatic metrics and a human evaluation dataset. - C) Amazon Augmented AI (A2I) configured to route every single generated summary to a human worker, while a Lambda function computes ROUGE in parallel. - D) Amazon SageMaker Ground Truth Plus, manually managing the ROUGE score calculation outside of AWS. (Select ONE)	Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock Evaluations allows you to run evaluation jobs that mix automated programmatic metrics (like ROUGE, BERTScore) and human evaluation within a single unified framework. This provides a cohesive report comparing models or prompt variants across both objective and subjective metrics. Distractor Analysis: - A is Incorrect: Using Lambda and SES is a highly manual, disjointed approach that lacks tracking, aggregation, and a unified reporting interface. - C is Incorrect: A2I is designed for real-time or near-real-time inference gating, not for offline batch evaluation of thousands of samples alongside programmatic metrics. - D is Incorrect: Ground Truth Plus handles the human side, but does not natively compute programmatic NLP metrics like ROUGE in the same unified evaluation job like Bedrock Evaluations does. > Key Takeaway: Amazon Bedrock Evaluations is the unified hub for assessing GenAI models, capable of running programmatic, LLM-as-a-judge, and human evaluations together in a single job.
### Question 7 - Sourcing Human Workforces in AWS A financial institution is implementing a human-in-the-loop workflow using Amazon SageMaker Ground Truth to evaluate the output of a custom fraud detection model. Due to strict data privacy regulations, the reviewers must be employees of the bank who have passed background checks, and the evaluation data cannot leave the company's AWS environment. Which type of workforce should the ML engineer configure in SageMaker Ground Truth? - A) A Public Workforce (Amazon Mechanical Turk) - B) A Vendor Workforce (third-party data labeling companies listed on the AWS Marketplace) - C) A Private Workforce (using Amazon Cognito) - D) An Automated Workforce (using AWS Lambda functions to simulate human clicks) (Select ONE)	Correct Answer: C Correct Answer Analysis: - C is Correct: A Private Workforce in SageMaker Ground Truth allows you to use your own employees as human reviewers. It is managed via Amazon Cognito (or OIDC integration) to authenticate internal users, ensuring that sensitive data stays completely within your control and meets strict privacy regulations. Distractor Analysis: - A is Incorrect: A Public Workforce (Mechanical Turk) consists of external, anonymous gig workers, which violates the privacy and security constraints of the financial institution. - B is Incorrect: A Vendor Workforce uses third-party companies. While they may have some security clearances, they are not bank employees, violating the requirement. - D is Incorrect: An "Automated Workforce" is not a valid concept for human-in-the-loop evaluation, and using Lambda would defeat the purpose of introducing human judgment. > Key Takeaway: Use a SageMaker Private Workforce (backed by Amazon Cognito or your corporate IdP via OIDC) when data privacy policies mandate that only internal employees can view the data.
### Question 8 - Content Quality and Bias Validation A generative AI application is generating marketing copy for a diverse audience. The AI team wants to continuously evaluate the model's outputs for potential demographic bias and harmful content. The evaluation must combine automated toxicity detection with subjective human assessment of cultural sensitivity. The team wants to use AWS services to implement this most effectively. Which combination of services should the team use? (Select TWO) - A) Amazon Comprehend to detect toxic or harmful text automatically. - B) Amazon Bedrock Guardrails to block harmful content during inference, completely replacing the need for human evaluation. - C) Amazon Bedrock Evaluations to configure a human evaluation job specifically rating bias and cultural sensitivity. - D) Amazon Rekognition to extract text from images of the marketing copy. - E) AWS IAM policies to restrict the LLM from generating biased content.	Correct Answer: A and C Correct Answer Analysis: - A is Correct: Amazon Comprehend provides built-in APIs (such as DetectToxicContent) that can automatically score text for toxicity and harmful content, serving as the programmatic evaluation layer. - C is Correct: Amazon Bedrock Evaluations can be used to set up an evaluation job where human raters review the marketing copy for subjective measures like demographic bias, cultural sensitivity, and tone, integrating their feedback into a unified report. Distractor Analysis: - B is Incorrect: While Bedrock Guardrails block harmful content in real-time, they do not "completely replace" human evaluation. Guardrails are filters, whereas human evaluation is needed for nuanced subjective assessment (e.g., cultural sensitivity) that automated filters often miss. - D is Incorrect: Rekognition is for image and video analysis, not text-based bias evaluation. - E is Incorrect: IAM policies control access to AWS resources and API calls; they cannot semantically restrict the specific words or biases an LLM generates. > Key Takeaway: Effective bias and content quality validation combines programmatic toxicity detection (e.g., Amazon Comprehend) with subjective human review (e.g., Amazon Bedrock Evaluations).
### Question 9 - Step Functions and A2I Integration A media company is building a content moderation pipeline. When a user uploads an image, an AWS Lambda function invokes a SageMaker endpoint to predict if the image contains explicit content. If the model's confidence is between 60% and 80% (an ambiguous "gray area"), the image must be sent to a human reviewer. The team wants to orchestrate this entire stateful workflow, including waiting for the human to complete their review before finalizing the result, with minimal custom polling code. Which AWS service provides native support to orchestrate this human-in-the-loop workflow? - A) AWS Batch - B) Amazon EventBridge Scheduler - C) AWS Step Functions - D) Amazon SWF (Simple Workflow Service) (Select ONE)	Correct Answer: C Correct Answer Analysis: - C is Correct: AWS Step Functions has native integration with Amazon Augmented AI (A2I). You can use the Human-in-the-Loop task state in Step Functions (using the A2I service integration) to start a human loop. Step Functions will natively pause the execution and wait for the human reviewer to complete their task before moving to the next step, eliminating the need for custom polling code. Distractor Analysis: - A is Incorrect: AWS Batch is for running batch computing workloads, not for orchestrating real-time human-in-the-loop workflows with individual state tracking. - B is Incorrect: EventBridge Scheduler can trigger tasks on a schedule, but it does not maintain state or natively wait for asynchronous human review completion. - D is Incorrect: Amazon SWF is a legacy service largely superseded by Step Functions. Step Functions offers native A2I integration, whereas SWF would require custom deciders and polling logic. > Key Takeaway: AWS Step Functions provides a native service integration for Amazon Augmented AI (A2I), allowing workflows to pause asynchronously until a human reviewer completes their task.
### Question 10 - Human Review Output Storage An ML engineer has successfully deployed an Amazon Augmented AI (A2I) workflow. Human reviewers are successfully reviewing low-confidence predictions from a SageMaker model. The downstream application needs to access the final, human-reviewed labels to update the database. Where does A2I automatically store the output of the human review loops, and how can the application retrieve it? - A) The output is stored in Amazon DynamoDB, and the application polls DynamoDB Streams. - B) The output is stored in an Amazon S3 bucket, and the application retrieves the JSON object from the S3 path provided in the A2I response. - C) The output is stored in Amazon CloudWatch Logs, and the application uses Logs Insights to query the results. - D) The output is stored directly inside the SageMaker model container, and the application queries the SageMaker Endpoint. (Select ONE)	Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Augmented AI (A2I) automatically stores the output of completed human loops (including the original input, the model's prediction, and the human's response) as a JSON object in the Amazon S3 bucket specified during the Flow Definition creation. The application can retrieve this output directly from S3. Distractor Analysis: - A is Incorrect: A2I does not natively write to DynamoDB. It writes to S3. - C is Incorrect: CloudWatch Logs are used for operational logging and metrics, not for storing the payload of human review results. - D is Incorrect: The human review output is completely decoupled from the SageMaker model container. It is persisted in S3. > Key Takeaway: A2I persists all human review results (input, model output, human output, and metadata) as JSON files in an Amazon S3 bucket that you designate during the workflow setup.