Skill 1.3.2: Label and annotate data

Question	Answer & Explanation
### Question 1 - SageMaker Ground Truth & Automated Labeling An autonomous vehicle company needs to label millions of images for object detection (bounding boxes). They have a small internal team of expert annotators and need to minimize labeling costs while ensuring high accuracy. The pipeline must be built using AWS native architecture, with a constraint of achieving the most cost-effective solution with the least operational overhead. Which solution meets these requirements? (Select ONE) - A) Use a public workforce for all images and have the internal team manually review every single label. - B) Set up an Amazon SageMaker Ground Truth labeling job using an automated labeling mechanism with a private workforce for the difficult cases. - C) Create a custom AWS Lambda function to distribute images to the internal team and collect their bounding box JSON outputs via Amazon SQS. - D) Use Amazon Rekognition Custom Labels to auto-annotate all images, bypassing human review entirely.	Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon SageMaker Ground Truth supports automated data labeling (active learning). It uses a machine learning model to auto-label the "easy" data and sends only the "difficult" or low-confidence cases to human annotators (the private workforce). This dramatically reduces costs and human effort while maintaining high accuracy. Distractor Analysis: - A is Incorrect: Using a public workforce for all images and manually reviewing every single label is extremely expensive and introduces massive operational overhead. - C is Incorrect: Building a custom Lambda and SQS pipeline to manage label distribution and collection introduces unnecessary operational overhead compared to using SageMaker Ground Truth's native capabilities. - D is Incorrect: While Rekognition can detect objects, relying solely on an auto-annotation model without human-in-the-loop validation for edge cases is not viable for high-accuracy autonomous vehicle requirements. > Key Takeaway: SageMaker Ground Truth's automated data labeling (active learning) optimizes cost by only sending low-confidence data points to human workers.
### Question 2 - Inter-Annotator Agreement & Labeling Instructions A healthcare startup is labeling medical text to identify adverse drug reactions. Multiple annotators are consistently classifying the same sentences differently (e.g., categorizing "severe nausea" as either "moderate" or "severe"). The ML team must improve label consistency across the dataset. The solution must address the root cause of the issue without compromising dataset volume. Which action should the team take to fulfill these requirements? (Select ONE) - A) Add more annotators to the team to outvote the incorrect ones on ambiguous sentences. - B) Remove all ambiguous sentences from the dataset to avoid confusion during model training. - C) Refine the labeling instructions to include clear, mutually exclusive definitions and edge-case examples for severity levels. - D) Use Amazon Comprehend Medical to auto-label all adverse drug reactions instead of relying on humans.	Correct Answer: C Correct Answer Analysis: - C is Correct: Persistent disagreement among annotators indicates ambiguous labeling instructions. Refining the instructions with mutually exclusive definitions and explicit edge-case examples directly resolves low inter-annotator agreement at the root cause. Distractor Analysis: - A is Incorrect: Adding more annotators to outvote incorrect labels treats a symptom rather than the root cause; the underlying instruction ambiguity will still pollute the dataset. - B is Incorrect: Removing ambiguous sentences reduces dataset volume and biases the dataset away from difficult edge cases, which the model will ultimately encounter in production. - D is Incorrect: While Comprehend Medical can detect entities, it does not replace the need for custom human labeling guidelines when creating a bespoke training dataset for specific severity classifications. > Key Takeaway: If annotators consistently disagree on a class boundary, the labeling instructions are ambiguous; fix the instructions rather than firing the annotators or dropping the data.
### Question 3 - Measuring Annotator Reliability A retail company is using Amazon SageMaker Ground Truth to label customer reviews as positive, neutral, or negative. The ML team wants to measure the reliability of the annotations before using the dataset for model training. The constraint is to use a statistically valid metric specifically designed for inter-annotator agreement. Which technique should the team use? (Select ONE) - A) Calculate the overall accuracy of the dataset against a held-out test set. - B) Measure the Cohen's Kappa coefficient between annotator pairs. - C) Run a profiling job in AWS Glue DataBrew to detect outliers in the label column. - D) Train a baseline model and check its validation accuracy.	Correct Answer: B Correct Answer Analysis: - B is Correct: Cohen's Kappa is a standard statistical metric used to measure inter-rater reliability (agreement between two annotators) for categorical items. It accounts for agreement occurring by chance, making it objectively valid for measuring annotator consistency. Distractor Analysis: - A is Incorrect: "Accuracy of the dataset" is not a defined metric for annotation quality, as there is no ground truth available prior to model training. - C is Incorrect: AWS Glue DataBrew profiles data distributions and missing values, but it does not calculate inter-annotator reliability metrics. - D is Incorrect: Training a baseline model to check accuracy evaluates downstream model performance, not the reliability of the human labeling process itself. > Key Takeaway: To validate label quality before training, measure inter-annotator agreement using metrics like Cohen's Kappa, rather than waiting for downstream model metrics.
### Question 4 - Label Quality Ceiling on Model Performance A data science team is frustrated that their image classification model plateaus at 85% accuracy despite trying multiple state-of-the-art neural network architectures and increasing training epochs. They inspect the training data and find that 15% of the images were labeled incorrectly by the initial annotation team. They need to improve model performance. What is the MOST effective solution? (Select ONE) - A) Increase the learning rate and use a larger batch size to force the model to ignore the noisy labels. - B) Add more layers to the neural network to increase its capacity to memorize the incorrect labels. - C) Improve the quality of the dataset by re-labeling the incorrect examples, as label quality sets a ceiling on model quality. - D) Apply dropout regularization to prevent the model from overfitting to the incorrect labels.	Correct Answer: C Correct Answer Analysis: - C is Correct: Label quality sets a hard ceiling on model quality that no amount of architectural tuning or training compute can raise. Re-labeling the incorrect data is the only objective way to surpass the 85% plateau. Distractor Analysis: - A is Incorrect: Increasing the learning rate will not selectively ignore noisy labels; it will likely cause unstable convergence and degrade overall performance. - B is Incorrect: Adding more layers to memorize incorrect labels worsens the problem by reinforcing the noise into the model's learned weights. - D is Incorrect: Dropout regularization prevents overfitting to training data in general, but it cannot teach the model the correct pattern when the labels themselves are fundamentally wrong. > Key Takeaway: No amount of model architecture tuning can fix a fundamentally flawed dataset; label quality defines the maximum achievable model performance.
### Question 5 - Active Learning for Cost Reduction A natural language processing (NLP) team needs to label a massive dataset of 1 million text messages for sentiment analysis. They have a strict budget constraint and must minimize the number of human-annotated records while ensuring the dataset is highly accurate. The architecture must use AWS native services. Which approach is the MOST cost-effective? (Select ONE) - A) Manually label 10,000 records, train a model, and use it to auto-label the remaining 990,000 records without human review. - B) Configure an Amazon SageMaker Ground Truth automated data labeling job, which uses active learning to send only difficult examples to humans and auto-labels the rest. - C) Use Amazon Mechanical Turk directly to label all 1 million messages at the lowest possible price per message. - D) Use AWS Glue DataBrew to categorize the text messages based on keyword matching, skipping human annotation entirely.	Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker Ground Truth's automated data labeling uses active learning. It automatically trains a model on a small labeled subset and uses it to label the rest. Only low-confidence (difficult) examples are routed to human workers, minimizing human labor and cost. Distractor Analysis: - A is Incorrect: Auto-labeling the remaining 990,000 records without any human review or active learning validation introduces massive label noise and bias. - C is Incorrect: Using Mechanical Turk directly for all 1 million messages is the most expensive option and does not leverage active learning to reduce human labor. - D is Incorrect: Keyword matching via DataBrew cannot understand sentiment context accurately and skips the supervised learning requirement entirely. > Key Takeaway: Active learning in SageMaker Ground Truth minimizes human labeling costs by auto-labeling high-confidence data and routing only low-confidence edge cases to humans.
### Question 6 - Workforce Selection for Compliance A financial institution is developing a supervised ML model to classify sensitive loan application documents. Strict compliance regulations require that no personally identifiable information (PII) from the documents leaves the company's secure AWS environment or is viewed by external public parties. The labeling workflow must enforce zero external data retention. Which workforce configuration should the ML engineer use in Amazon SageMaker Ground Truth? (Select ONE) - A) Use the Amazon SageMaker Ground Truth public workforce to quickly scale the labeling process. - B) Use a third-party vendor workforce integrated with SageMaker Ground Trust. - C) Set up an Amazon SageMaker Ground Truth private workforce using AWS IAM Identity Center for internal employees. - D) Use the public workforce but apply Amazon Bedrock Guardrails to redact all PII before sending the documents.	Correct Answer: C Correct Answer Analysis: - C is Correct: A private workforce in SageMaker Ground Truth uses internal employees or contracted workers who authenticate via AWS IAM Identity Center or Amazon Cognito. The data remains strictly within the enterprise's secure AWS environment, satisfying the compliance requirement. Distractor Analysis: - A is Incorrect: The public workforce consists of external Amazon Mechanical Turk workers, which violates the constraint that data cannot be viewed by external parties. - B is Incorrect: A third-party vendor workforce still involves external parties and may not guarantee the strict zero external data retention required by financial compliance rules. - D is Incorrect: While Bedrock Guardrails can redact PII, sending sensitive loan documents to public workers still violates the strict mandate that PII cannot leave the secure AWS environment or be viewed externally. > Key Takeaway: For strict data privacy and compliance requirements, use a SageMaker Ground Truth Private Workforce rather than public or vendor workforces.
### Question 7 - Consolidating Bounding Box Annotations A drone startup is using Amazon SageMaker Ground Truth to label objects in aerial images. To improve accuracy, they assign 3 human annotators to each image. The ML team needs to combine these 3 separate annotations into a single, high-confidence bounding box for each object. The constraint is to implement this with the least operational overhead using native AWS features. Which approach satisfies the requirement? (Select ONE) - A) Write a custom Python script in an AWS Lambda function to calculate the average bounding box coordinates after the Ground Truth job completes. - B) Use the bounding box consolidation algorithm built into Amazon SageMaker Ground Truth. - C) Train a secondary ML model to predict the "true" bounding box based on the three human-drawn boxes. - D) Have a single senior reviewer manually inspect every image to choose the best bounding box from the three.	Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker Ground Truth natively supports annotation consolidation. For bounding boxes, it has a built-in algorithm that merges multiple worker annotations into a single, high-confidence label without requiring custom code. Distractor Analysis: - A is Incorrect: Writing a custom Lambda function introduces unnecessary operational overhead since Ground Truth already handles this natively. - C is Incorrect: Training a secondary ML model for consolidation is an overly complex, high-overhead solution for a problem solved natively by the service. - D is Incorrect: Manual inspection by a senior reviewer defeats the purpose of using a distributed 3-annotator workflow and creates a severe operational bottleneck. > Key Takeaway: SageMaker Ground Truth provides built-in annotation consolidation algorithms (e.g., majority voting for text, bounding box merging for images) to combine multiple worker labels natively.
### Question 8 - Small vs Large Datasets & Label Quality An agricultural tech company wants to build a model to identify rare plant diseases. They have two options for training data: a dataset of 50,000 images labeled quickly by a crowdsourced platform with minimal instructions, or a dataset of 5,000 images carefully labeled by expert botanists using strict guidelines. The constraint is to achieve the highest possible model quality. Which dataset should the team choose and why? (Select ONE) - A) Choose the 50,000-image dataset because deep learning models always perform better with more data, regardless of label noise. - B) Choose the 5,000-image dataset because a small, carefully labeled dataset can outperform a much larger one labeled loosely. - C) Combine both datasets and let the model's loss function naturally down-weight the loosely labeled data. - D) Choose the 50,000-image dataset and use heavy dropout regularization to fix the label noise.	Correct Answer: B Correct Answer Analysis: - B is Correct: Label quality sets a ceiling on model quality. A smaller dataset with precise, carefully defined labels (especially for domain-specific tasks like rare plant diseases) will provide a cleaner signal for the model, outperforming a larger dataset with high label noise. Distractor Analysis: - A is Incorrect: Deep learning models do not always perform better with noisy data; 15%+ label noise in a large dataset can significantly degrade model convergence and boundary definitions. - C is Incorrect: Standard loss functions (like Cross-Entropy) do not "naturally down-weight" noisy labels without specialized robust loss techniques; the noisy labels will still corrupt the training gradient. - D is Incorrect: Dropout prevents overfitting to the training data, but it cannot teach the model the correct feature mappings when the labels are fundamentally wrong. > Key Takeaway: For specialized domains, a small, meticulously labeled dataset is often superior to a large, loosely labeled one because model quality cannot exceed label quality.
### Question 9 - Labeling Workflow Setup (Ground Truth Plus) An enterprise needs to quickly stand up a complex multi-modal data labeling pipeline (text, images, and video) for a new generative AI model. They lack the internal resources to manage a labeling workforce, write custom HTML UIs, and set up the AWS infrastructure. The constraint is to use the solution with the least operational overhead for the enterprise's ML team. Which AWS service fulfills this requirement? (Select ONE) - A) Set up an Amazon EMR cluster to process the images and use PySpark to assign labels automatically. - B) Use Amazon SageMaker Ground Truth Plus, which provides an expert workforce and manages the entire labeling workflow without the need to build infrastructure. - C) Manually create an Amazon SageMaker Ground Truth labeling job using the public workforce and custom HTML templates. - D) Hire an external consulting firm to build a custom annotation platform on Amazon EC2 instances.	Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon SageMaker Ground Trust Plus is a managed service where AWS provides the expert workforce, manages the workflow setup, and handles the UI and infrastructure. This offers the absolute lowest operational overhead for the enterprise. Distractor Analysis: - A is Incorrect: Amazon EMR is a big data processing framework and does not provide UIs or human labeling capabilities. - C is Incorrect: Manually creating a Ground Truth job with custom HTML templates and managing the public workforce requires significant operational overhead, which the enterprise wants to avoid. - D is Incorrect: Building a custom platform on EC2 introduces the highest possible operational overhead compared to managed AWS services. > Key Takeaway: When you need a fully managed labeling experience without managing UIs, infrastructure, or workforces, choose Amazon SageMaker Ground Truth Plus.
### Question 10 - Handling Data Ordering Bias in Annotation A training dataset for an image classification model was collected and annotated in sequential batches grouped by geographic region. The ML team notices that the model performs poorly on underrepresented regions. They want to fix this issue during the annotation and dataset preparation phase. The constraint is to use the most appropriate data preparation technique. Which technique addresses the root cause of the underrepresentation in the mini-batches? (Select ONE) - A) Apply mean imputation to the pixel values of the images from the underrepresented regions. - B) Shuffle the records so the collection order does not align with region, ensuring samples are evenly distributed across mini-batches. - C) Use Amazon Rekognition to auto-label the images from the underrepresented regions. - D) Convert the images to a columnar format like Parquet to optimize the data loading pipeline.	Correct Answer: B Correct Answer Analysis: - B is Correct: When data is collected sequentially by region, training without shuffling causes mini-batch gradient descent to process entire regions sequentially. Shuffling breaks the alignment between collection order and batching, ensuring samples from the sparse region are distributed evenly across all mini-batches. Distractor Analysis: - A is Incorrect: Mean imputation is used for missing tabular numeric values, not for pixel data or resolving geographic batch ordering bias. - C is Incorrect: Auto-labeling the underrepresented regions with Rekognition does not fix the mini-batch ordering bias during training and might introduce label noise. - D is Incorrect: Converting images to a columnar format (Parquet) is not a standard practice for image data and does not affect batch sequence order or training sample distribution. > Key Takeaway: Shuffling data before training breaks sequential collection biases, ensuring minority groups are properly distributed across mini-batches rather than being isolated.