Skill 1.3.7: Clean data (for example, by detecting outliers, imputing missing data, deduplication)

Question	Answer & Explanation
### Question 1 - SageMaker Data Wrangler for Missing Data Imputation A financial services ML team is preparing a loan default prediction dataset stored in Amazon S3. The `annual_income` column has approximately 10% missing values. The team must impute these missing values using the median of the observed data to minimize the effect of outliers, and they want to export the transformed data directly to an Amazon SageMaker Feature Store with the least operational overhead. Which AWS service natively supports this interactive transformation and direct export to the feature store? - A) AWS Glue Data Quality - B) Amazon SageMaker Data Wrangler - C) Amazon Macie - D) AWS Lambda	Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon SageMaker Data Wrangler is a purpose-built tool for ML data preparation that provides interactive, no-code transformations (like median imputation) and allows direct export of the transformed data flow into Amazon SageMaker Feature Store, minimizing operational overhead. Distractor Analysis: - A is Incorrect: AWS Glue Data Quality is used for evaluating data against declarative rules (DQDL), not for performing interactive data transformations or exporting to SageMaker Feature Store. - C is Incorrect: Amazon Macie is a data security service that discovers and protects sensitive data (like PII); it does not perform statistical imputation for ML preprocessing. - D is Incorrect: Using AWS Lambda would require writing custom code to calculate the median, perform the imputation, and handle the export logic, introducing significant operational overhead. > Key Takeaway: For interactive ML data transformations (like imputing missing values) with direct pipeline integration into SageMaker Feature Store, SageMaker Data Wrangler is the native, lowest-overhead choice.
### Question 2 - Outlier Detection with AWS Glue DataBrew An e-commerce company is profiling a large product catalog dataset stored in Amazon S3. The data engineering team needs to detect outliers in the `price` column using statistical methods (e.g., standard deviations) and visually inspect the distributions before deciding on a cleaning strategy. The solution must require no code and provide automated profiling. Which AWS service should the team use to profile and visually inspect these outliers? - A) Amazon Comprehend - B) AWS Glue Data Quality - C) AWS Glue DataBrew - D) Amazon SageMaker Ground Truth	Correct Answer: C Correct Answer Analysis: - C is Correct: AWS Glue DataBrew provides a no-code, visual interface for data profiling. It can automatically surface statistical anomalies, missing values, and outliers in numeric columns, allowing users to inspect distributions interactively before applying cleaning steps. Distractor Analysis: - A is Incorrect: Amazon Comprehend is a natural language processing (NLP) service used for extracting insights from text, not for profiling numeric data to find statistical outliers. - B is Incorrect: AWS Glue Data Quality evaluates datasets against declared rulesets; it does not provide the interactive, visual data profiling interface needed to visually inspect distributions. - D is Incorrect: Amazon SageMaker Ground Truth is used for data labeling and human review workflows, not for automated statistical profiling of numeric columns. > Key Takeaway: When you need a no-code, visual interface to profile datasets, identify statistical anomalies, and understand distributions before cleaning, AWS Glue DataBrew is the ideal service.
### Question 3 - Deduplication Validation via Glue Data Quality A healthcare startup ingests patient records daily into an AWS Glue Data Catalog table. To comply with data governance rules, the data must be checked for exact duplicate records before being used for model training. The engineering team needs to declare a repeatable rule to fail the pipeline if duplicates exist. Which approach fulfills this requirement with the least operational overhead? - A) Write a custom PySpark script in an AWS Glue job to count duplicates and fail if count > 0. - B) Use AWS Glue Data Quality to author a ruleset using DQDL, such as `Distinct "patient_id" = 1.0`. - C) Use Amazon SageMaker Data Wrangler to manually identify and delete duplicates interactively. - D) Use an AWS Lambda function to iterate through records and drop duplicates before loading.	Correct Answer: B Correct Answer Analysis: - B is Correct: AWS Glue Data Quality allows you to define declarative rules using the Data Quality Definition Language (DQDL). A rule like `Distinct "patient_id" = 1.0` ensures there are no duplicate patient IDs, automatically failing the pipeline run if duplicates are detected, all without writing custom code. Distractor Analysis: - A is Incorrect: Writing a custom PySpark script introduces unnecessary operational overhead and code maintenance compared to declaring a DQDL rule. - C is Incorrect: SageMaker Data Wrangler is designed for interactive, visual data preparation, not for running automated, repeatable validation checks that fail data pipelines on ingestion. - D is Incorrect: An AWS Lambda function has execution time limits and requires custom logic, introducing high operational overhead for a task natively handled by Glue Data Quality. > Key Takeaway: To enforce data quality constraints (like preventing duplicates) in an automated data pipeline with minimal code, use AWS Glue Data Quality with DQDL rulesets.
### Question 4 - Median vs. Mean Imputation for Outliers A manufacturing company is building a predictive maintenance model. Sensor data contains missing values in the `temperature` column. The data is heavily skewed due to occasional extreme overheating events (outliers). The ML engineer decides to impute missing values with the median rather than the mean. What is the primary impact of this decision on the imputed dataset? - A) It will pull the dataset's distribution toward the middle and significantly decrease variance. - B) It preserves the original distribution of the dataset exactly. - C) It minimizes the influence of extreme outlier values on the imputed data compared to using the mean. - D) It converts the `temperature` column into a categorical feature.	Correct Answer: C Correct Answer Analysis: - C is Correct: The median is the middle value of a sorted dataset and is robust to outliers. By imputing with the median instead of the mean, the imputed values will not be artificially skewed by the extreme overheating events, preserving a more realistic center for the missing data. Distractor Analysis: - A is Incorrect: Pulling the distribution toward the middle and decreasing variance is the effect of mean imputation, not median imputation. - B is Incorrect: Imputation cannot preserve the original distribution exactly; it substitutes missing values with a constant (the median), which slightly alters the shape and variance. - D is Incorrect: Imputing a numeric column with its median retains its continuous numeric data type; it does not convert it to a categorical feature. > Key Takeaway: When a dataset contains extreme outliers, median imputation is preferred over mean imputation because the median is robust to outliers and prevents extreme values from skewing the imputed data.
### Question 5 - Fuzzy Matching for Near-Duplicates A marketing agency has a CRM dataset with near-duplicate records where customer names differ slightly due to typos (e.g., "John Doe" vs. "Jon Doe"). The team needs to identify and consolidate these near-duplicate records using a visual, no-code interface with the least operational overhead. Which AWS service provides native transformations for fuzzy matching and deduplication? - A) AWS Glue DataBrew - B) Amazon Comprehend - C) AWS Glue Data Quality - D) Amazon SageMaker Feature Store	Correct Answer: A Correct Answer Analysis: - A is Correct: AWS Glue DataBrew includes built-in transforms for fuzzy matching (e.g., matching strings with minor typos or variations) and deduplication, accessible through a no-code visual interface. Distractor Analysis: - B is Incorrect: Amazon Comprehend is an NLP service for extracting entities or sentiment from text, not for identifying and consolidating near-duplicate tabular records. - C is Incorrect: While AWS Glue Data Quality can check for exact uniqueness (e.g., `Distinct`), it is not designed to perform fuzzy matching or automated consolidation of near-duplicate rows. - D is Incorrect: Amazon SageMaker Feature Store is used for storing and serving ML features, not for cleaning or deduplicating raw data. > Key Takeaway: For visual, no-code fuzzy matching and consolidation of near-duplicate tabular records, AWS Glue DataBrew provides the necessary built-in transformations.
### Question 6 - Capping Outliers in ML Pipelines A logistics company tracks delivery times in a dataset. Due to occasional GPS logging errors, there are impossible values (e.g., negative delivery times or times exceeding 30 days). The ML team wants to automatically cap or remove these outliers visually before training a forecasting model. Which AWS service offers a built-in transformation to detect and handle outliers in an ML data preparation pipeline? - A) Amazon SageMaker Data Wrangler - B) Amazon Macie - C) AWS Glue Data Catalog - D) Amazon Bedrock Guardrails	Correct Answer: A Correct Answer Analysis: - A is Correct: Amazon SageMaker Data Wrangler provides built-in transformations for handling outliers, such as capping values to a specified quantile range or standard deviation, making it ideal for preparing ML datasets. Distractor Analysis: - B is Incorrect: Amazon Macie is a security service for discovering sensitive data; it does not perform statistical outlier removal. - C is Incorrect: AWS Glue Data Catalog is a metadata repository; it does not contain data transformation or cleaning capabilities. - D is Incorrect: Amazon Bedrock Guardrails are used for filtering prompts and model responses in generative AI, not for cleaning tabular data in an ML pipeline. > Key Takeaway: Amazon SageMaker Data Wrangler contains purpose-built ML transformations like outlier capping and flooring, allowing data scientists to visually clean datasets before exporting them to training pipelines.
### Question 7 - Scheduling Automated Data Cleaning A retail company's data pipeline ingests daily sales data into Amazon S3. The pipeline requires automated, repeatable cleaning jobs that impute missing values in the `sales_amount` column using a custom SQL expression, and the jobs must be scheduled to run daily without provisioning infrastructure. Which AWS service is the MOST appropriate to create and schedule this data cleaning job with minimal effort? - A) Amazon EC2 with a cron job - B) Amazon SageMaker Processing Jobs - C) AWS Glue DataBrew - D) AWS Glue Data Quality	Correct Answer: C Correct Answer Analysis: - C is Correct: AWS Glue DataBrew allows you to create visual "recipes" for data cleaning (including SQL-based transformations) and schedule them to run automatically on a daily basis as serverless jobs, requiring no infrastructure provisioning. Distractor Analysis: - A is Incorrect: Using Amazon EC2 requires provisioning servers, managing OS patches, and maintaining cron, resulting in high operational overhead. - B is Incorrect: Amazon SageMaker Processing Jobs require writing and packaging custom Python scripts, which is more overhead than using a visual recipe in DataBrew. - D is Incorrect: AWS Glue Data Quality evaluates rules; it does not execute transformation recipes to impute missing values based on SQL expressions. > Key Takeaway: AWS Glue DataBrew recipes can be saved and scheduled as serverless jobs to automate repeatable data cleaning tasks like imputation and SQL transformation with zero infrastructure management.
### Question 8 - PII Redaction as Data Cleaning A customer support team is preparing a dataset of chat transcripts to train a custom NLP model. The ML team must clean the text data by detecting and redacting Personally Identifiable Information (PII) such as names and credit card numbers before the data reaches the training environment. The solution must use a managed AWS service optimized for text analysis. Which service should they use to clean the text? - A) Amazon Comprehend - B) AWS Glue DataBrew - C) Amazon SageMaker Feature Store - D) Amazon Rekognition	Correct Answer: A Correct Answer Analysis: - A is Correct: Amazon Comprehend provides managed APIs like `DetectPii` and `ContainsPii` that can identify sensitive data in unstructured text, allowing the team to programmatically redact or mask PII before training NLP models. Distractor Analysis: - B is Incorrect: AWS Glue DataBrew is excellent for tabular data transformations, but it does not natively possess ML models to identify and redact unstructured text PII like Comprehend does. - C is Incorrect: Amazon SageMaker Feature Store is a repository for ML features, not a text analysis or PII redaction tool. - D is Incorrect: Amazon Rekognition is a computer vision service used for analyzing images and videos, not text data. > Key Takeaway: For cleaning unstructured text datasets by detecting and redacting PII, Amazon Comprehend is the native, managed AWS NLP service to use.
### Question 9 - Consequence of Dropping Missing Rows A data science team is preparing a dataset for a binary classification model. A critical feature, `user_age`, has 5% missing values. The team decides to drop the rows with missing values rather than imputing them. What is the primary consequence of this decision on the remaining dataset? - A) It preserves the original statistical distribution of the remaining `user_age` data but loses the other features in those rows. - B) It pulls the distribution of the `user_age` data toward the middle and decreases variance. - C) It artificially increases the variance of the `user_age` column. - D) It automatically resolves any class imbalance in the target variable.	Correct Answer: A Correct Answer Analysis: - A is Correct: Dropping rows with missing values (listwise deletion) ensures that the statistical distribution of the observed `user_age` values is not altered by artificial imputation. However, the tradeoff is the loss of all other feature data contained in those discarded rows. Distractor Analysis: - B is Incorrect: Pulling the distribution toward the middle and decreasing variance is the effect of mean imputation, not dropping rows. - C is Incorrect: Dropping rows with missing values does not inherently increase the variance of the observed data; it simply truncates the dataset. - D is Incorrect: Dropping rows with missing features has no guaranteed relationship to the target variable's class balance; it could potentially worsen class imbalance if the missing values correlate with the minority class. > Key Takeaway: Dropping rows with missing data preserves the feature's original statistical distribution but sacrifices the totality of data points in those rows, which can reduce model training volume.
### Question 10 - Exact Deduplication Using Visual Recipes A global enterprise maintains a customer database where exact duplicate records are periodically introduced by upstream systems. The data engineering team wants to create a reusable, visual cleaning recipe that removes exact duplicates based on a `customer_id` primary key and exports the clean data to a new S3 bucket. The solution must avoid writing custom code. Which AWS service meets these requirements with the least operational overhead? - A) AWS Glue Data Quality - B) Amazon SageMaker Ground Truth - C) AWS Glue DataBrew - D) AWS Lambda	Correct Answer: C Correct Answer Analysis: - C is Correct: AWS Glue DataBrew provides a visual interface to easily build cleaning recipes, including a "Remove duplicates" transform based on specific keys (like `customer_id`), and can output the results directly to S3 without writing code. Distractor Analysis: - A is Incorrect: AWS Glue Data Quality can detect duplicates and fail a pipeline, but its primary role is rule evaluation; it does not create transformation recipes to clean and export data to S3. - B is Incorrect: Amazon SageMaker Ground Truth is a data labeling service using human reviewers, not a data deduplication tool. - D is Incorrect: AWS Lambda requires writing custom code to read the data, identify duplicates, write to S3, and manage execution, resulting in high operational overhead compared to a visual DataBrew recipe. > Key Takeaway: For no-code, visual deduplication of tabular datasets and exporting cleaned results to S3, AWS Glue DataBrew's interactive recipes are the most operationally efficient solution.