Skip to content

Skill 1.3: Validate data quality and manage bias

Skill 1.3.7: Clean data (detecting outliers, imputing missing data, deduplication)

Question 1 - Data Quality Ruleset and Deduplication Strategy

A financial services company is building a fraud detection model using transaction data stored in Amazon S3. The data engineering team needs to implement an automated data quality validation pipeline that runs before each model training job. The pipeline must:

  • Detect and remove duplicate transaction records that have identical values across all columns.
  • Flag records where the transaction amount is more than 3 standard deviations from the mean (potential outliers for review).
  • Validate that critical fields like customer_id and transaction_amount are never null.
  • Minimize operational overhead with a serverless approach.

Which TWO AWS services or features should the data engineer use to meet these requirements? (Select TWO)

  • A) Use AWS Glue Data Quality with DQDL rules to define IsComplete for critical columns, DataFreshness for timestamp validation, and then write a custom Python shell in AWS Glue to perform deduplication and outlier detection.
  • B) Use AWS Glue DataBrew data quality recipe with Deduplicate and DetectOutliers transforms, and schedule the recipe as a job.
  • C) Use Amazon SageMaker Data Wrangler to create a data flow that includes deduplication and outlier detection transforms, then export the flow to a SageMaker Pipeline for scheduled execution.
  • D) Use AWS Glue Data Quality with DQDL rules including Uniqueness on a composite key, ColumnValues with a threshold for statistical outliers, and Completeness for required fields.
  • E) Use Amazon Bedrock Guardrails with sensitive information filters to detect anomalous transaction amounts and remove duplicate records.
Click to reveal Answer & Explanation

Correct Answers: B and C

Correct Options Analysis

  • B is Correct: AWS Glue DataBrew provides built-in recipe transforms for deduplication and outlier detection that can be visually applied and scheduled as jobs. DataBrew recipes handle both cleaning operations in a low-code, serverless manner.
  • C is Correct: Amazon SageMaker Data Wrangler includes data quality transforms for deduplication and outlier handling, and integrates with SageMaker Pipelines for scheduled, repeatable execution of the cleaning workflow.

Distractor Analysis

  • A is Incorrect: While AWS Glue Data Quality can validate completeness and uniqueness, it is a monitoring service that evaluates rules against the Data Catalog—it does not perform deduplication or outlier removal. Writing custom Glue code for the cleaning itself is valid but introduces operational overhead beyond the two services requested.
  • D is Incorrect: AWS Glue Data Quality DQDL supports Uniqueness and Completeness rules, but does not support statistical outlier detection (e.g., "more than 3 standard deviations from the mean") as a native DQDL rule type. It also cannot remove duplicates—it only reports on uniqueness violations.
  • E is Incorrect: Amazon Bedrock Guardrails screens prompts and model responses for content safety, denied topics, and PII—it is not a data cleaning service for tabular financial datasets.

Key Takeaway: AWS Glue Data Quality evaluates rules (validate), while DataBrew and SageMaker Data Wrangler perform cleaning transforms (dedup, outlier removal, imputation).

Question 2 - Mean Imputation vs Mode Imputation for Missing Data

A marketing analytics team is preparing a customer churn dataset stored as a CSV file in Amazon S3 for a binary classification model. The dataset contains the following columns:

  • age (numeric, continuous): 8% of values are missing
  • income_level (numeric, ordinal): 5% of values are missing
  • customer_region (categorical): 12% of values are missing
  • tenure_months (numeric, integer): 3% of values are missing

The data scientist wants to use Amazon SageMaker Data Wrangler to perform imputation with minimal impact on downstream model performance. The team has decided to use simple imputation for speed rather than iterative imputation.

Which imputation strategy is MOST appropriate for each column type? (Select ONE)

  • A) Use mean imputation for all columns (age, income_level, customer_region, tenure_months) because it preserves dataset size and is computationally efficient.
  • B) Use mean imputation for age and tenure_months, mode imputation for customer_region, and mean imputation for income_level since it maintains numeric consistency.
  • C) Use mean imputation for age and tenure_months, mode imputation for customer_region, and median imputation for income_level to handle the ordinal nature and reduce outlier influence.
  • D) Use median imputation for age and tenure_months, mode imputation for customer_region, and mean imputation for income_level because median is always better than mean for continuous variables.
  • E) Drop all rows with any missing values to maintain the original distribution without introducing bias.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

  • C is Correct: Median imputation is more robust than mean for continuous variables like age and tenure_months when outliers may exist, and median is specifically appropriate for ordinal data like income_level because it respects the monotonic order without assuming equal intervals. Mode imputation is the standard for categorical variables like customer_region.

Distractor Analysis

  • A is Incorrect: Mean imputation on categorical columns (customer_region) is mathematically invalid—categorical values cannot be averaged. This would produce nonsensical values.
  • B is Incorrect: While mode imputation for customer_region is correct, using mean for ordinal income_level assumes equal intervals between ordinal ranks, which violates the assumption of ordinal data. Median is the preferred measure of central tendency for ordinal features.
  • D is Incorrect: Median imputation is a valid choice, but the blanket statement that "median is always better than mean for continuous variables" is false. When data is symmetric without outliers, mean imputation is more efficient (lower variance of the estimator). The correct answer depends on data distribution—a nuance the exam may test.
  • E is Incorrect: Listwise deletion (dropping rows with missing values) with 8%, 5%, 12%, and 3% missing rates could remove up to 28% of rows, significantly reducing training data. It also introduces selection bias if missingness correlates with any feature.

Key Takeaway: Match imputation strategy to data type: mean for symmetric continuous, median for skewed continuous or ordinal, and mode for categorical.

Question 3 - Outlier Detection and Handling in SageMaker

A healthcare startup is building a predictive model to estimate patient hospitalization costs using historical claims data. The dataset is stored in Amazon EMR and will be processed using Amazon SageMaker Data Wrangler. The data science team has identified that the procedure_cost field contains extreme values: some are data entry errors (e.g., $50 for a major surgery), while others are legitimate high-cost procedures (e.g., $500,000 for organ transplants).

The compliance team requires that no records be silently deleted—every outlier must be either corrected or explicitly documented. The team wants to automatically flag outliers for human review while preserving legitimate data.

Which approach BEST satisfies the operational and compliance requirements? (Select ONE)

  • A) Use Data Wrangler's Handle Outliers transform to automatically remove all values beyond 3 standard deviations, then export the cleaned dataset without review.
  • B) Use Data Wrangler to create a Custom Transform that converts outliers to the column mean (imputation), ensuring the dataset remains complete.
  • C) Use Data Wrangler's outlier detection to flag records as potential outliers, export the flagged set to an S3 bucket, and route them to a SageMaker Ground Truth human review workflow for correction or documented approval.
  • D) Use AWS Glue Data Quality with a DQDL rule that fails the pipeline if procedure_cost exceeds 3 standard deviations, requiring manual intervention before training can continue.
  • E) Use Amazon Comprehend's PII detection to identify anomalous procedure_cost values automatically and redact them.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

  • C is Correct: This approach separates detection from correction—Data Wrangler identifies outliers, routes them to a human-in-the-loop workflow via Ground Truth for review, and ensures compliance by creating an audit trail (human reviews and labels each flagged record as either corrected or approved). This preserves legitimate values while fixing data entry errors.

Distractor Analysis

  • A is Incorrect: Automatically removing all outliers beyond 3 standard deviations silently deletes legitimate records (e.g., $500,000 organ transplants), which violates the compliance requirement. Legitimate outliers contain valuable information for healthcare cost prediction.
  • B is Incorrect: Mean imputation for outliers replaces extreme values (both errors and legitimate) with the average, heavily biasing the distribution and losing critical high-cost signal. It also fails the "corrected or documented" requirement since no distinction is made between errors and legitimate values.
  • D is Incorrect: While AWS Glue Data Quality can detect outliers via DQDL rules (e.g., ColumnValues or EvaluateDataQuality), it only manages monitoring alerts and pipeline failure—it does not route records to human review workflows. It also fails the entire pipeline without providing a remediation path.
  • E is Incorrect: Amazon Comprehend detects PII (names, dates, social security numbers) in text, not numerical outliers in structured healthcare claims data. It is the wrong service entirely for this task.

Key Takeaway: Outlier handling ≠ outlier deletion. Use detection to flag, then route to human review when compliance requires an audit trail; only remove outliers when they are confirmed data entry errors and the business logic tolerates exclusion.

Question 4 - Deduplication Using AWS Glue DataBrew

A data engineering team is building a customer 360 dataset from multiple source systems (CRM, ERP, and web analytics). The team needs to deduplicate customer records before building a machine learning model for customer lifetime value prediction. The following requirements apply:

  • Exact duplicates (all fields identical) must be removed.
  • Near-duplicates (same email and birth_date, but slightly different name or address) should be identified and merged with the most recently updated record taking precedence.
  • The solution must support visual, low-code data preparation.
  • The solution must run on a recurring schedule.

Which approach MOST effectively implements these requirements with the least operational overhead? (Select ONE)

  • A) Use AWS Glue DataBrew recipe with Deduplicate on all columns, Group By on email + birth_date with an aggregation selecting max(last_updated) to merge near-dupes, and schedule the DataBrew job for recurring execution.
  • B) Use AWS Glue Studio to author an ETL job with custom PySpark code performing df.dropDuplicates() and a window function (row_number()) to select the most recent record. Schedule using AWS Glue triggers.
  • C) Use AWS Glue Data Quality with Uniqueness and StandardDeviation DQDL rules to remove duplicates and merge similar records.
  • D) Use Amazon SageMaker Data Wrangler with a Group By transform and export to a SageMaker Processing job for scheduled deduplication.
  • E) Use Amazon Bedrock Knowledge Bases ingestion job to parse customer records and automatically deduplicate them using vector similarity matching.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: AWS Glue DataBrew supports exact deduplication via the Deduplicate recipe transform and can handle fuzzy/near-duplicate merging using Group By + aggregation (e.g., selecting max(updated_date)). Recipes are visual, low-code, and can be scheduled as recurring DataBrew jobs.

Distractor Analysis

  • B is Incorrect: AWS Glue Studio with custom PySpark is functionally valid, but it requires writing and maintaining code (higher operational overhead) compared to DataBrew's visual, recipe-based approach. The question specifies minimal operational overhead with low-code preference.
  • C is Incorrect: AWS Glue Data Quality only evaluates and monitors datasets—it does not perform cleaning operations like deduplication or merging. DQDL Uniqueness can report duplicate violations, but cannot remove or merge them.
  • D is Incorrect: SageMaker Data Wrangler is a valid option, but it does not have a dedicated Deduplicate transform for near-duplicate merging with record precedence logic. It is primarily a SageMaker-centric tool, and the combination of exact dedup + fuzzy merge is more naturally expressed in DataBrew recipes.
  • E is Incorrect: Amazon Bedrock Knowledge Bases is for RAG (Retrieval-Augmented Generation) content ingestion—it does not perform tabular data deduplication or record merging. Vector similarity matching does not implement business rules like "most recent record takes precedence."

Key Takeaway: DataBrew = low-code cleaning transforms (dedup, merge, impute); Glue Data Quality = rules-based validation and monitoring (reports violations, doesn't fix them).

Question 5 - Handling Missing Data with Bias Awareness

A public health organization is building a model to predict hospital readmission risk. The training dataset contains missing values that are not randomly distributed:

  • home_address is missing for homeless patients (who are at higher readmission risk).
  • lab_test_value is missing for patients who skipped follow-up tests (which correlates with worse outcomes).
  • income is missing for patients who declined to report (often younger patients).

The data scientist is using SageMaker Data Wrangler and wants to avoid introducing bias through imputation. Which TWO techniques should be applied? (Select TWO)

  • A) Use mean imputation for all missing values to maintain dataset size and minimize distributional impact.
  • B) Create a binary "is_missing" indicator flag for home_address, lab_test_value, and income, retaining the information that a value was absent.
  • C) Use mode imputation for home_address since the majority class (non-homeless) will minimize model bias.
  • D) Drop rows where home_address is missing because homeless patients are a small minority and unlikely to affect model performance.
  • E) Use domain-aware imputation (e.g., a special category for "homeless" in home_address, a clinical imputation strategy for missing lab values) rather than a generic statistical substitution.
Click to reveal Answer & Explanation

Correct Answers: B and E

Correct Options Analysis

  • B is Correct: Creating an indicator flag (also called a "missing indicator") preserves the predictive signal that the absence of a value may carry. Here, missingness correlates with outcome (homelessness → higher readmission), so the absence itself is informative.
  • E is Correct: Domain-aware imputation replaces missing values with meaningful placeholders (e.g., "homeless" category for home_address, "test_not_done" for lab_test_value) that explicitly model the underlying cause rather than masking it with a generic mean/mode. This preserves the risk signal while maintaining dataset size.

Distractor Analysis

  • A is Incorrect: Mean imputation for all columns masks the causal relationship between missingness and outcome. For home_address (categorical), mean is mathematically invalid. For lab_test_value, mean imputation assumes missing 'at random'—here it is not random and correlates with worse outcomes.
  • C is Incorrect: Mode imputation for home_address simply assigns the most common value (non-homeless) to homeless patients, systematically erasing the high-risk category and biasing the model toward underestimating their readmission risk.
  • D is Incorrect: Dropping rows where home_address is missing (i.e., homeless patients) removes a high-risk subpopulation from the training data, creating representation bias and reducing the model's ability to make accurate predictions for this group.

Key Takeaway: When missingness is informative (correlates with outcome), add an "is_missing" indicator or use domain-aware category substitution—never blindly apply mean/mode imputation.

Question 6 - Near-Duplicate Detection in Text Data

A legal technology company is building a document classification model to categorize legal contracts into practice areas (e.g., "M&A," "Employment," "Intellectual Property"). The training dataset of 50,000 documents stored in Amazon S3 contains near-duplicate contracts that differ only in client names, dates, and minor formatting (e.g., template boilerplate). These near-dups inflate certain categories, potentially causing the model to over-learn template structure rather than meaningful legal concepts.

The data science team is using Amazon SageMaker Data Wrangler and wants to remove near-duplicates before training. Which approach is MOST appropriate? (Select ONE)

  • A) Convert each document to a vector embedding using a model like Amazon Titan Embeddings or a pre-trained sentence transformer, then apply cosine similarity clustering (e.g., DBSCAN) to group near-duplicates and retain representative samples.
  • B) Use AWS Glue Data Quality with a Uniqueness DQDL rule to identify and remove near-duplicate documents.
  • C) Use Amazon Comprehend to detect duplicate phrases and remove documents containing similar language.
  • D) Use Amazon Bedrock Guardrails word filters to block documents containing template terms.
  • E) Use SageMaker Data Wrangler's built-in Deduplicate transform to remove all records with similar text.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Vector embeddings + cosine similarity clustering is the appropriate technique for near-duplicate detection in unstructured text. Embeddings capture semantic similarity, so contracts that differ only in names/dates/formatting will have near-identical vector representations. DBSCAN or similar clustering groups them for deduplication.

Distractor Analysis

  • B is Incorrect: AWS Glue Data Quality DQDL Uniqueness detects exact duplicates in structured tabular data—it does not handle near-duplicate detection in freeform text.
  • C is Incorrect: Amazon Comprehend provides entity extraction, sentiment, and PII detection, but does not have a "duplicate detection" feature for identifying near-duplicate documents.
  • D is Incorrect: Amazon Bedrock Guardrails word filters block specific words—they do not detect semantic near-duplicate documents. This approach would likely flag all legal contracts (which share common terms) rather than identify true duplicates.
  • E is Incorrect: SageMaker Data Wrangler's Deduplicate transform operates on exact row equality in structured datasets. It does not perform fuzzy or semantic deduplication of free-form text documents.

Key Takeaway: For exact duplicates in tabular data, use DataBrew/SageMaker Deduplicate. For near-duplicate text documents, use embeddings + cosine similarity clustering.

Question 7 - Outlier Detection Using AWS Glue Data Quality

A weather forecasting company streams real-time sensor data into Amazon S3 via Kinesis Data Firehose. The ETL pipeline runs on AWS Glue Jobs, and the team wants to add data quality validation to catch outlier sensor readings (e.g., temperature readings of -200°C or 500°C due to sensor malfunction) before the data reaches the model training dataset.

The requirement is to declare explicit rules for valid temperature ranges and allow the pipeline to fail gracefully with a CloudWatch alarm when outliers exceed a threshold. The solution must use serverless, managed services with minimal code.

Which approach BEST satisfies these requirements? (Select ONE)

  • A) Use AWS Glue Data Quality with a DQDL ruleset (ColumnValues "temperature" between -50.0 and 60.0) attached to the Glue job, and configure the Data Quality result to trigger a CloudWatch alarm when the rule violation rate exceeds 5%.
  • B) Use AWS Lambda with a custom Python function that reads the data, calculates z-scores to detect outliers, and sends the results to Amazon SNS if anomalies exceed a threshold.
  • C) Use SageMaker Model Monitor with a data quality monitoring job to detect sensor outliers after model training.
  • D) Use Amazon OpenSearch anomaly detection to monitor incoming temperature readings and alert when unusual values arrive.
  • E) Use AWS Glue DataBrew profile job to identify outliers, and manually review the profile report before each training run.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: AWS Glue Data Quality with DQDL rules provides declarative, serverless outlier detection via ColumnValues (range checks) and standard deviation-based rules. The Data Quality result can be published to CloudWatch metrics, triggering alarms when thresholds are exceeded. This satisfies the "fail gracefully," "explicit rules," and "minimal code" constraints.

Distractor Analysis

  • B is Incorrect: While valid, a custom Lambda function reintroduces operational overhead (writing and maintaining code) that AWS Glue Data Quality eliminates with DQDL. The question specifies minimal code with managed services.
  • C is Incorrect: SageMaker Model Monitor's data quality monitoring runs after model deployment to monitor inference input distributions—it is not designed for pre-training ETL outlier detection in upstream Glue pipelines.
  • D is Incorrect: Amazon OpenSearch anomaly detection is designed for real-time search and log analytics, not batch ETL data validation in Glue pipelines. It would require maintaining an OpenSearch cluster and moving sensor data into it, adding operational complexity.
  • E is Incorrect: DataBrew profile job provides one-time profiling summaries—it does not enforce rules on every pipeline run or trigger alarms automatically. Manual review of profile reports contradicts the requirement for automated, fail-gracefully validation.

Key Takeaway: Glue Data Quality = automated, rule-based validation in ETL pipelines (fail on rule violations); DataBrew profile = one-time exploratory profiling for human review.

Question 8 - Imputation with Domain Knowledge Using SageMaker Processing

A retail company is building a demand forecasting model using historical sales data. The promotion_discount column has missing values. Domain analysis reveals:

  • A zero discount (0%) is recorded as "0" in the system.
  • A missing discount typically means the item was sold during a flash sale where discounts ranged from 20–50%.
  • The flash sale flag exists in a separate column flash_sale (Yes/No).

The data scientist wants to use SageMaker Processing with a script to impute missing values accurately. Which imputation logic is MOST appropriate? (Select ONE)

  • A) Use multiple imputation by chained equations (MICE) on the entire dataset, treating promotion_discount as a continuous variable with missing data at random.
  • B) Impute missing values as follows: if flash_sale == "Yes", fill promotion_discount with the historical median discount during flash sales; if flash_sale == "No", fill with 0 (no promotion).
  • C) Fill all missing values with the global mean of promotion_discount to minimize distributional impact.
  • D) Drop rows where promotion_discount is missing, assuming they are negligible in number.
  • E) Impute missing values with a linear regression model that predicts promotion_discount from all other features, including flash_sale.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: This approach uses domain knowledge (flash sale context) to perform conditional imputation. Instead of a global statistic, it substitutes missing discounts with the median value observed during similar contexts (flash sales), which is statistically appropriate given the 20–50% range and outlier sensitivity. For non-flash-sale rows, 0 is correct (no promotion).

Distractor Analysis

  • A is Incorrect: MICE is a valid advanced imputation technique, but it assumes missing at random (MAR). Here, missingness is systematic—missing values indicate flash sales, not random omissions. MICE would incorrectly blend flash sale and regular pricing distributions, and it is unnecessarily complex for this task.
  • C is Incorrect: Global mean imputation ignores the domain insight that missingness corresponds to flash sales. Since flash sale discounts (20–50%) differ from regular promotions (often 0–10%), the global mean poorly represents both groups and biases downstream forecasts.
  • D is Incorrect: Dropping rows with missing promotion_discount would eliminate all flash sale records, removing an important demand signal (flash sales typically spike demand). This creates representation bias in the training data.
  • E is Incorrect: Linear regression imputation is a valid approach but is unnecessarily complex when domain knowledge provides a direct, interpretable rule. It also introduces model estimation error and may fail if flash_sale is collinear with other features. Simple, domain-informed conditional imputation is more robust.

Key Takeaway: When missing data has a known structural cause (e.g., "missing = flash sale"), use conditional/domain-aware imputation instead of generic statistical methods like MICE or global mean.

Question 9 - Deduplication in a Streaming Pipeline

A ride-sharing company processes GPS telemetry data from vehicles in near-real-time using Amazon Kinesis Data Streams and AWS Glue Streaming Jobs. The telemetry stream occasionally produces duplicate records due to network retries and client-side retry logic, where the same vehicle_id + timestamp + gps_coordinates combo appears multiple times.

The downstream ML model for ETA prediction requires exactly-once processing of telemetry events to avoid over-weighting certain trips. The team wants to implement deduplication within the streaming pipeline with minimal code.

Which approach MOST effectively implements deduplication? (Select ONE)

  • A) Use a stateful AWS Glue streaming job with a windowed deduplication transform that tracks seen records by composite key (vehicle_id, timestamp, gps_coordinates) and drops duplicates within a time window.
  • B) Use AWS Glue Data Quality with a Uniqueness DQDL rule on the streaming job to drop duplicates before they reach the ML model.
  • C) Use Amazon S3 Event Notifications to trigger a Lambda that checks whether a record was already written and blocks duplicates.
  • D) Store all records in Amazon DynamoDB with a composite primary key and use conditional writes (attribute_not_exists) to enforce uniqueness.
  • E) Use Amazon Bedrock Guardrails to filter duplicate content from incoming telemetry.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: AWS Glue streaming jobs support stateful processing (via checkpointing) that can maintain a watermark window of seen records and deduplicate within that window. The composite key ensures the same vehicle-time-coordinate combination is processed exactly once. This approach is serverless and requires minimal custom code.

Distractor Analysis

  • B is Incorrect: AWS Glue Data Quality evaluates and reports on rules—it does not perform cleaning transformations like deduplication. Uniqueness DQDL can detect duplicates in a dataset, but it cannot remove them from a streaming pipeline.
  • C is Incorrect: S3 Event Notifications + Lambda is an ad hoc, batch-triggered approach that does not integrate with the streaming semantics of Kinesis and Glue Streaming. It introduces latency and a custom code path, adding operational overhead.
  • D is Incorrect: DynamoDB conditional writes can enforce uniqueness, but this approach requires a separate database write for every telemetry event, introducing significant latency (~10ms per record) and cost at scale. It also requires custom integration code. For streaming deduplication, Glue's windowed state is more efficient.
  • E is Incorrect: Amazon Bedrock Guardrails is for generative AI prompt/response safety filtering, not data deduplication. It has no capability to process GPS telemetry or remove duplicate records.

Key Takeaway: In streaming pipelines, use stateful streaming frameworks (Glue Streaming, Flink) for deduplication via watermark windows; avoid costly per-record database lookups (DynamoDB) unless required for cross-pipeline consistency.

Question 10 - Combined Data Quality and Cleaning Pipeline

A healthcare insurance company is building an ML model to predict insurance claim approval probability. The pipeline processes semi-structured JSON claim data from Amazon S3. The data team needs to implement a complete data quality and cleaning workflow:

  1. Detect outliers in claim_amount (values > 99th percentile or < 1st percentile).
  2. Impute missing patient_age using median values grouped by patient_gender and patient_region.
  3. Remove exact duplicates (all fields identical).
  4. Ensure claim_id is never null.

The solution must be serverless, low-code, and scheduled daily.

Which architecture MOST completely satisfies all requirements? (Select ONE)

  • A) Use AWS Glue DataBrew recipes with DetectOutliers (flag and remove), FillMissingValues with grouped median (Group By: patient_gender, patient_region → Median), and Deduplicate, scheduled as a daily DataBrew job; Use AWS Glue Data Quality to validate claim_id completeness in the Data Catalog.
  • B) Use SageMaker Data Wrangler for all cleaning transforms, then manually export and upload the cleaned dataset to S3 daily via a Jupyter notebook schedule.
  • C) Use Amazon Bedrock with Foundation Models to detect anomalies, impute missing values, and deduplicate records based on semantic similarity.
  • D) Use AWS Lambda functions triggered by S3 Event Notifications to perform outlier detection, median imputation, and deduplication with custom Python code.
  • E) Use AWS Glue Data Quality exclusively for all four cleaning tasks using DQDL rules (ColumnValues, Completeness, Uniqueness, StandardDeviation).
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: AWS Glue DataBrew provides low-code, visual data cleaning with built-in transforms for outlier detection, median imputation with grouping, and deduplication. Recipes can be scheduled to run daily as serverless jobs. AWS Glue Data Quality complements this by declaring a Completeness DQDL rule on claim_id to validate the result in the Data Catalog.

Distractor Analysis

  • B is Incorrect: Manual export + Jupyter notebook scheduling is not a production-grade, serverless, automated approach. It requires a running notebook instance (operational overhead and cost) and human intervention. SageMaker Data Wrangler is a valid cleaning tool, but the architecture lacks automation and scheduling.
  • C is Incorrect: Amazon Bedrock FMs are for generative AI tasks (text generation, summarization, embeddings), not tabular data quality operations like percentile-based outlier detection or grouped median imputation. This is an invalid use of the service.
  • D is Incorrect: Building the entire cleaning pipeline with custom Lambda code introduces significant operational overhead (code maintenance, error handling, testing) compared to DataBrew's low-code visual transforms. The requirement specified minimal operational overhead.
  • E is Incorrect: AWS Glue Data Quality only validates—it cannot perform outlier removal, imputation, or deduplication. DQDL rules can report violations but cannot clean the dataset. Using it exclusively would not satisfy the data cleaning requirements.

Key Takeaway: Glue Data Quality validates (reports violations), Glue DataBrew cleans (performs transformations). Use them together: DataBrew for cleaning transforms, Data Quality for rule-based monitoring of the cleaned output.