Skip to content

Skill 1.3.1: Validate data quality (for example, by using DataBrew, AWS Glue Data Quality)

Question Answer & Explanation
### Question 1 - Declarative Data Quality Rules in AWS Glue

An enterprise machine learning team is preparing a customer demographics dataset stored in Amazon S3 and cataloged in the AWS Glue Data Catalog. The compliance team requires that explicit, repeatable data quality rules—such as ensuring the customer_id column is 100% complete and the age column is strictly between 18 and 100—are evaluated automatically on every pipeline run.

The team must implement this validation with the least operational overhead and without writing custom evaluation scripts.

Which ONE approach satisfies these requirements? (Select ONE)

- A) Create an AWS Lambda function triggered by S3 events to parse the data and validate the completeness and age ranges using Python Pandas.
- B) Author a ruleset using the Data Quality Definition Language (DQDL) and use AWS Glue Data Quality to evaluate it against the Data Catalog table.
- C) Use Amazon SageMaker Data Wrangler to interactively profile the dataset and manually verify that ages fall within the required range.
- D) Create an AWS Glue DataBrew profile job to run on the dataset, which will automatically infer anomalies and missing values based on statistical distributions.		 Correct Answer: B

Correct Answer Analysis:
- B is Correct: AWS Glue Data Quality uses the Data Quality Definition Language (DQDL) to define explicit, declarative rules (e.g., Completeness "customer_id" = 1.0, ColumnValues "age" between 18 and 100). These rulesets can be evaluated automatically against datasets in the AWS Glue Data Catalog on every pipeline run, enforcing strict validation with minimal operational overhead.

Distractor Analysis:
- A is Incorrect: Writing a custom Lambda function with Pandas introduces unnecessary operational overhead, requires infrastructure management, and does not natively integrate with Glue Data Catalog declarative rules.
- C is Incorrect: Amazon SageMaker Data Wrangler is an interactive tool for visual data preparation and transformation, not an automated, repeatable ruleset evaluation engine for pipeline gates.
- D is Incorrect: AWS Glue DataBrew profiling generates statistical summaries and infers anomalies, but it does not enforce strict, explicit declarative rules (like exactly 100% completeness and specific value bounds) in the way DQDL does.

> Key Takeaway: Use AWS Glue Data Quality with DQDL when you need to enforce explicit, declarative data validation rules automatically within your data pipelines.
### Question 2 - Surface Data Defects Before Cleaning

A retail company is ingesting daily transaction logs into Amazon S3 as CSV files. The data science team needs to understand the current state of the data—specifically identifying missing values, inconsistent date formats, and unexpected ranges in numeric columns—before deciding on a specific cleaning strategy.

The solution must minimize infrastructure management and provide built-in visualizations of the dataset's characteristics.

Which ONE AWS service should the team use to profile the dataset? (Select ONE)

- A) AWS Glue Data Quality
- B) AWS Glue DataBrew
- C) Amazon SageMaker Ground Truth
- D) AWS Lambda with Amazon Macie		 Correct Answer: B

Correct Answer Analysis:
- B is Correct: AWS Glue DataBrew is a visual data preparation tool that allows users to run profile jobs on datasets. A DataBrew profile job measures the dataset and generates statistics, surfacing missing values, duplicate rows, outlier information, and inconsistent formats. This allows the team to see the defects before choosing a repair approach.

Distractor Analysis:
- A is Incorrect: AWS Glue Data Quality evaluates datasets against pre-defined declarative rules (DQDL). While it can generate a recommended ruleset, its primary purpose is rule enforcement, not open-ended statistical profiling to discover unknown data defects.
- C is Incorrect: Amazon SageMaker Ground Truth is used for data labeling and human review workflows, not for profiling data quality or identifying missing values.
- D is Incorrect: AWS Lambda is a serverless compute service, and Amazon Macie is a data security service for discovering sensitive data (PII). Neither provides statistical dataset profiling.

> Key Takeaway: Profiling comes before cleaning; use AWS Glue DataBrew to statistically profile datasets and discover unknown anomalies, missing values, and format inconsistencies.
### Question 3 - Interactively Resolving Class Imbalance and Outliers

A machine learning engineer is working on a fraud detection model. The raw dataset contains extreme outliers in the transaction_amount column and suffers from severe class imbalance, where fraudulent transactions represent less than 1% of the data. The engineer needs to experiment with different outlier removal techniques and oversampling strategies, viewing the impact on the data distribution interactively before committing to a final pipeline.

Which ONE AWS tool provides the MOST appropriate interactive environment for this task? (Select ONE)

- A) AWS Glue DataBrew
- B) AWS Glue Data Quality
- C) Amazon SageMaker Data Wrangler
- D) Amazon SageMaker Ground Truth		 Correct Answer: C

Correct Answer Analysis:
- C is Correct: Amazon SageMaker Data Wrangler is designed for interactive data preparation and feature engineering. It contains built-in transformations for detecting and handling outliers, as well as tools for resolving class imbalance (like SMOTE). The engineer can apply these transforms and immediately visualize the effect on the distribution.

Distractor Analysis:
- A is Incorrect: While AWS Glue DataBrew has visual data preparation capabilities, SageMaker Data Wrangler is purpose-built for ML workflows, offering native, interactive ML-specific transforms like SMOTE for class imbalance directly within the ML studio environment.
- B is Incorrect: AWS Glue Data Quality enforces declarative rules but does not provide an interactive UI for experimenting with ML-specific data augmentation or class imbalance resolution.
- D is Incorrect: SageMaker Ground Truth is for data labeling, not for transforming data or resolving class imbalance.

> Key Takeaway: For interactive ML-specific data transformations—like outlier handling and class imbalance resolution—SageMaker Data Wrangler provides the most purpose-built environment with immediate visual feedback.
### Question 4 - Stopping Pipelines on Quality Failures

A financial services company uses AWS Step Functions to orchestrate a complex ETL pipeline before model training. The data is cataloged in the AWS Glue Data Catalog. The team needs to ensure that if a dataset fails specific data quality rules (e.g., a required column has more than 5% missing values), the Step Functions pipeline immediately halts to prevent corrupt data from reaching the model training phase.

Which ONE approach achieves this with the least operational overhead? (Select ONE)

- A) Use AWS Glue DataBrew to profile the data and output a CloudWatch metric. Create a CloudWatch Alarm that triggers a Lambda function to stop the Step Function execution.
- B) Author an AWS Glue Data Quality ruleset in DQDL and attach it to the AWS Glue Data Catalog table. Configure the Step Functions pipeline to evaluate the ruleset and halt if the evaluation fails.
- C) Write a custom AWS Glue PySpark job that counts null values and throws an exception if the threshold is exceeded.
- D) Use Amazon Macie to scan the S3 bucket for data quality issues and block the training job if issues are detected.		 Correct Answer: B

Correct Answer Analysis:
- B is Correct: AWS Glue Data Quality integrates natively with the AWS Glue Data Catalog. You can define a DQDL ruleset (e.g., Completeness "column_name" > 0.95) and evaluate it. The evaluation results can be used by orchestrators like Step Functions to branch or halt the pipeline, providing a low-overhead, declarative gating mechanism.

Distractor Analysis:
- A is Incorrect: Using DataBrew profiling to output metrics, setting up CloudWatch Alarms, and invoking Lambda introduces unnecessary architectural complexity and operational overhead compared to native Glue Data Quality integration.
- C is Incorrect: Writing a custom PySpark job requires development and maintenance effort, violating the least operational overhead constraint.
- D is Incorrect: Amazon Macie is a security service for discovering and protecting sensitive data (PII), not for enforcing general data quality rules like column completeness.

> Key Takeaway: AWS Glue Data Quality acts as a pipeline gate; its declarative rule evaluations can be used by orchestrators like Step Functions to halt pipelines automatically on failure.
### Question 5 - Recommending a Starting Ruleset

A junior data engineer has just registered a new dataset in the AWS Glue Data Catalog. The dataset contains historical sales records. The engineer wants to enforce data quality rules but is unsure where to start. They need a tool that can automatically analyze the dataset and propose a starting set of data quality rules based on the dataset's current characteristics.

Which ONE AWS service provides the capability to automatically recommend a starting data quality ruleset? (Select ONE)

- A) AWS Glue Data Quality
- B) Amazon SageMaker Feature Store
- C) AWS Lambda
- D) Amazon Comprehend		 Correct Answer: A

Correct Answer Analysis:
- A is Correct: For a dataset stored in the AWS Glue Data Catalog, AWS Glue Data Quality can analyze the dataset and automatically propose a starting set of Data Quality Definition Language (DQDL) rules. The engineer can then review, modify, and adopt these rules for ongoing validation.

Distractor Analysis:
- B is Incorrect: Amazon SageMaker Feature Store is used for storing, retrieving, and managing ML features, not for generating or recommending data quality rules.
- C is Incorrect: AWS Lambda is a compute service. While you could write custom code to analyze data, it does not natively recommend DQDL rules.
- D is Incorrect: Amazon Comprehend is an NLP service used for extracting insights from text, not for evaluating tabular data quality rules.

> Key Takeaway: AWS Glue Data Quality can automatically recommend a starting DQDL ruleset based on a dataset in the AWS Glue Data Catalog, accelerating data quality setup.
### Question 6 - Automating Profiling on a Schedule

A healthcare analytics team receives new patient treatment records in an Amazon S3 bucket every week. The team needs to automate a process that reads this unstructured and structured data, identifies anomalies, missing values, and duplicate records, and generates a report for the data stewards. The solution must be serverless and require no infrastructure management.

Which ONE AWS service is purpose-built to run this profiling job on a schedule with the least operational overhead? (Select ONE)

- A) AWS Glue Data Quality
- B) AWS Glue DataBrew
- C) Amazon EMR
- D) AWS Step Functions		 Correct Answer: B

Correct Answer Analysis:
- B is Correct: AWS Glue DataBrew is a serverless data preparation service. You can create a DataBrew "Profile job" that runs on a schedule, automatically reading data from S3, computing statistical profiles, detecting duplicates, and identifying anomalies and missing values, all without managing any underlying compute infrastructure.

Distractor Analysis:
- A is Incorrect: While Glue Data Quality is serverless, it evaluates pre-defined DQDL rules rather than open-endedly profiling data to discover anomalies and duplicates statistically for a general report.
- C is Incorrect: Amazon EMR requires cluster management and configuration, violating the "no infrastructure management" constraint.
- D is Incorrect: AWS Step Functions is an orchestrator, not a data profiling engine. It would need to orchestrate other services (like Lambda or Glue) to achieve this.

> Key Takeaway: AWS Glue DataBrew is a serverless data profiling and preparation tool perfect for scheduled, automated statistical analysis of datasets without managing infrastructure.
### Question 7 - Validating Unique Constraints

An e-commerce platform ingests daily order data into an AWS Glue Data Catalog table. The ML team needs to ensure that the order_id column is completely unique, and that the order_status column only contains one of four predefined values ('PENDING', 'SHIPPED', 'DELIVERED', 'CANCELLED'). If these rules fail, the data must be quarantined.

Which ONE approach implements this validation declaratively? (Select ONE)

- A) Write an AWS Glue DataBrew recipe with custom transformations to filter out duplicate order_ids and invalid statuses.
- B) Use AWS Glue Data Quality to define a ruleset utilizing Uniqueness and AllowedValues rules in the DQDL.
- C) Create a custom PySpark script in an AWS Glue Spark job to perform dropDuplicates() and filter() operations.
- D) Train a SageMaker model to classify valid versus invalid orders based on historical data.		 Correct Answer: B

Correct Answer Analysis:
- B is Correct: AWS Glue Data Quality's Data Quality Definition Language (DQDL) natively supports rules like Uniqueness "order_id" = 1.0 and AllowedValues "order_status" in ['PENDING', 'SHIPPED', 'DELIVERED', 'CANCELLED']. This declarative approach cleanly enforces the constraints without custom code.

Distractor Analysis:
- A is Incorrect: DataBrew recipes are for transforming and cleaning data, not for declaratively validating rules to gate a pipeline. While it can filter data, it doesn't natively "validate" in the same declarative pipeline-gating way.
- C is Incorrect: Writing a custom PySpark script introduces unnecessary code maintenance and operational overhead compared to a declarative DQDL ruleset.
- D is Incorrect: Training an ML model is unnecessary overkill for simple rule-based validation of distinct and enumerated values.

> Key Takeaway: DQDL provides specific rule types like Uniqueness and AllowedValues to declaratively validate column constraints in AWS Glue Data Quality.
### Question 8 - Handling Near-Duplicate Records

A data engineering team is preparing a dataset of customer reviews. They have identified "near-duplicate" records where the text of the review is identical, but the timestamp differs by a few milliseconds due to a frontend submission bug. The team needs to remove these near-duplicates to prevent skewing the NLP model's training data.

They want to use an AWS service that visually identifies these duplicates and provides a built-in transform to deduplicate the records based on specific columns.

Which ONE service should they use? (Select ONE)

- A) AWS Glue Data Quality
- B) AWS Glue DataBrew
- C) Amazon Macie
- D) AWS Glue Data Catalog		 Correct Answer: B

Correct Answer Analysis:
- B is Correct: AWS Glue DataBrew provides built-in transforms for data cleaning, including deduplication. In the interactive DataBrew project UI, engineers can identify duplicate or near-duplicate rows and use the "Remove duplicate rows" transform, configuring it to keep the first or last occurrence based on specific columns (like the review text).

Distractor Analysis:
- A is Incorrect: AWS Glue Data Quality can enforce a Uniqueness rule, but it evaluates and fails the dataset; it does not provide interactive visual transforms to clean or deduplicate the data.
- C is Incorrect: Amazon Macie is for security and PII discovery, not data deduplication.
- D is Incorrect: AWS Glue Data Catalog is a metadata repository, not a data transformation or cleaning tool.

> Key Takeaway: AWS Glue DataBrew excels at interactive data cleaning, offering built-in visual transforms like deduplication to repair dataset defects.
### Question 9 - Monitoring Data Quality Job Metrics

A company runs daily AWS Glue Data Quality evaluations on their feature datasets. The data engineering manager wants to set up an alert if a specific data quality rule—such as the completeness of the email_address column—drops below 90%, so they can take proactive action before the ML training pipeline runs.

Which ONE AWS service natively collects these Data Quality evaluation metrics for alerting? (Select ONE)

- A) Amazon CloudWatch
- B) AWS CloudTrail
- C) Amazon EventBridge
- D) AWS Security Hub		 Correct Answer: A

Correct Answer Analysis:
- A is Correct: Amazon CloudWatch collects metrics and logs from the jobs that validate and clean data (including AWS Glue Data Quality and DataBrew jobs). The results of DQDL rule evaluations are published as CloudWatch metrics, allowing you to create CloudWatch Alarms to alert the team when rules fail or metrics drop below a threshold.

Distractor Analysis:
- B is Incorrect: AWS CloudTrail records API calls made in your account (who did what), but it does not collect metric data regarding the internal outcomes of data quality rules.
- C is Incorrect: Amazon EventBridge is used to route events (like a Glue job state change from RUNNING to FAILED), but the actual rule evaluation metrics (like completeness percentages) are collected by CloudWatch.
- D is Incorrect: AWS Security Hub is for security posture management, not data quality metric monitoring.

> Key Takeaway: Amazon CloudWatch natively collects metrics from AWS Glue Data Quality and DataBrew, enabling alerts when data quality thresholds are violated.
### Question 10 - Combining Profiling and Declarative Rules

A machine learning team is building a data pipeline in AWS. The data scientists first need to explore the raw dataset to understand its statistical properties, missing values, and outliers without writing code. Once the defects are understood, the data engineers need to implement strict, repeatable validation rules that will run automatically on every new batch of data before it enters the SageMaker training pipeline.

Which TWO AWS services should be combined to fulfill these requirements? (Select TWO)

- A) Use AWS Glue DataBrew to profile and interactively explore the raw dataset.
- B) Use AWS Glue Data Quality to author DQDL rulesets for automated pipeline validation.
- C) Use Amazon SageMaker Ground Truth to label the missing values.
- D) Use Amazon Comprehend to extract key phrases from the outliers.
- E) Use AWS Lambda to manually calculate the statistical properties of the dataset.		 Correct Answers: A and B

Correct Answer Analysis:
- A is Correct: AWS Glue DataBrew is the right tool for the initial exploration phase, providing no-code visual profiling to understand statistical properties, outliers, and missing values.
- B is Correct: Once the defects are understood, AWS Glue Data Quality allows engineers to author explicit DQDL rulesets that can be evaluated automatically on every pipeline run to gate the training data.

Distractor Analysis:
- C is Incorrect: SageMaker Ground Truth is for human labeling of unlabeled data, not for statistical profiling or rule validation.
- D is Incorrect: Amazon Comprehend is an NLP service for text analysis, not for tabular data quality validation.
- E is Incorrect: Writing a custom Lambda function violates the "without writing code" constraint and introduces unnecessary operational overhead compared to DataBrew.

> Key Takeaway: A robust AWS data quality strategy often combines DataBrew (for initial interactive profiling and understanding) with Glue Data Quality (for automated declarative rule enforcement in production pipelines).