Task 1.3 - Validate data quality and manage bias

Overview

Task 1.3, Validate data quality and manage bias, is the gate before training. It covers proving the data is complete and internally consistent and labeling it correctly where labels are needed. It also covers checking that the distribution of what you collected is not going to teach the model something you did not intend.

Bias work starts before training rather than after a model is already producing results, and it does not stop at launch.

As you review these lessons for Task 1.3, check that you understand how to do the following.

Validate data quality (for example, by using DataBrew, AWS Glue Data Quality).
Label and annotate data.
Identify and mitigate sources of bias in data by using AWS tools and techniques (for example, dataset splitting, shuffling, augmentation).
Optimize multimodal data distributions by applying bias metrics across numeric, text, and image assets.
Resolve class imbalance in numeric, text, and image datasets.
Validate AI training data integrity (for example, prompt-response pair validation, content safety screening).
Clean data (for example, by detecting outliers, inputting missing data, deduplication).

AWS services overview

The services here support three jobs: measuring quality against rules you declare, producing labels, and screening content before it reaches a model. Review each service to learn more.

AWS Service	Description
AWS Glue Data Quality	Measures a dataset against rules that are declared rather than inferred. Rulesets are written in the Data Quality Definition Language, and for a dataset in the AWS Glue Data Catalog the service can propose a starting set.
AWS Glue DataBrew	A profile job measures the dataset and writes out what it found, surfacing missing values, unexpected ranges, and inconsistent formats before a cleaning approach is chosen.
Amazon Bedrock Guardrails	Configurable filters that screen prompts and model responses. The ones bearing on data preparation include content filters, denied topics, word filters, and sensitive information filters.
Amazon SageMaker AI	Managed labeling and human-review workflows for datasets needing annotation. Availability of a specific labeling capability changes, so confirm what is open to a new account first.
Amazon SageMaker Data Wrangler	Transforms for cleaning and for balancing a dataset, applied interactively so the effect is visible as it is made.
Amazon Comprehend	Detects personally identifiable information in English or Spanish text, which supports both cleaning and safety screening.
Amazon CloudWatch	Collects metrics and logs from the jobs that validate and clean data, so a validation failure is visible rather than silent.

Knowledge Check

A team wants to declare explicit, repeatable rules about a dataset, such as required completeness and allowed value ranges, and have those rules evaluated on every run. Which option fits? (Choose 1)

A. Apply a content filter to the dataset
B. Convert the dataset to a columnar format
C. Author a ruleset in the Data Quality Definition Language and let AWS Glue Data Quality evaluate it
D. Ingest the dataset into Amazon SageMaker Feature Store

Answer & Explanation

Correct Answer: C. Author a ruleset in the Data Quality Definition Language and let AWS Glue Data Quality evaluate it

Explanation:
AWS Glue Data Quality uses Data Quality Definition Language (DQDL) to define explicit, declarative rules (such as Completeness "customer_id" > 0.98 or ColumnValues "age" between 18 and 100). These DQDL rulesets can be evaluated automatically against data pipelines or datasets in the AWS Glue Data Catalog on every run to catch bad or missing data before it reaches downstream ML models or analytics.

Why the other options are incorrect: * A. Apply a content filter to the dataset: Content filters (such as Amazon Bedrock Guardrails) screen prompts, completions, or unstructured text for safety, toxicity, or denied topics, rather than validating dataset column completeness or value ranges. * B. Convert the dataset to a columnar format: Storing data in columnar formats (like Apache Parquet or ORC) optimizes query execution and column pruning, but does not enforce or evaluate data quality rules. * D. Ingest the dataset into Amazon SageMaker Feature Store: SageMaker Feature Store stores and serves engineered feature sets for training and inference consistency, but does not provide declarative DQDL rule evaluation on dataset runs.

AWS skills overview

In this lesson, you review the skills behind Task 1.3 as a light refresher. Each task groups a handful of related skills into a few topics. Read through them and notice which ones already feel familiar, and which ones feel shaky enough that you want to come back to them before exam day.

Validating and cleaning data

In practice, this group of skills lets the machine learning engineer decide whether a dataset is fit to train on, and repair it where it is not.

Be comfortable doing the following:

Validating data quality by declaring rules and evaluating them, rather than by inspecting samples by hand.
Cleaning data, which covers detecting outliers, filling in missing values, and removing duplicates.
Recognizing that profiling comes before cleaning, because you cannot choose a repair before you know the defect.

Cleaning decisions carry assumptions worth stating out loud. Filling a missing value with a column average keeps the row usable and pulls the distribution toward the middle. Dropping the row preserves the distribution and loses whatever else that row contained. Neither is automatically right, and the choice matters more when the missing values are not spread evenly across the dataset. Deduplication has a similar edge: near-duplicate records that differ in one field are the ambiguous case, and treating them as distinct inflates whatever pattern they share.

Knowledge Check

Missing values in a column are filled with that column's average. What does that choice do to the dataset? (Choose 1)

A. It keeps the rows usable and pulls the distribution toward the middle
B. It preserves the original distribution exactly
C. It removes the affected rows from the dataset
D. It converts the column to a categorical feature

Answer & Explanation

Correct Answer: A. It keeps the rows usable and pulls the distribution toward the middle

Explanation:
Mean imputation replaces missing values with the average (\(\mu\)) of the observed data points. This retains the rows for downstream model training instead of dropping them. However, injecting multiple records equal to the mean decreases overall variance and increases the frequency of central values, artificially pulling the feature's distribution toward the middle.

Why the other options are incorrect: * B. It preserves the original distribution exactly: Mean imputation alters the variance, standard deviation, and covariance of the feature, artificially smoothing variability and narrowing confidence intervals. * C. It removes the affected rows from the dataset: Dropping affected rows refers to listwise deletion (dropping records with missing values), which mean imputation explicitly avoids. * D. It converts the column to a categorical feature: Imputing a column with its numeric mean retains its continuous numeric data type.

Labeling and annotating data

In practice, this group of skills lets the machine learning engineer produce the labels a supervised model learns from, and trust them.

Be comfortable doing the following:

Setting up a labeling and annotation workflow, including where a human reviewer belongs in it.
Writing labeling instructions precise enough that two annotators reach the same answer on the same item.
Measuring agreement between annotators, and treating persistent disagreement as a signal about the instructions rather than about the annotators.

Label quality sets a ceiling on model quality that no amount of training compute can raise. If annotators disagree on a class boundary, the model learns a blurred boundary, and its errors will cluster exactly where the instructions were ambiguous. That makes the ambiguous cases the ones worth reviewing first. It is also the reason a small, carefully labeled dataset can outperform a much larger one labeled loosely.

Knowledge Check

Two annotators consistently disagree on the same category boundary. What is the MOST useful interpretation? (Choose 1)

A. The labeling instructions are ambiguous at that boundary
B. The dataset needs more records
C. The model architecture is the wrong choice
D. One annotator should be removed from the workflow

Answer & Explanation

Correct Answer: A. The labeling instructions are ambiguous at that boundary

Explanation:
When multiple human annotators repeatedly disagree on a specific class boundary, it indicates low inter-annotator agreement (inter-rater reliability). This signals that the labeling instructions, edge-case definitions, or category rubrics lack clear, mutually exclusive guidelines. Refining the instruction manual and providing clear "good vs. bad" edge-case examples directly resolves this ambiguity before training.

Why the other options are incorrect: * B. The dataset needs more records: Collecting additional unlabeled data does not clarify ambiguous definitions; it will simply result in more inconsistently labeled noise. * C. The model architecture is the wrong choice: Model selection takes place downstream during the modeling phase. Disagreement during data annotation is an isolated data quality issue independent of model selection. * D. One annotator should be removed from the workflow: Removing a worker treats a symptom rather than the root cause. Consistent disagreement across trained labelers points to systemic ambiguity in the instructions rather than individual worker error.

Identifying and mitigating bias

In practice, this group of skills lets the machine learning engineer find the places where a dataset misrepresents the population it is supposed to describe, and correct them before training.

Be comfortable doing the following:

Identifying sources of bias in data and mitigating them with techniques such as dataset splitting, shuffling, and augmentation.
Applying bias metrics across numeric, text, and image assets, so a multimodal dataset is checked in every modality rather than only in its tabular part.
Recognizing that bias detection starts before training and continues after deployment, so the data preparation stage is where it begins rather than where it ends.

Each technique addresses a specific defect, so naming the defect first is what makes the choice obvious. Shuffling removes bias introduced by collection order, such as records that arrived grouped by region or by date. Careful splitting keeps a group from appearing in only one of your training and evaluation sets, which would otherwise leave you unable to see the problem. Augmentation adds variation to a group the dataset underrepresents. Reaching for the wrong one is easy: shuffling a dataset that is genuinely missing a group changes nothing about the gap, because shuffling reorders what is there and adds nothing that is not.

Knowledge Check

A dataset genuinely underrepresents one group relative to the population it describes. What does shuffling the records accomplish? (Choose 1)

A. Nothing about the gap, because shuffling reorders existing records without adding any
B. It fills the gap by redistributing records across groups
C. It converts the underrepresented group into the majority
D. It removes the need to measure bias metrics

Answer & Explanation

Correct Answer: A. Nothing about the gap, because shuffling reorders existing records without adding any

Explanation:
Shuffling randomizes row positions to prevent temporal or ordering biases during mini-batch gradient descent and cross-validation splits. However, shuffling is purely a permutation operation—it does not change class or group proportions, nor does it add new data. Resolving dataset underrepresentation or class imbalance requires techniques like stratified sampling, oversampling (e.g., SMOTE), class re-weighting, or collecting additional real-world data.

Why the other options are incorrect: * B. It fills the gap by redistributing records across groups: Reordering existing rows does not alter feature counts or generate missing representation. * C. It converts the underrepresented group into the majority: Shuffling preserves the exact mathematical ratio of all existing categories and groups. * D. It removes the need to measure bias metrics: Row order has no effect on underlying statistical bias; measuring bias and fairness metrics remains essential.

Class imbalance and AI training data integrity

In practice, this group of skills lets the machine learning engineer handle datasets where one outcome dominates, and verify the data that customizes a generative AI model.

Be comfortable doing the following:

Resolving class imbalance in numeric, text, and image datasets, and recognizing the standard levers: adding examples of the rare class, reducing examples of the common class, generating additional variation for the rare class, or weighting the rare class more heavily during training.
Validating AI training data integrity, including checking that prompt and response pairs are well formed and screening content for safety before it reaches a model.
Recognizing that overall accuracy is a misleading measure when one class dominates.

Class imbalance and representation bias are distinct problems that get treated as one. Imbalance is about the outcome label: one answer appears far more often than another, so a model can score well by predicting the common answer nearly every time. Bias is about who or what is represented in the data at all. A dataset can be perfectly balanced across outcome labels and still badly misrepresent a group, and it can represent every group faithfully and still be heavily imbalanced. The remedies are not interchangeable, and rebalancing by duplicating rare-class records can deepen a representation problem if those records all come from the same narrow slice.

For generative AI data, integrity checks are structural as well as statistical. A malformed pair, where a response does not answer its prompt or a field is missing, teaches the model the wrong association. Safety screening runs before the data reaches customization, because content that goes in can come back out in a model response.

Knowledge Check

One outcome label dominates a dataset, and a model trained on it reports high overall accuracy. Why is that measure misleading? (Choose 1)

A. A model can score well by predicting the dominant label nearly every time
B. Accuracy cannot be calculated on imbalanced datasets
C. Imbalance makes the model train more slowly
D. The dominant label carries less information than the rare one

Answer & Explanation

Correct Answer: A. A model can score well by predicting the dominant label nearly every time

Explanation:
This phenomenon is known as the accuracy paradox. In highly imbalanced datasets (e.g., 99% negative cases and 1% positive cases), a naive model that predicts the majority/dominant class 100% of the time will achieve 99% overall accuracy despite failing to identify a single minority instance. For imbalanced datasets, evaluation metrics such as Precision, Recall, F1-score, and Area Under the Precision-Recall Curve (PR-AUC) provide a much clearer evaluation of minority-class detection performance.

Why the other options are incorrect: * B. Accuracy cannot be calculated on imbalanced datasets: Accuracy can easily be calculated mathematically (\(\frac{\text{True Positives} + \text{True Negatives}}{\text{Total Samples}}\)); it is simply uninformative and misleading for imbalanced distributions. * C. Imbalance makes the model train more slowly: Class imbalance affects model convergence quality and prediction bias, but it does not inherently increase computation time or slow down training iterations. * D. The dominant label carries less information than the rare one: While minority examples are often of higher business value (e.g., fraud or rare disease detection), accuracy is misleading specifically because majority-class performance masks minority-class failure.

Knowledge Check

A training dataset was collected in batches grouped by region, and one region is barely represented. Which technique addresses the underrepresentation specifically? (Choose 1)

A. Shuffling records so collection order does not align with region
B. Augmenting the underrepresented region to add variation for it
C. Filling missing numeric values with the column average
D. Converting the dataset to a columnar format

Answer & Explanation

Correct Answer: A. Shuffling records so collection order does not align with region

Explanation:
When data is ingested or collected in sequential batches grouped by a category (like region), training without shuffling causes mini-batch gradient descent to process entire regions sequentially. The underrepresented region gets isolated into a tiny batch whose weight updates are quickly overwritten by subsequent large regional batches. Shuffling breaks the alignment between collection order and batching, ensuring samples from the sparse region are distributed evenly across all mini-batches throughout the entire training epoch.

Why the other options are incorrect: * B. Augmenting the underrepresented region to add variation for it: While augmentation adds synthetic sample diversity, it does not fix the underlying mini-batch ordering artifact caused by regional batch grouping. * C. Filling missing numeric values with the column average: Mean imputation addresses missing feature values within columns, not batch sampling or group distribution across training steps. * D. Converting the dataset to a columnar format: Columnar storage (e.g., Parquet) optimizes disk I/O and query speed, but has no effect on batch sequence order or training sample distribution.