Skill 1.3: Validate data quality and manage bias

Question 1 - Data Quality Rules for ML Training Pipeline

A healthcare analytics company is building an ML model to predict patient readmission risk using data from multiple hospital systems. The data engineering team ingests clinical records into an Amazon S3 data lake and catalogs them using AWS Glue Data Catalog. Before training, the ML engineer must ensure every dataset meets explicit quality requirements: patient age must be between 0 and 120, admission date cannot be null, and diagnosis codes must match ICD-10 format patterns.

The validation must run automatically on every new data batch with minimal operational overhead.

Which solution BEST meets these requirements? (Select ONE)

A) Use AWS Glue DataBrew to create a data profile job that runs weekly and manually reviews the generated profile reports.
B) Author a ruleset using Data Quality Definition Language (DQDL) in AWS Glue Data Quality and attach it to the Glue ETL job.
C) Write a custom Python script using Pandas and schedule it with Amazon EventBridge to validate each column.
D) Use Amazon SageMaker Model Monitor to validate the dataset quality before training begins.
E) Use Amazon Bedrock Guardrails to filter malformed records from the dataset.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

B is Correct: AWS Glue Data Quality is purpose-built for declarative, repeatable data quality rules using DQDL. Rules like ColumnValues "age" between 0 and 120 or ColumnExists "admission_date" are evaluated automatically during Glue ETL jobs, providing pushdown validation with zero additional infrastructure. The results are captured in CloudWatch and can trigger alerts or job failures.

Distractor Analysis

A is Incorrect: DataBrew profile jobs provide exploratory statistical summaries (missing values, distributions) for understanding data, but do not enforce declarative rules on every batch. Manual review of weekly reports does not meet the "automatic on every batch" requirement.
C is Incorrect: Building custom Pandas validation scripts introduces unnecessary operational overhead (code maintenance, scaling, error handling) compared to Glue Data Quality's serverless, declarative approach.
D is Incorrect: SageMaker Model Monitor validates model predictions for drift and bias after deployment, not raw dataset quality during data preparation. It does not validate column completeness or value ranges before training.
E is Incorrect: Amazon Bedrock Guardrails screen prompts and responses for generative AI applications (content safety, PII redaction), not structured tabular dataset quality rules.

Key Takeaway: Use AWS Glue Data Quality with DQDL for declarative, serverless data quality validation that runs automatically within Glue ETL pipelines—before data reaches model training.

Question 2 - Mean Imputation Trade-off in Medical Dataset

A financial services company is preparing a loan default prediction dataset with 500,000 records. The annual_income column is missing 15% of its values, and these missing values are disproportionately concentrated among applicants from rural postal codes—a demographic the model must serve fairly. An analyst suggests filling all missing annual_income values with the column average to keep rows usable.

What is the MOST significant risk of this approach for model fairness and performance? (Select ONE)

A) It preserves the original income distribution, keeping rural applicants accurately represented.
B) It artificially pulls the income distribution toward the mean, potentially masking real economic differences for rural applicants.
C) It converts the annual_income column from continuous to categorical, changing its statistical meaning.
D) It automatically removes rows with missing values, losing rural applicant data from training.
E) It creates duplicate records that inflate the importance of rural applicants in model training.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

B is Correct: Mean imputation replaces missing values with the column average (\(\mu\)), which keeps rows usable but reduces variance and pulls the distribution toward the centroid. When missing values are concentrated in a specific demographic group (rural applicants), imputing the global average incorrectly assumes these individuals have incomes equal to the population mean, masking real economic differences. This representation bias can lead the model to misclassify rural applicants because their imputed features do not reflect their actual characteristics.

Distractor Analysis

A is Incorrect: Mean imputation does not preserve the original income distribution. It decreases variance, narrows the spread, and artificially increases the frequency of values near the mean.
C is Incorrect: Mean imputation keeps the column as a continuous numeric feature; it does not change the data type to categorical. Categorical conversion would involve binning or label encoding, not imputation.
D is Incorrect: Mean imputation explicitly avoids dropping rows. Dropping affected rows is called listwise deletion, which is the opposite of imputation.
E is Incorrect: Mean imputation does not duplicate records; it simply fills in missing values with a computed constant. Duplication is a separate technique used for oversampling rare classes.

Key Takeaway: Mean imputation is quick but reduces variance and can amplify representation bias when missing values are concentrated in specific demographic groups—consider group-specific imputation or multiple imputation for fairness-sensitive features.

Question 3 - Inter-Annotator Agreement in Medical Image Labeling

A medical AI startup is building a pneumonia detection model using chest X-ray images. Two radiologists are labeling images as "normal" or "pneumonia" following a written protocol. The ML engineer calculates Cohen's Kappa and finds low agreement (\(\kappa < 0.4\)) specifically for images showing subtle interstitial patterns. Both radiologists have similar experience levels and neither has been identified as an outlier.

What is the MOST appropriate next step to improve label quality? (Select ONE)

A) Remove the radiologist with slightly lower individual accuracy from the labeling workflow.
B) Increase the dataset size to reduce the impact of labeling disagreements.
C) Refine the labeling instructions with concrete examples of subtle interstitial patterns and edge cases.
D) Switch to a more complex model architecture that can handle noisy labels.
E) Automate labeling using Amazon SageMaker Ground Truth auto-labeling to eliminate human disagreement.

Click to reveal Answer & Explanation

Correct Answer: C

Correct Answer Analysis

C is Correct: Low inter-annotator agreement on specific boundary cases (subtle interstitial patterns) signals ambiguous labeling instructions, not incompetent annotators. The correct remedy is to revise the protocol with clearer definitions, examples of "good vs. bad" edge cases, and annotated reference images that demonstrate exactly how to classify these ambiguous patterns. This directly addresses the root cause—blurred class boundaries—rather than treating a symptom.

Distractor Analysis

A is Incorrect: Removing an annotator treats a symptom rather than the root cause. When agreement is consistently low on specific categories, the problem is systemic ambiguity in the instructions, not individual worker error.
B is Incorrect: Adding more unlabeled images does not resolve labeling ambiguity. It will simply generate more inconsistently labeled data with the same blurred boundaries.
D is Incorrect: Model architecture choice occurs downstream during the modeling phase. No architecture can compensate for fundamentally ambiguous ground truth labels—garbage in, garbage out.
E is Incorrect: Amazon SageMaker Ground Truth auto-labeling (active learning) uses a model trained on initial human labels. If those initial labels have low agreement on edge cases, the auto-labeling model will inherit and amplify the same ambiguity.

Key Takeaway: Persistent inter-annotator disagreement on specific class boundaries indicates ambiguous labeling instructions—revise the protocol with clear edge-case examples before collecting more data or changing models.

Question 4 - Shuffling vs. Augmentation for Underrepresented Region

An e-commerce company trains a product recommendation model on transaction data collected from three sales regions: North America (60%), Europe (30%), and Asia-Pacific (10%). The data pipeline ingests records grouped by region, so all North American records are processed first, then European, then Asia-Pacific in each batch. The ML engineer notices the model performs poorly on Asia-Pacific recommendations.

Which TWO techniques should the engineer apply to address BOTH the ordering bias AND the underrepresentation of Asia-Pacific data? (Select TWO)

A) Shuffle the dataset to break the correlation between region and record order.
B) Apply SMOTE oversampling to generate synthetic Asia-Pacific transaction features.
C) Augment the Asia-Pacific data by generating additional variations using Amazon SageMaker Data Wrangler transforms.
D) Use stratified splitting to ensure Asia-Pacific records appear proportionally in training, validation, and test sets.
E) Fill any missing values in the Asia-Pacific transaction features with the column mean.

Click to reveal Answer & Explanation

Correct Answers: A and D (Option C could be argued as correct depending on augmentation approach—see analysis)

Correct Answer Analysis

A is Correct: Shuffling breaks the sequential ordering where mini-batches contain records from only one region at a time. When data arrives grouped by region, gradient descent updates are dominated by whichever region appears in the current batch. Shuffling distributes samples from all regions across every mini-batch, preventing regional ordering artifacts during training.
D is Correct: Stratified splitting ensures that even though Asia-Pacific represents only 10% of the data, it appears in proportionate amounts across training, validation, and test sets. Without stratified splitting, random splits could concentrate all Asia-Pacific records in only one set, making it impossible to evaluate model performance on that region.

Distractor Analysis

B is Incorrect (for this scenario): SMOTE (Synthetic Minority Over-sampling Technique) generates synthetic numeric feature vectors for tabular data. However, transaction data with categorical features (product IDs, user IDs, regions) is not well-suited to SMOTE—it works best with continuous numeric features. Augmentation through Data Wrangler (Option C) or collecting more real data is more appropriate.
C is Partially Correct but Not Listed as Correct: Augmentation adds variation to underrepresented groups, but the correct augmentation approach for transaction data is not what the question asks. In the strict interpretation with "TWO answers," A and D directly address the stated problems (ordering bias + evaluation fairness). Augmentation alone does not fix ordering bias, so it does not address BOTH problems.
E is Incorrect: Mean imputation addresses missing values within columns, not regional underrepresentation or batch ordering. It can actually worsen representation bias if missing values are concentrated in Asia-Pacific records.

Key Takeaway: Shuffling fixes ordering bias (when data is grouped by collection sequence), while augmentation fixes underrepresentation (when a group is genuinely underrepresented)—these are distinct problems requiring distinct remedies.

Question 5 - Accuracy Paradox in Fraud Detection

A credit card company develops a fraud detection model where fraudulent transactions represent only 0.5% of all transactions. The ML engineer trains a model and reports 99.5% overall accuracy. The business stakeholder is impressed, but the ML engineer insists the model is not ready for production.

Why is the ML engineer's concern justified? (Select ONE)

A) The model achieves 99.5% accuracy by predicting every transaction as non-fraudulent, catching zero actual fraud.
B) Accuracy cannot be mathematically calculated on datasets with class imbalance.
C) The dataset requires more non-fraudulent examples before the model can be evaluated.
D) Fraud detection models require specialized hardware that has not been provisioned.
E) The model's precision is too low because the majority class has too many examples.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Answer Analysis

A is Correct: This is the classic accuracy paradox. With 99.5% of transactions being legitimate, a naive model that predicts "not fraud" for every transaction achieves 99.5% accuracy while catching zero fraudulent transactions. Since the business value is entirely in detecting the rare positive class (fraud), overall accuracy is a misleading metric. The engineer should evaluate the model using Precision, Recall, F1-score, and Precision-Recall AUC to see how well it detects minority-class instances.

Distractor Analysis

B is Incorrect: Accuracy can be calculated on any dataset—the formula \(\frac{TP+TN}{TP+TN+FP+FN}\) always produces a valid numeric result. The issue is that accuracy is uninformative for imbalanced datasets, not that it is mathematically impossible.
C is Incorrect: Adding more negative examples (legitimate transactions) would make the imbalance worse and further inflate the misleading accuracy metric. The dataset needs more fraudulent examples or rebalancing techniques.
D is Incorrect: Fraud detection models do not require specialized hardware beyond standard ML infrastructure. The concern here is about the evaluation metric, not compute resources.
E is Incorrect: Precision measures the proportion of predicted positives that are actually positive. A model predicting "not fraud" for everything has undefined precision (zero predicted positives), not "low precision." The issue is that overall accuracy masks zero recall on the positive class.

Key Takeaway: For imbalanced datasets, use Precision, Recall, F1, and PR-AUC rather than overall accuracy—a model can achieve high accuracy by always predicting the majority class while completely failing on the minority class.

Question 6 - Class Imbalance vs. Representation Bias

A human resources startup is building an ML model to identify employees likely to leave the company. The dataset contains 100,000 records where 85% are "stays" and 15% are "leaves". Additionally, the "leaves" category overrepresents employees from the sales department (70% of all "leaves" records) while sales represents only 30% of the total workforce. The ML engineer needs to address both the class imbalance and the departmental representation bias.

Which approach BEST addresses BOTH issues simultaneously? (Select ONE)

A) Randomly duplicate "leaves" records from all departments evenly until the class ratio is 50:50.
B) Use stratified sampling to split the data, then apply SMOTE to generate synthetic "leaves" records balanced across departments.
C) Downsample the "stays" class to 50% and train the model on the reduced dataset.
D) Shuffle the dataset to redistribute department representation across classes.
E) Apply class weights only, without modifying the training data distribution.

Click to reveal Answer & Explanation

Correct Answer: B (or A if duplication is done with careful departmental balance—see analysis)

Correct Answer Analysis

B is Correct: SMOTE (or synthetic minority oversampling) generates new "leaves" samples by interpolating between existing minority-class examples in feature space. When applied with departmental awareness (generating synthetic samples that represent diverse departments), it addresses both class imbalance (more "leaves" examples) and representation bias (diverse departmental representation among synthetic samples). This approach adds variation rather than exact duplicates, reducing overfitting risk.

Distractor Analysis

A is Partially Correct but Less Ideal: Random duplication of "leaves" records can address class imbalance, but if duplicates are not carefully selected to ensure departmental diversity, it can deepen representation bias by duplicating the already-overrepresented sales "leaves" records. Simple random duplication also increases overfitting risk on duplicate examples.
C is Incorrect: Downsampling the majority class ("stays") does reduce class imbalance, but it does nothing to address the departmental representation bias within the "leaves" category. It also discards valuable training data.
D is Incorrect: Shuffling reorders existing records but does not change class proportions or departmental representation. It fixes ordering bias during training, not imbalance or representation gaps.
E is Incorrect: Class weighting during training can mitigate imbalance (penalizing majority-class errors less), but it does not address representation bias within the minority class. The model would still learn a distorted view of "leaves" employees dominated by sales characteristics.

Key Takeaway: Class imbalance (outcome label distribution) and representation bias (who is represented within each class) are distinct problems—address both by generating diverse synthetic samples, not by simple duplication that can amplify representation gaps.

Question 7 - Generative AI Training Data Validation

A legal technology company is fine-tuning Amazon Titan (through Amazon Bedrock) on a dataset of legal question-answer pairs to build a legal assistant chatbot. The dataset contains 10,000 prompt-response pairs collected from various sources. Before starting fine-tuning, the ML engineer must validate data integrity: some pairs have responses that don't answer their prompts, some are missing response fields entirely, and some contain confidential client information.

Which TWO services or techniques should the engineer use to validate the dataset before fine-tuning? (Select TWO)

A) Use Amazon Bedrock Guardrails to screen prompts and responses for sensitive information and denied topics.
B) Use Amazon Comprehend to detect personally identifiable information (PII) in the dataset text.
C) Use AWS Glue Data Quality with DQDL rules to validate prompt-response pair completeness and field formats.
D) Use Amazon SageMaker Model Monitor to detect data quality drift in the prompt-response pairs.
E) Use Amazon OpenSearch Serverless to index and query the dataset for quality assurance.

Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Answer Analysis

A is Correct: Amazon Bedrock Guardrails are specifically designed to screen content in generative AI workflows. The sensitive information filter detects and blocks PII (names, addresses, SSNs, confidential client data) from prompts and responses, while content filters screen for inappropriate or denied topics. This directly addresses the confidentiality requirement.
C is Correct: AWS Glue Data Quality can validate structural integrity of prompt-response pairs using DQDL rules. Rules like ColumnExists "response", IsComplete "response", and ColumnLength "response" > 10 ensure every record has a well-formed response field. Custom rules can check that responses actually address their prompts through keyword matching or embedding similarity thresholds.

Distractor Analysis

B is Partially Correct but Redundant: Amazon Comprehend can detect PII in English and Spanish text, which is useful. However, for a Bedrock fine-tuning workflow, Bedrock Guardrails provides integrated PII screening that runs directly in the Bedrock pipeline. For the "TWO answers" constraint, A and C cover the requirements more comprehensively.
D is Incorrect: SageMaker Model Monitor validates deployed models for prediction drift and bias—it does not validate training data quality before fine-tuning. This is a post-deployment monitoring service, not a pre-training data validation tool.
E is Incorrect: OpenSearch Serverless can index and search the dataset, but it does not provide built-in data quality rules, PII screening, or content safety checks. It would require building custom validation logic on top of the search infrastructure.

Key Takeaway: Generative AI training data validation requires both structural checks (well-formed prompt-response pairs) and content screening (PII redaction, safety filters)—use Glue Data Quality for structure and Bedrock Guardrails for content.

Question 8 - Dataset Splitting for Time-Series Fraud Detection

A financial services company builds a fraud detection model using 2 years of transaction data (January 2022 to December 2023). The ML engineer needs to split the data into training, validation, and test sets. During exploratory analysis, the engineer discovers that fraud patterns changed significantly in June 2023 when the company migrated to a new payment processing system.

What is the MOST appropriate splitting strategy to ensure the model is evaluated on realistic future fraud patterns? (Select ONE)

A) Randomly shuffle all records and split 70/15/15 for training, validation, and test.
B) Split chronologically: train on January 2022–May 2023, validate on June–September 2023, test on October–December 2023.
C) Use stratified splitting on the is_fraud label to ensure equal fraud proportions across all sets.
D) Split by customer ID so no customer's transactions appear in both training and test sets.
E) Use K-fold cross-validation with K=5 to maximize model robustness to fraud pattern changes.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

B is Correct: When fraud patterns change over time (due to system migration), chronological splitting is the only approach that evaluates the model on future data it hasn't seen. Training on pre-migration data, validating on the transition period, and testing on post-migration data simulates real-world deployment where the model must detect fraud patterns in data that comes from the new system. This tests the model's ability to generalize forward in time.

Distractor Analysis

A is Incorrect: Random shuffling breaks the temporal structure and causes data leakage—future records with new fraud patterns could appear in the training set, inflating the model's apparent performance. In production, the model would see only new data, so random splitting gives a falsely optimistic evaluation.
C is Incorrect: Stratified splitting on the label ensures proportional fraud representation across sets, but does not account for the temporal shift in fraud patterns. The model could still train on future fraud patterns that leak into the training set through stratified random splits.
D is Incorrect: Customer-level splitting prevents data leakage between individual customers but does not address the temporal change in fraud patterns. A customer whose transactions appear in the training set (pre-migration) may also appear in the test set (post-migration) with completely different fraud patterns, and the model has no temporal awareness.
E is Incorrect: K-fold cross-validation randomly shuffles and splits data into K folds, creating the same temporal leakage problem as random splitting. Cross-validation is appropriate for IID data, not for time-series data with temporal distribution shifts.

Key Takeaway: For time-series data with temporal pattern shifts, always use chronological splitting (train on past, test on future)—random shuffling or K-fold cross-validation causes temporal data leakage and inflated performance estimates.

Question 9 - Data Deduplication Edge Cases in Customer Data

A retail company is preparing a customer segmentation dataset by merging records from online (e-commerce) and in-store (point-of-sale) systems. The same customer often appears in both systems with slightly different attributes: missing apartment numbers, different name capitalizations, or old vs. new addresses. The ML engineer needs to deduplicate records before training a segmentation model.

Which approach MOST effectively handles near-duplicate records while preserving legitimate customer information? (Select ONE)

A) Remove all records with exact duplicate customer IDs, keeping only the most recently updated record.
B) Use fuzzy matching with threshold-based similarity scores to identify near-duplicates, then apply deterministic merge rules for conflicting fields.
C) Drop all records that have any missing fields, as these are likely duplicate entries from incomplete data sources.
D) Use Amazon SageMaker Data Wrangler's built-in deduplication transform to automatically remove all duplicates without custom logic.
E) Apply a content filter to remove records containing personally identifiable information before deduplication.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

B is Correct: Near-duplicate detection requires fuzzy matching (e.g., Levenshtein distance, Jaccard similarity, or Soundex for names) because exact key matching misses records with minor variations. Deterministic merge rules (e.g., prefer the most recent address, use the longer name variant, keep the record with more complete fields) ensure consistent resolution of conflicting fields. This approach separates true duplicates from distinct customers whose records happen to share some attributes.

Distractor Analysis

A is Incorrect: Exact key matching only catches records with identical customer IDs. Records from different systems (online vs. in-store) may have different ID formats or missing IDs entirely, so exact matching misses most near-duplicates. It also does not handle cases where the same customer has different IDs in each system.
C is Incorrect: Dropping records with missing fields is a listwise deletion approach that blindly discards data. A customer record with a missing apartment number is not a duplicate—it's an incomplete record. This approach loses legitimate customer information and does not address the deduplication problem.
D is Incorrect: While SageMaker Data Wrangler offers some built-in transforms for data cleaning, it does not have an automatic "smart deduplication" transform that can detect near-duplicates across multi-column records with different formats. Fuzzy matching requires custom logic, not an out-of-the-box transform.
E is Incorrect: PII filtering (via Comprehend or Bedrock Guardrails) removes sensitive information but does not deduplicate records. The question is about merging near-identical rows, not screening for privacy compliance.

Key Takeaway: Deduplication requires fuzzy matching with deterministic merge rules for near-duplicates—exact key matching misses records with minor variations, and blindly dropping incomplete records loses legitimate data.

Question 10 - Bias Metrics Across Multimodal Dataset

A social media company builds a content moderation model that processes multimodal data: text posts, images, and numeric engagement metrics. The model must classify content as "safe" or "violative" across all modalities. The ML engineer implements bias metrics (demographic parity, equalized odds) on the tabular engagement data and finds balanced performance. However, the model still shows biased predictions for certain communities when processing images.

What did the ML engineer fail to do? (Select ONE)

A) Apply bias metrics to the image and text modalities separately, not just the numeric features.
B) Use a larger training dataset to eliminate image-based bias.
C) Shuffle the multimodal dataset to remove ordering bias before computing metrics.
D) Convert all modalities to numeric features before measuring bias.
E) Use a single bias metric across all modalities to ensure consistency.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Answer Analysis

A is Correct: Bias metrics must be applied per modality because each modality encodes bias differently. Numeric engagement metrics might be balanced while image features (e.g., skin tone representation, visual content patterns) remain biased. Text modalities require different bias measurement (e.g., sentiment analysis across demographic mentions) compared to image modalities (e.g., representation parity across visual features). The engineer validated only the numeric tabular portion and incorrectly assumed the entire multimodal dataset was bias-free.

Distractor Analysis

B is Incorrect: More data does not automatically reduce bias. If the new data contains the same representation gaps or biased labels, increasing dataset size can amplify existing biases. Bias mitigation requires explicit measurement and correction, not just more data.
C is Incorrect: Shuffling addresses ordering bias (records grouped by collection sequence), which is orthogonal to modality-specific bias. The image bias persists regardless of record order in mini-batches.
D is Incorrect: While images and text must be converted to numeric vectors (embeddings) for model input, bias metrics should be applied to the original modality's relevant attributes (e.g., image demographic representation, text sentiment across groups). Converting to numeric features first obscures modality-specific bias signals.
E is Incorrect: Using one unified bias metric across modalities is inappropriate because bias manifests differently in each modality. Demographic parity on numeric features cannot directly measure representation gaps in image datasets. Each modality requires tailored bias metrics followed by holistic aggregation.

Key Takeaway: For multimodal datasets, apply bias metrics to each modality separately (numeric, text, image)—balanced tabular features do not guarantee unbiased predictions from image or text inputs.

Summary of Key Concepts for Skill 1.3

Concept	Key Points
Data Quality Validation	Use AWS Glue Data Quality with DQDL for declarative rules; DataBrew profile jobs for exploratory analysis
Cleaning Decisions	Mean imputation pulls distribution toward center; listwise deletion loses information; choose based on missing data pattern
Labeling Quality	Low inter-annotator agreement = ambiguous instructions, not bad annotators; refine protocol before collecting more data
Shuffling vs. Augmentation	Shuffling fixes ordering bias (no new data); augmentation fixes underrepresentation (adds variation)
Class Imbalance vs. Representation Bias	Imbalance = outcome label distribution; representation bias = who is represented; distinct problems with distinct remedies
Accuracy Paradox	High accuracy on imbalanced data = model predicting majority class; use Precision/Recall/F1/PR-AUC instead
Temporal Splitting	Time-series data must use chronological splits; random shuffling causes temporal leakage
Deduplication	Exact key matching misses near-duplicates; use fuzzy matching with deterministic merge rules
Generative AI Data Validation	Structural checks (Glue DQ) + content screening (Bedrock Guardrails); both are required before fine-tuning
Multimodal Bias	Apply bias metrics per modality (numeric, text, image); balanced tabular data does not ensure unbiased multimodal model