Skill 1.3.3: Identify and mitigate sources of bias in data by using AWS tools and techniques (for example, dataset splitting, shuffling, augmentation)
Task 1.3: Data Preparation for ML and AI - Validate data quality and manage bias
Exam North Star: Bias is a data-preparation defect, not a model-tuning afterthought. Detect before training; monitor after deployment. Fix the root cause—representation, collection order, label ambiguity—before changing model architecture.
Core objective: Ensure training data represents the target population and does not encode unintended historical, sampling, or annotation skew. AWS tools measure bias, generate labels, transform data, and screen content.
Bias Taxonomy → AWS Mapping
| Bias source | Symptom | AWS tool / technique |
|---|---|---|
| Representation / selection | Group missing or under-represented | SageMaker Clarify, Data Wrangler bias report, augmentation, collect more data |
| Sampling | Collection order correlates with label/group | Shuffling, stratified/group splits |
| Label / annotation | Low inter-annotator agreement | SageMaker Ground Truth, clear instructions, consensus |
| Measurement | Feature measures proxy, not intent | Data Wrangler, Glue DataBrew profiling |
| Temporal | Time-based leakage | Time-series split, not random |
| Class imbalance | One outcome dominates | Resampling, class weights, synthetic augmentation, Clarify metrics |
| Multimodal | Bias only in text/image | Per-modality checks, Bedrock Guardrails, Comprehend |
Mitigation Decision Matrix
| Problem | Use | Avoid |
|---|---|---|
| Collection-order bias | Shuffle before split/training | Using order as signal |
| Underrepresented group | Augment, collect more, reweight | Shuffling alone |
| Class imbalance | Stratified split, class weights, resample | Accuracy as metric |
| Group leakage | Group-aware split | Random split |
| Temporal leakage | Time-based split | Random shuffle |
| Label ambiguity | Refine instructions, multi-annotator consensus | Removing annotators |
| Missing values | Impute or drop with documented assumption | Silent imputation |
| Duplicates | Deduplicate, review near-duplicates | Treating near-dupes as distinct |
Splitting, Shuffling, Augmentation
- Shuffling: Reorders existing rows; removes collection-order bias. Does not add missing groups or fix imbalance.
- Splitting: Random for i.i.d.; stratified preserves class ratios; group-aware prevents same entity across splits; temporal for time series. Split before augmentation to avoid synthetic leakage.
- Augmentation: Adds variation to rare/underrepresented classes. Image: crop, flip, rotate, color jitter. Text: paraphrase, back-translation, synonym replacement. Numeric: noise, interpolation. Risk: if rare class samples are narrow, augmentation amplifies the same slice. Validate augmented data with bias metrics.
AWS Service Cheat Sheet
| Service | Primary bias/data-prep role | Exam note |
|---|---|---|
| SageMaker Clarify | Pre-training bias metrics (CI, DPL, KL, JS, LP, TVD, KS, CDD) and post-training bias metrics (DPPL, DCA, RD, AD); SHAP explainability | Configure label, facet, sensitive attributes; integrates with Model Monitor for drift |
| SageMaker Data Wrangler | Visual profiling, bias report, balance data transform, outliers/missing handling | Interactive cleaning before training |
| SageMaker Ground Truth | Managed labeling workflows | Multi-annotator consensus, gold-standard questions, clear instructions; measure inter-annotator agreement |
| AWS Glue Data Quality | Declarative DQDL rulesets: completeness, uniqueness, ranges | Data quality gates, not fairness metrics |
| AWS Glue DataBrew | Profiling jobs surface missing values, ranges, formats | Profile before choosing cleaning method |
| Amazon Bedrock Guardrails | Content filters, denied topics, word filters, sensitive info filters | Screen prompts/responses for GenAI training and inference |
| Amazon Comprehend | PII detection in English/Spanish | Cleaning and safety screening |
| Amazon CloudWatch | Metrics/logs for validation jobs | Make failures visible, not silent |
Class Imbalance vs Representation Bias
Critical distinction: Imbalance = outcome label frequency. Representation bias = who/what is in the data. A dataset can be balanced and biased, or representative and imbalanced. Remedies are not interchangeable.
- Imbalance fixes: Stratified split, class weights, oversample rare, undersample common, generate synthetic rare examples, use PR-AUC/F1/recall.
- Representation fixes: Collect more data from missing groups, augment with diversity, reweight, audit labels.
- Anti-pattern: Duplicating rare-class records from one narrow slice deepens representation bias.
GenAI / Multimodal Integrity
- Validate prompt-response pairs: Response answers prompt, fields complete, no malformed pairs.
- Screen content before customization: Bedrock Guardrails, Comprehend PII.
- Check bias per modality: Numeric, text, image. Tabular-only bias checks miss text/image skew.
- Safety screening before training: Content that goes in can come out.
Security & Compliance
- IAM roles: Least privilege for SageMaker, Glue, DataBrew, Ground Truth, Bedrock.
- Encryption: KMS at rest for S3, SageMaker, Glue; TLS in transit.
- Network: VPC endpoints for S3, SageMaker, Glue; keep data off public internet.
- PII: Comprehend detection; Bedrock Guardrails sensitive info filters; tokenize/mask before training.
- Audit: CloudTrail + CloudWatch Logs for data-prep jobs.
Exam Gotchas / Anti-Patterns
Shuffling does not add data. It only reorders. If a group is missing, shuffling changes nothing about the gap. Accuracy is misleading on imbalanced data. Use precision, recall, F1, PR-AUC. Split before augment. Augmenting before split leaks synthetic copies into validation/test. Random split leaks groups. Use GroupShuffleSplit or equivalent for same-user/entity data. Temporal data must not be randomly shuffled. Use time-based split. Label quality sets model ceiling. Persistent annotator disagreement = fix instructions, not annotators. Bias detection is not one-time. Monitor post-deployment with Clarify + Model Monitor. Multimodal bias needs per-modality checks. Tabular-only audits miss text/image bias. Don't use Glue Data Quality for fairness metrics. It's for data quality rules; use Clarify for bias.
Quick Recall
- Detect: Clarify, Data Wrangler, Glue DataBrew, Ground Truth agreement.
- Mitigate: Split (stratified/group/temporal), shuffle, augment, resample, reweight.
- Validate: Glue Data Quality, DataBrew, Comprehend, Bedrock Guardrails.
- Monitor: CloudWatch, Clarify Model Monitor.