Skip to content

Skill 1.3.4: Optimize multimodal data distributions by applying bias metrics across numeric, text, and image assets

Task 1.3: Data Preparation for ML and AI - Validate data quality and manage bias

Core objective: Ensure training data represents the target population across numeric, text, and image modalities. Measure bias before training, mitigate by defect, validate integrity/safety, and monitor post-deployment.

AWS Service Map

Need AWS Service Key Capability
Declarative DQ rules AWS Glue Data Quality DQDL rulesets: completeness, ranges, uniqueness
Profile before clean AWS Glue DataBrew Missing values, outliers, inconsistent formats
Bias metrics & explainability SageMaker Clarify Pre/post-training bias for tabular, text, image
Interactive cleaning/balancing SageMaker Data Wrangler Transforms, bias report, class balancing
Labeling & human review SageMaker Ground Truth / A2I Annotation workflows, inter-annotator agreement
Content safety screening Amazon Bedrock Guardrails Content filters, denied topics, word/PII filters
PII detection Amazon Comprehend English/Spanish PII in text
Job observability Amazon CloudWatch Metrics/logs for validation/cleaning jobs

Bias Metrics & Mitigation by Modality

Modality Bias Risk Example Metrics Mitigation Levers
Numeric/Tabular Group underrepresentation, label imbalance, proxy features CI, DPL, KL, JS, LP, TVD; per-group mean/variance/KS Stratified sampling, reweighting, SMOTE, class weights, drop proxies
Text Stereotypes, toxicity, sentiment skew, embedding bias Demographic parity in sentiment/toxicity; WEAT/SEAT; label distribution Balanced sampling, paraphrase/back-translation, dedup, PII redaction, content filters
Image Attribute underrepresentation, context bias, label noise Per-group accuracy, attribute distribution, label agreement, fairness gaps Augmentation (flip/crop/color), synthetic data, re-sampling, re-weighting, balanced batches
Cross-modal Misaligned image-text pairs, prompt-response mismatch Pair validity, alignment scores, safety scores Pair validation, content safety screening, dedup, alignment filtering

Distribution Optimization Levers

Defect Correct Lever Exam Note
Collection order bias Shuffling Reorders only; does not add representation
Group missing/underrepresented Stratified sampling, augmentation, synthetic data Must add real or synthetic variation for the missing group
Split leakage Stratified/group splitting Keeps groups in train, validation, and test
Class imbalance Oversample, undersample, class weights, SMOTE Do not rely on overall accuracy
Missing values Impute or drop Mean imputation pulls distribution to center; dropping preserves distribution but loses row
Duplicates Deduplication Near-duplicates inflate shared patterns
Label ambiguity Refine instructions, measure agreement Persistent disagreement = instruction problem, not annotator problem

Class Imbalance vs Representation Bias

Problem Definition Wrong Fix Right Fix
Class imbalance Outcome label dominates Overall accuracy PR-AUC, F1, recall; resample/reweight/SMOTE
Representation bias Group missing/underrepresented Shuffling Collect/augment rare group, stratified splitting, reweighting

Gotcha: Shuffling reorders existing records; it adds no representation. Rebalancing by duplicating a narrow slice can deepen bias.

Validation & Integrity Checks

  • Prompt-response pair validation: response must answer prompt, no missing fields, no malformed associations.
  • Content safety screening: run before customization; content that goes in can come back out.
  • PII redaction: use Comprehend or Bedrock sensitive information filters.
  • Inter-annotator agreement: measure via Ground Truth; low agreement = ambiguous instructions.
  • Post-deployment monitoring: Clarify + Model Monitor for bias drift.

Anti-Patterns & Exam Traps

  • Shuffling for missing groups – no effect on representation.
  • Mean imputation without checking missingness pattern – pulls distribution to center, may hide group-specific gaps.
  • Accuracy on imbalanced data – misleading; use PR-AUC/F1/recall.
  • Ignoring text/image in multimodal – checking only tabular leaves bias in other modalities.
  • Post-training-only bias detection – bias starts before training.
  • Duplicating rare class from one slice – synthetic bias.
  • Treating imbalance and representation bias as same – remedies differ.
  • No inter-annotator agreement – ambiguous labels create blurred boundaries.

Hard rules: - Bias detection is continuous: pre-training, in-training, post-deployment. - Label quality sets the ceiling for model quality. - For generative AI, validate prompt-response pairs and screen content before customization. - Use IAM roles, KMS CMK at rest, TLS in transit, VPC endpoints for S3/SageMaker/Bedrock.

Quick Decision Matrix

Symptom First Tool Action
Missing values, outliers DataBrew Profile, then choose impute/drop/dedup
Need repeatable DQ rules Glue Data Quality Author DQDL ruleset
Bias across groups SageMaker Clarify Compute pre/post-training bias metrics
Class imbalance Data Wrangler / Clarify Resample, reweight, SMOTE, PR-AUC
Toxic/unsafe text Bedrock Guardrails Content filters, denied topics, word filters
PII in text Comprehend Detect and redact
Label ambiguity Ground Truth / A2I Refine instructions, measure agreement