Skill 1.3.5: Resolve Class Imbalance in Numeric, Text, and Image Datasets
Task 1.3: Data Preparation for ML and AI – Validate Data Quality and Manage Bias
Exam North Star: Class imbalance = outcome label skew. Representation bias = group representation skew. They are distinct problems; remedies are not interchangeable.
Imbalance vs. Bias: Know the Difference
| Concept | Definition | Primary Remedy | Trap |
|---|---|---|---|
| Class Imbalance | One outcome label appears far more often than another. | Resampling, class weights, focal loss, augmentation. | High accuracy can hide minority-class failure. |
| Representation Bias | A group is under/over-represented in the data. | Collect more data, stratified splitting, targeted augmentation, bias metrics. | Shuffling does not fix a missing group. |
Shuffling reorders existing records; it adds nothing. It removes collection-order bias but cannot fill an underrepresented group.
Detection & Metrics: Do Not Trust Accuracy
- Accuracy paradox: With 99% majority, predicting the majority every time yields 99% accuracy but 0% minority recall.
- Use instead: Precision, Recall, F1, PR-AUC (preferred for imbalance), ROC-AUC, confusion matrix, per-class recall.
- Stratified splitting: Preserves class ratios across train/validation/test. Non-negotiable for imbalanced data.
- SageMaker Clarify: Bias metrics across tabular, text, and image datasets (e.g., DI, DPL, CDD).
- AWS Glue Data Quality: Declarative rules in DQDL for completeness, ranges, uniqueness; evaluate every run.
- AWS Glue DataBrew: Profile jobs surface missing values, unexpected ranges, inconsistent formats before cleaning.
Technique Selection Matrix by Modality
| Modality | Common Techniques | AWS Tooling | Trade-offs / Traps |
|---|---|---|---|
| Numeric / Tabular | Class weights, random oversampling, random undersampling, SMOTE, ADASYN, Borderline-SMOTE, Tomek links, NearMiss, balanced ensembles | SageMaker Data Wrangler (Balance data), SageMaker Clarify, Glue Data Quality | SMOTE can create noise/overlap; undersampling discards majority info; weights do not change data but can overfit minority. |
| Text | Class weights, oversample rare classes, synonym replacement, random insert/delete/swap, back translation, LLM paraphrasing | SageMaker Ground Truth, A2I, Bedrock Guardrails, Comprehend (PII) | Synthetic text can drift semantics; validate prompt-response pairs; screen safety before customization. |
| Image | Flip, rotate, crop, color jitter, mixup, cutmix, GANs, hard negative mining, focal loss, weighted loss, resampling | SageMaker Ground Truth, SageMaker Clarify, SageMaker training | Augmentation must preserve label; GANs add artifacts; focal loss for extreme imbalance; do not duplicate identical images. |
Numeric / Tabular Deep Dive
- Profile → Clean → Balance. DataBrew profile → Glue Data Quality rules → Data Wrangler balancing.
- Missing values: Mean imputation keeps rows but pulls distribution toward the middle. Dropping preserves distribution but loses row information.
- Deduplication: Near-duplicates inflate shared patterns. Treat ambiguous near-duplicates carefully.
- Class weights: Preferred when data is limited. Adjusts loss to penalize minority errors without synthetic data.
- SMOTE: Generates synthetic minority points via k-NN interpolation. Good for continuous numeric features; avoid categorical-heavy or very small minority counts.
- Undersampling: Fast but discards majority information. Use when majority is redundant.
- Stratified splits: Mandatory to preserve class ratios in train/validation/test.
Text Deep Dive
- Imbalance in text classification: Rare class often has fewer documents. Use class weights, oversampling, and augmentation.
- Augmentation: Synonym replacement, random insertion/deletion/swap, back translation, LLM-generated paraphrases. Preserve label semantics.
- Prompt-response pair validation: Response must answer prompt; no missing fields; check formatting.
- Content safety screening: Bedrock Guardrails (content filters, denied topics, word filters, sensitive info filters) before data reaches customization.
- PII detection: Amazon Comprehend detects PII in English/Spanish; clean before training.
- Label quality: Ambiguous instructions cause inter-annotator disagreement. Refine instructions before collecting more data.
Image Deep Dive
- Augmentation: Geometric (flip, rotate, crop, scale), photometric (brightness, contrast, color jitter), mixup/cutmix, GANs. Must preserve class label.
- Hard negative mining: Focus training on confusing majority-class examples.
- Focal loss: Down-weights easy examples, focuses on hard minority examples. Good for extreme imbalance.
- Weighted loss: Assigns higher penalty to minority class.
- Resampling: Oversample rare images or undersample common; beware overfitting to exact duplicates.
- SageMaker Ground Truth: Labeling and annotation workflows.
- SageMaker Clarify: Bias metrics on image datasets.
Generative AI & Multimodal Integrity
- Prompt-response pair validation: Structural integrity; response answers prompt; no missing fields.
- Content safety screening: Bedrock Guardrails before customization. Content that goes in can come back out.
- Multimodal bias metrics: Apply across numeric, text, and image assets—not just tabular.
- Bias starts before training and continues after deployment. Data preparation is where it begins, not where it ends.
AWS Service Map
| Service | Role in Imbalance / Bias |
|---|---|
| AWS Glue Data Quality | Declarative rules (DQDL); completeness, ranges, uniqueness; evaluate every run. |
| AWS Glue DataBrew | Profile jobs; surface missing values, ranges, formats before cleaning. |
| SageMaker Data Wrangler | Interactive cleaning and balancing transforms; visualize effect. |
| SageMaker Clarify | Bias metrics across tabular, text, image; pre- and post-training bias detection. |
| SageMaker Ground Truth | Labeling and annotation workflows; human review. |
| Amazon Augmented AI (A2I) | Human review of ML predictions; can review low-confidence minority cases. |
| Amazon Bedrock Guardrails | Content filters, denied topics, word filters, sensitive info filters; screen prompts/responses. |
| Amazon Comprehend | PII detection in English/Spanish; cleaning and safety screening. |
| Amazon CloudWatch | Metrics/logs for validation and cleaning jobs; make failures visible. |
Anti-Patterns & Exam Gotchas
Shuffling does not fix underrepresentation. It reorders existing records; it adds no new minority examples. Use augmentation, resampling, or collect more data.
Accuracy is misleading when one class dominates. A model can predict the majority class every time and score high. Use PR-AUC, F1, per-class recall.
Class imbalance ≠ representation bias. A dataset can be balanced by label but still misrepresent a group, and vice versa. Remedies are not interchangeable.
Duplicating rare-class records can deepen representation bias if those records come from one narrow slice. Augment with variation, not just copies.
Rebalancing before cleaning can amplify noise. Profile → clean → balance.
SMOTE is not for categorical-heavy or text/image data without adaptation. Use modality-appropriate augmentation.
Stratified splitting is non-negotiable for imbalanced datasets to preserve class ratios in train/validation/test.
Content safety screening must happen before customization; content that goes in can come back out.
Quick Decision Flow
- Profile data (DataBrew) → detect missing, outliers, duplicates.
- Declare rules (Glue Data Quality DQDL) → evaluate every run.
- Check bias metrics (Clarify) across all modalities.
- Split stratified → shuffle to remove collection order.
- Choose remedy by modality:
- Numeric: class weights, SMOTE/ADASYN, undersampling, ensembles.
- Text: class weights, oversampling, augmentation, LLM paraphrases.
- Image: augmentation, focal loss, hard negative mining, weighted loss.
- Validate integrity: prompt-response pairs, PII, safety (Comprehend, Bedrock Guardrails).
- Monitor with CloudWatch.