Skill 1.3.2: Label and annotate data

Task 1.3: Data Preparation for ML and AI — Validate data quality and manage bias

Governing rule: Label quality sets the ceiling on model quality. No amount of training compute, architecture search, or hyperparameter tuning raises it. Budget labeling effort before modeling effort.

Service Selection — Decision Matrix

Need	Service	Why / Trap
Create labels from unlabeled data at scale	SageMaker Ground Truth	Managed labeling jobs; you choose workforce, schema, consolidation
Turnkey labeling, no workforce ops	Ground Truth Plus	AWS-managed expert workforce + project management; priced per labeled object
Human review of model predictions	Amazon Augmented AI (A2I)	Review loops on low-confidence inferences; built-in for Textract & Rekognition
Detect PII before data reaches a workforce	Amazon Comprehend	PII detection in EN/ES text; redact before public-workforce exposure
Screen prompts/responses before customization	Amazon Bedrock Guardrails	Content filters, denied topics, word filters, sensitive-info filters
Measure representation/label bias	SageMaker Clarify	Detects and quantifies — does not mitigate
Balance/clean interactively pre-labeling	SageMaker Data Wrangler	Visual transforms for imbalance and cleaning

Trap: A2I = review, Ground Truth = creation. If the stem says "humans verify low-confidence predictions," it is A2I.

Workforce Types — Cost vs. Confidentiality

Workforce	Cost	Speed	Confidentiality	Use when
Mechanical Turk (public)	Lowest	Fastest	None — no NDAs, no vetting	Simple, high-volume, non-sensitive
Private workforce (Cognito)	Moderate	Moderate	High	Sensitive/regulated data, domain expertise
Vendor-managed (Marketplace)	Higher	Moderate	High (vetted, NDA)	Confidential data + quality guarantee + scale
Ground Truth Plus (AWS-managed)	Highest	Managed	High	No labeling-ops team; end-to-end delivery

Gotcha: A private labeling job requires at least one worker in the private workforce before the job can be created. VPC-only (no-internet) jobs work only with a private workforce.

Labeling Workflow — Ordered Steps

Define the schema — mutually exclusive, exhaustive categories; one decision per item.
Write instructions — per-class definition, positive and negative examples, explicit edge-case rules. Target: two annotators reach the same answer on the same item.
Pilot a small batch, then measure inter-annotator agreement before scaling.
Refine instructions — persistent disagreement is a signal about the rubric, not the annotators.
Scale with workforce + model-assisted pre-labeling / auto-labeling.
QC in-flight via gold/control tasks and per-worker accuracy.
Consolidate annotations into one final label per item.
Audit the output manifest before it feeds training.

Annotation Consolidation

Task type	Default consolidation
Classification / NER	Majority vote
Bounding box / segmentation	IoU-weighted merge across annotators
Custom need	Custom consolidation Lambda — you own the merge logic

Overlapping boxes are normal; consolidation merges rather than picking one worker's answer.
Annotator density (3–5 workers/item) trades cost for reliability; 1 worker/item yields no agreement signal at all.
Consolidated output is what trains the model — not the raw per-worker annotations.

Quality Control Levers

Gold/control tasks — known-answer items embedded in the input manifest; per-worker accuracy computed against them; low-accuracy workers filtered out.
Worker metrics — accuracy and agreement with the consolidated label, surfaced per worker.
Pilot → IAA → revise is the cheapest quality lever available.
Model-assisted pre-labeling / auto-labeling — seed a model, auto-accept above a confidence threshold, route the rest to humans. Cuts cost and latency; does not replace QC, and can bake an existing model's bias into the labels.

Measuring Agreement

Metric	Annotators	Notes
Percent agreement	Any	Simple; ignores chance agreement
Cohen's kappa	Exactly 2	Chance-corrected
Fleiss' kappa	3+	Chance-corrected, multi-rater
Krippendorff's alpha	Any; handles missing data	Most flexible

Interpretation rule: low agreement → ambiguous rubric. Fix the instructions; do not collect more data. More unlabeled data just produces more inconsistently labeled noise.

Bias in Labels — Sources & Remedies

Source	Symptom	Remedy
Collection order (batches grouped by region/date)	Mini-batches process whole groups sequentially; sparse group's updates get overwritten	Shuffle before batching/splitting
Representation gap	A group is genuinely missing or thin	Stratified sampling, collect more real data, augment a present-but-thin group
Class imbalance	One outcome label dominates	Resample, class weights, synthesize rare-class variation
Annotator bias	Systematic skew by annotator/culture/language	Multiple annotators + agreement checks + adjudication
Ambiguous boundary	Errors cluster exactly where the rubric was vague	Rewrite instructions with edge cases
Label artifacts	Model learns shortcut cues, not the concept	Audit instructions, mine counter-examples

Hard distinction #1: Imbalance is about the outcome label (one answer dominates). Representation bias is about who/what is in the data at all. A dataset can be perfectly balanced and still badly misrepresent a group — and vice versa. Remedies are not interchangeable.

Hard distinction #2: Shuffling reorders; it adds nothing. It cannot fix a genuine representation gap. Duplicating rare-class records drawn from one narrow slice can deepen representation bias while "fixing" imbalance.

Accuracy is misleading under imbalance. A model predicting the dominant label every time scores ~99% on a 99/1 split. Use Precision, Recall, F1, PR-AUC.

Clarify metric families: pre-training label metrics (CI, DPL, KL, JS, LP, TVD, CDDL) and facet metrics (DI, DCA, DCR) require labels; post-training metrics (DPPL, DI, AD, DCR, RD, TE) require predicted labels plus ground truth.

Multimodal Coverage

Numeric/tabular — facet balance across groups; note mean imputation pulls the distribution toward the center and shrinks variance; listwise deletion preserves distribution but loses the row's other fields.
Text — NER/sequence labeling, intent classification, prompt-response pair validation (does the response actually answer the prompt? are required fields present?).
Image/video — classification, bounding box, semantic/instance segmentation, keypoints; consolidation is IoU-based.
3D point cloud / audio — LiDAR object detection & segmentation; transcription and audio classification.
Apply bias metrics per modality. Checking only the tabular part of a multimodal dataset leaves text and image facets unmeasured.

Security, Privacy & Compliance

IAM execution role needs S3 read on the input-manifest bucket and S3 write on the output bucket; add KMS permissions when using SSE-KMS.
Encrypt at rest with SSE-KMS on input and output buckets; TLS in transit.
VPC-only labeling jobs — private workforce only; requires VPC endpoints for S3, KMS, and the Ground Truth service; no public internet egress.
PII redaction before a public workforce ever sees the data — Comprehend detection, or exclude from MTurk entirely.
Content safety screening runs before data reaches customization/RLHF — content that goes in can come back out in a model response.
Never send confidential, regulated, or PII data to the public workforce — no NDAs, no vetting.

Anti-Patterns & Exam Traps

Skipping the pilot — scaling a vague rubric multiplies ambiguity at full cost.
Blaming annotators for persistent disagreement — it is the rubric.
Using overall accuracy as the label-quality metric on imbalanced data.
Assuming Clarify fixes bias — Clarify measures; mitigation is data-side (resample, reweight, augment, re-collect).
Using A2I when labels don't exist yet — that is Ground Truth.
Using Ground Truth for a pure inference-review loop — that is A2I.
Reaching for augmentation when the group is absent, not merely thin.
Shuffling to fix underrepresentation — reorders what exists, adds nothing.
Ignoring the output manifest — audit the consolidated labels before training.