Skill 1.3.6: Validate AI Training Data Integrity (Prompt-Response Pair Validation, Content Safety Screening)

Task 1.3: Data Preparation for ML and AI — Validate Data Quality and Manage Bias

Business objective: Guarantee that every record reaching a customization, fine-tuning, distillation, or RAG-indexing job is parseable, semantically correct, and permissible — because a malformed or unsafe pair teaches the wrong association and content that goes in can come back out in a model response.

Integrity Has Four Lenses — Validate All Four

Lens	Question It Answers	Fails When
Structural	Does the record parse and conform to the target model's schema?	Malformed JSON, missing keys, unknown keys, blank lines, encoding errors
Semantic	Does the response actually answer its prompt?	Off-topic, refusal, hallucination, truncated completion, prompt echoed back
Safety	Is the content permissible to learn from?	Toxicity, PII, denied topics, prompt-injection payloads
Statistical	Is the set representative, deduplicated, and leak-free?	Near-duplicate pairs, train/eval leakage, class or group imbalance

Structural validity ≠ semantic validity. A pair can parse perfectly and still teach the wrong association. Both gates must run — schema checks catch format defects, human or model-judge review catches meaning defects.

Prompt-Response Pair Validation — The Structural Gate

Format: line-delimited JSON (JSONL), UTF-8, one complete object per line, no trailing commas, no BOM, no blank lines.
Schema varies by model family: older text customization uses prompt/completion; chat-style models use a messages array of role/content turns. Validate against the target model's documented schema before submitting — do not assume.
Required keys present and non-empty. null, "", and whitespace-only values fail semantic validation even when the schema passes.
No unknown/extra keys. Bedrock customization applies a strict schema; unexpected fields are a top cause of job rejection.
Length: prompt + completion must fit the target model's context window; flag and drop or re-generate truncated completions (visible as a length-distribution tail with no terminal punctuation).
Preference/RLHF pairs (chosen vs. rejected): the two responses must differ meaningfully. Identical or trivially different pairs carry no learning signal.
Deduplicate before splitting — exact and near-duplicates (embedding or MinHash similarity) inflate whatever pattern they share and silently leak across train/validation boundaries.
Leakage check: confirm no evaluation prompt (or near-duplicate) appears in the training file. Provide an explicit held-out validation set; never evaluate on the training file.
Stratify the split by group, not just randomly, so no group appears in only one split.

Defect → Detection → Fix

Defect	Signal	Correct Action
Malformed JSONL line	Job fails at validation with a line number	Repair or reject the line — never silently skip
Missing required key	Schema check fails	Drop pair or re-generate response
Unknown extra key	Strict schema rejection	Strip to expected keys only
Response doesn't answer prompt	Low alignment score; human/LLM-judge review	Re-label, regenerate, or drop
Near-duplicate pairs	Similarity above threshold	Deduplicate, keep one
Train/eval leakage	Eval prompt found in train set	Purge from training before split
PII in prompt or completion	Sensitive-information filter or PII detection hit	Redact (anonymize) or drop
Toxic / policy-violating content	Content filter or toxicity score hit	Screen out before customization

Content Safety Screening — Service Selection

Need	Service	Key Behavior
Screen arbitrary prompt/response text for policy violations	Amazon Bedrock Guardrails	Applied at inference by default; the `ApplyGuardrail` operation lets you screen any text offline, including training records
Detect/redact PII in text	Amazon Comprehend	English and Spanish only — no other languages
Discover sensitive data at rest in S3	Amazon Macie	Finds PII/sensitive data in buckets before it enters a training pipeline
Moderate image/video assets	Amazon Rekognition Content Moderation	Moderation labels with a confidence threshold; moderation adapters for custom categories
Screen audio	Amazon Transcribe → Comprehend/Guardrails	Transcribe first, then apply text screening to the transcript
Human review of ambiguous or low-confidence cases	Amazon Augmented AI (A2I), SageMaker Ground Truth	Keeps a human in the loop where automation is uncertain
Declarative, repeatable rules on tabular/catalog data	AWS Glue Data Quality	DQDL rulesets evaluated on every run
Profile/EDA before choosing a cleaning approach	AWS Glue DataBrew profile job, Data Wrangler	Profiling comes first — you cannot pick a repair before you know the defect
Observe validation job outcomes	Amazon CloudWatch	Metrics/logs so a validation failure is visible rather than silent

Bedrock Guardrails Filter Levers (Reusable as an Offline Screening Toolkit)

Filter	What It Does	Data-Prep Use
Content filters	Six categories — hate, insults, sexual, violence, misconduct, prompt attacks — each with strength NONE/LOW/MEDIUM/HIGH	Screen both the prompt and the completion field
Denied topics	Natural-language topic definitions with examples	Remove domain-prohibited subject matter
Word filters	Custom exact/regex lists plus a managed profanity list	Strip banned terms deterministically
Sensitive information filters	PII entities via regex or custom patterns, action BLOCK or ANONYMIZE	Redact PII before the data reaches customization
Contextual grounding check	Grounding + relevance scores against a source	Detect unsupported/hallucinated responses in RAG training data
Automated reasoning	Validates responses against logical rules	High-assurance or regulated domains

Integrity vs. Representation — Do Not Conflate

	Integrity	Representation / Imbalance
Concern	Is the record valid and safe?	Is the population faithfully described?
Example defect	Response doesn't answer the prompt; PII present	One outcome label dominates; one group barely appears
Remedy	Repair, re-label, regenerate, redact, drop	Add rare-class examples, reweight, augment, stratified split
Wrong fix	—	Shuffling reorders existing records and adds nothing — it never fills a representation gap

Anti-Patterns

"It parsed, so it's valid." Schema conformance is the first gate, not the only gate.
Expecting Guardrails to clean an S3 training file automatically. Guardrails is inference-time; you must iterate the file and call it yourself.
Screening prompts but not completions. Both fields teach the model.
Using Comprehend PII outside English or Spanish — it will not work; use Guardrails regex/custom filters or Macie instead.
Judging quality by manual sample inspection instead of declared, repeatable, automated rules.
Evaluating on the training file, or splitting before deduplicating.
Reporting overall accuracy on an imbalanced dataset — use precision, recall, F1, and PR-AUC; a model can score well by predicting the dominant label nearly every time.
Fixing imbalance by duplicating rare-class records that all come from one narrow slice — this deepens the representation problem.
Treating bias work as a launch-time or post-deployment activity only. It begins in data preparation and continues after deployment.

Hard rules to memorize: Safety screening must run before the data reaches customization. Bedrock model customization uses a strict schema — unknown keys and malformed lines fail the job rather than being ignored. Comprehend PII detection supports English and Spanish only. Guardrails is not a data-prep pipeline; ApplyGuardrail screens text you send it. Shuffling never fixes underrepresentation. Overall accuracy is uninformative on imbalanced data.

Label quality sets the ceiling on model quality that no amount of training compute can raise. Persistent disagreement between annotators is a signal about ambiguous instructions, not about the annotators — fix the rubric, not the people.