Skill 1.2.6: Apply advanced text pre-processing techniques (for example, tokenization, domain-specific augmentation)
Task 1.2: Data Preparation for ML and AI — Perform data transformation, feature engineering, and pre-processing
Business objective: convert raw text into tokens and vectors a model can consume, without destroying the domain signal that makes the text predictive. Two outputs matter: tokens for the model's context window and embeddings for similarity. Everything else is a means to one of those.
Exam gotcha: Every limit that matters — context window, chunk size, embedding input max, pricing — is measured in tokens, never words or characters. A "1,000-word document" can exceed a token-bounded chunk limit.
1. Tokenization — the mechanics that drive every downstream limit
| Algorithm | Used by (typical) | Mechanism | When it wins |
|---|---|---|---|
| BPE (Byte-Pair Encoding) | GPT family, RoBERTa | Iteratively merges the most frequent character pairs | General English, robustness to typos/OOV |
| WordPiece | BERT, DistilBERT | Merges by likelihood gain rather than raw frequency; ## marks continuations |
Encoder-style classification/NER |
| SentencePiece / Unigram | T5, mT5, many multilingual models | Language-agnostic, trains on raw text — no whitespace pre-split assumption | Multilingual, CJK, low-resource scripts |
| Word-level | Classic NLP (TF-IDF, Naive Bayes) | Split on whitespace/punctuation | Huge vocab, brittle to OOV and typos |
| Character-level | Rare / noisy text | One token per character | Very long sequences, spelling-heavy domains |
- Subword tokenization is the default answer for OOV. Rare and domain terms shatter into multiple subword tokens rather than becoming
<UNK>— this is why subword models handle jargon, but it also inflates token counts for specialized text (a product code or drug name may cost 5–10 tokens). - Special tokens are structural, not content:
[CLS]/[SEP](BERT-style),<s>/</s>(decoder-style), padding and mask tokens. Truncation/padding is applied after tokenization and must respect the model's max sequence length. - Truncation strategy is a decision, not a default. Head-only truncation silently drops the answer in long documents; head+tail truncation or chunking is the fix.
- Padding side matters: right-padding for training/classification, left-padding for batched autoregressive generation. Mismatched padding produces subtly wrong outputs without erroring.
Hard rule: The tokenizer is part of the model's contract. A tokenizer (or vocabulary) fitted on training data must be reused, never refitted, at inference. Refitting re-maps identical text to different token IDs → training-serving skew, the text equivalent of refitting a scaler.
Rule-of-thumb token math (English): 1 token ≈ 4 characters ≈ 0.75 words; 1,000 tokens ≈ 750 words. Never estimate with this in production — use a token-count API to size chunks and validate prompts against the model's context window.
2. Normalization & cleaning — order is the exam detail
Apply in this sequence; reordering changes results:
- Unicode normalization (NFKC) — collapse full-width/half-width, lookalike homoglyphs, and composed vs. decomposed accents. Prevents near-duplicate tokens that are visually identical.
- Markup/boilerplate removal — HTML tags, headers/footers, nav text. Boilerplate is the #1 cause of "retrieval returns junk."
- Whitespace & control-character cleanup — collapse runs, strip zero-width characters.
- Case folding — lowercasing helps classic models; hurts cased entities, acronyms, and gene/protein names. Cased transformer tokenizers already encode case, so aggressive lowercasing is usually a distractor answer.
- Stopword removal — ✔ for TF-IDF/bag-of-words. ✘ for transformers and embedding models — they need full context and stopwords carry syntactic signal.
- Stemming vs. lemmatization — see below.
- Tokenize → truncate/pad to the model's max sequence length.
| Choice | Stemming (e.g., Porter) | Lemmatization |
|---|---|---|
| Method | Rule-based suffix stripping | Dictionary + POS analysis |
| Output | Often not a real word | Valid dictionary lemma |
| Speed / accuracy | Fast / crude | Slower / accurate |
| Use when | Classic ML, high volume, recall matters | Semantic precision matters, NLP pipelines |
3. Chunking — token-aware splitting for retrieval
| Strategy | Mechanism | Use when |
|---|---|---|
| Fixed size (+ overlap) | Split by token count | Default; homogeneous prose. Managed default ≈ 300 tokens with ~20% overlap if unspecified |
| No chunking | Treat file as one unit | File is already one logical unit (a single ticket, a short FAQ entry) |
| Hierarchical | Parent/child split; retrieve child, return parent | Need precise matching plus surrounding context |
| Semantic | Split on meaning shifts, not length | Dense technical text where topic boundaries ≠ page boundaries |
- Overlap exists to prevent a sentence from being cut across two chunks — but overlap multiplies embedding cost and index size.
- Too large → retrieval drags irrelevant text in and dilutes the vector; also triggers "lost in the middle" at generation time. Too small → the sentence loses the context that made it meaningful. This is a two-sided tradeoff, not a one-way rule.
- Metadata extraction (source doc, section, product, page, effective date) enables pre-filtering before similarity is even computed and enables attribution/citations. Metadata is the lever when retrieval returns correct-but-unusable results.
- Chunk boundaries must respect the embedding model's token limit, not a character count.
4. Embeddings — the transform output
- Encoding ≠ embedding. Categorical encoding produces numbers that are label substitutes carrying no learned meaning; embeddings produce vectors where proximity = semantic similarity. Choosing encoding for text retrieval is a classic distractor.
- Normalize embeddings (L2) when using cosine similarity so inner-product search is equivalent — otherwise ranking is skewed by vector magnitude.
- Dimension tradeoff: fewer dimensions = cheaper/faster vector store, slightly lower recall. Match the index dimension to the model's output dimension exactly.
- Asymmetric workloads (short query vs. long document) may warrant models with distinct query/document modes; mixing modes silently degrades recall.
- Embedding model choice is a one-way door for an existing index — changing models invalidates every stored vector and requires full re-embedding.
Commonly-cited model reference (verify current limits): Amazon Titan Text Embeddings V2 — 8,192-token input, 1,024-dim default (configurable lower). Titan Embeddings G1 — 1,536 dims. Cohere Embed (English/Multilingual v3) — 512-token input, 1,024 dims.
5. Domain-specific augmentation
| Technique | Mechanism | Best for | Failure mode |
|---|---|---|---|
| Synonym replacement | Swap tokens via thesaurus/WordNet | Small text classification sets | Destroys technical precision; flips negation |
| Back-translation | Translate A→pivot→A | Paraphrase diversity, minority classes | Domain terms drift; translation errors become label noise |
| EDA (insert/delete/swap) | Random token perturbation | Robustness to typos | Corrupts numbers, dates, negations |
| LLM paraphrase (Bedrock) | FM generates variants | High-quality paraphrase at scale | Cost; hallucinated facts — requires filtering |
| Domain lexicon / custom vocabulary injection | Register jargon, acronyms, product codes | Specialized fields | Ignored vocabulary → terms shatter into subwords |
| Entity substitution / masking | Replace entities with placeholders | Privacy + generalization | Label alignment must be preserved |
| Targeted minority-class synthesis | Generate examples for rare labels | Imbalanced classification | Overfits to synthetic style |
| Continued pre-training | Self-supervised on unlabeled domain corpus | Absorb vocabulary and writing style | Highest compute cost of all options |
Non-negotiable augmentation rules:
Apply augmentation to the training split only. Augmenting validation/test leaks synthetic patterns into evaluation and inflates metrics. Always dedupe augmented examples against the holdout set.
Augmentation must preserve the label. Any transform that changes meaning creates label noise, which is worse than having less data.
Domain-adaptation ladder — cheapest to most expensive:
- Custom vocabulary / lexicon — teach the tokenizer the domain's terms.
- Prompt engineering / in-context glossary — no training, fastest to iterate.
- RAG — injects knowledge that changes frequently; re-index only.
- Continued pre-training — unlabeled domain text; absorbs vocabulary and style.
- Fine-tuning / SFT — labeled prompt-response pairs (JSONL); teaches behavior.
- Distillation — prepare a prompt set; the managed path generates teacher responses.
Exam trap: "Absorb the vocabulary and writing patterns of a specialized field from a large body of unlabeled documents" = continued pre-training, not fine-tuning, not RAG. Fine-tuning wants labeled pairs; RAG changes no weights; distillation wants prompts, not collected answers.
6. Masking, redaction, anonymization
| Technique | Effect on the value | Reversible? | Use when |
|---|---|---|---|
| Masking | Hides the value, keeps its shape/length (****-****-1234) |
Yes (with mapping) | Need format preserved for parsing/validation |
| Redaction | Removes the value entirely | No | Training data, logs, published artifacts |
| Anonymization | Alters data so an individual can no longer be identified | No | Analytics, sharing, GDPR/CCPA scope |
Order is a hard rule: redaction runs before training/customization, never after. Anything left in the training corpus can surface verbatim in a model output.
- Amazon Comprehend detects PII in English and Spanish — not "all languages."
- Amazon Macie discovers sensitive data across an S3 estate; Comprehend detects entities within a text payload. Locate-then-transform is the correct order.
- Amazon Bedrock Guardrails can detect and mask sensitive information at runtime on prompts and responses — this complements, and does not replace, pre-training redaction.
- Amazon Transcribe offers PII redaction for audio-derived text; Comprehend Medical for healthcare entities.
7. AWS service map for text pre-processing
| Need | Service |
|---|---|
| Managed ingestion: chunk → embed → vector store | Amazon Bedrock Knowledge Bases |
| Embedding models (text + image) | Amazon Bedrock (Titan, Cohere) |
| Vector storage / similarity search | OpenSearch Serverless (vector engine), Aurora PostgreSQL (pgvector), Neptune Analytics, MemoryDB |
| Large-scale distributed text transforms | AWS Glue (Spark), Amazon EMR |
| Low-code visual text cleaning | AWS Glue DataBrew |
| Interactive prep & exploration | SageMaker Data Wrangler / Canvas |
| Reusable text features & embeddings at inference | SageMaker Feature Store |
| PII detection in text | Amazon Comprehend |
| Sensitive data discovery in S3 | Amazon Macie |
| Back-translation augmentation | Amazon Translate |
| Runtime prompt/response PII masking | Bedrock Guardrails |
| Per-record streaming text normalization | AWS Lambda (stateless); Managed Service for Apache Flink for sliding-window state |
8. Anti-patterns & distractor traps
- Refitting a tokenizer, vectorizer, or TF-IDF on inference data → training-serving skew. Fit once on training, persist, reuse.
- Using an embedding model for tabular categorical data (or one-hot encoding for text retrieval) → wrong tool, wrong semantics.
- Applying stopword removal before embedding → strips context the transformer relies on.
- Augmenting the test set → leakage; metrics become meaningless.
- Aggressive lowercasing on cased-entity domains (medical, legal, gene names) → destroys discriminative signal.
- Fixed-size chunking on structured technical documents → splits tables and procedures mid-thought; semantic or hierarchical chunking is correct.
- Estimating token counts from word counts → silent context-window overflow and truncation.
- Redacting after fine-tuning → the PII is already in the weights.
- Choosing fine-tuning when the requirement is "data changes frequently" → RAG. Choosing RAG when the requirement is "adopt the domain's writing style" → continued pre-training.
- Reusing a vector index after swapping embedding models → all stored vectors are invalid; full re-embedding required.