Skill 1.2.8: Mask, Redact, and Anonymize Data
Task 1.2: Data Preparation for ML and AI – Perform data transformation, feature engineering, and pre-processing
Objective: Remove or obscure sensitive data to satisfy privacy/compliance, enable safe ML/GenAI training, RAG, fine-tuning, analytics, and prevent re-identification.
Core Concepts
- Masking: Hides a value while preserving format/length/shape. Often reversible via tokenization/vault. Use for display, analytics, referential integrity.
- Redaction: Permanently removes or obscures sensitive content. Irreversible. Use for text, transcripts, training data, logs.
- Anonymization: Irreversibly prevents identification, including via indirect/quasi-identifiers. Techniques: generalization, suppression, perturbation, k-anonymity, differential privacy.
- Pseudonymization/Tokenization: Replaces identifiers with tokens; reversible with a key. Not anonymization under GDPR/CCPA if re-identification is possible.
- De-identification: Umbrella term covering masking, redaction, anonymization, pseudonymization.
Exam gotcha: Masking preserves shape; redaction removes; anonymization must be irreversible. If a key can reverse it, it is pseudonymization—not anonymization.
Technique Decision Matrix
| Technique | Goal | Reversible? | Typical Use | AWS Tooling |
|---|---|---|---|---|
| Masking | Hide value, keep format/length | Often yes (token vault) | Display, analytics, referential integrity | DataBrew, Comprehend (MASK mode), Bedrock Guardrails |
| Redaction | Remove/obscure content | No | Text, transcripts, training data | Comprehend, Transcribe, Comprehend Medical, Bedrock Guardrails |
| Anonymization | Prevent re-identification | No | Data sharing, research, publishing | EMR/Spark, Glue, custom Lambda |
| Tokenization/Pseudonymization | Replace with token | Yes with key | Consistent joins, pseudonymous IDs | Lambda, Glue, DataBrew |
| Generalization/Suppression | Reduce precision/remove quasi-identifiers | No | k-anonymity, l-diversity | EMR/Spark, Glue |
| Perturbation/Differential Privacy | Add noise/protect stats | No | Statistical release, ML training | EMR/Spark, custom |
AWS Service Mapping
| Service | Role | Best For | Gotcha |
|---|---|---|---|
| Amazon Macie | Discover/classify sensitive data in S3 | Finding PII, credentials, custom identifiers; automated discovery; findings to Security Hub/EventBridge | Does NOT mask/redact/anonymize. Discovery only. |
| Amazon Comprehend | PII detection/redaction in text | English/Spanish text; real-time and async S3 jobs; RedactionMode MASK or REPLACE_WITH_PII_ENTITY_TYPE | Not for S3 estate discovery; not an anonymization guarantee. |
| Amazon Comprehend Medical | PHI detection in medical text | DetectPHI; offsets for redaction | Medical domain only. |
| Amazon Transcribe | PII redaction in transcripts | Audio/video transcription redaction; batch and real-time | Redacts transcripts, not raw audio. |
| Amazon Bedrock Guardrails | Runtime PII masking/blocking | FM prompts/responses, RAG, agents; sensitive information filters | Runtime policy, not batch data prep. |
| AWS Glue DataBrew | Visual data prep | Profiling, mask/replace transforms, low-code recipes | Visual; not for massive cluster-scale anonymization. |
| AWS Glue | Serverless ETL | Batch masking/redaction at scale; Spark transforms | Requires IAM role; manage job bookmarks. |
| Amazon EMR | Managed Spark | k-anonymity, differential privacy, complex anonymization | Cluster management overhead. |
| AWS Lambda | Event-driven compute | Per-record masking/redaction in streams | Stateful sliding windows need a stream processor. |
| SageMaker Feature Store | Feature storage | Store post-transformed features for training/serving | Do not store raw PII; encrypt with KMS. |
| Lake Formation | Fine-grained access control | Column/row-level security for analytics | Access control, not anonymization. |
| KMS / IAM / VPC endpoints | Security | Encrypt at rest/in transit, least privilege, private access | Macie/Comprehend/Bedrock/Glue support varies. |
Key API/Feature Nuances (Conceptual)
- Macie: Classification jobs, automated sensitive data discovery, custom data identifiers, allow lists. Findings are metadata; no data mutation.
- Comprehend: DetectPiiEntities (real-time), StartPiiEntitiesDetectionJob (async). Redaction modes: MASK (asterisks) or REPLACE_WITH_PII_ENTITY_TYPE. Supports English/Spanish.
- Comprehend Medical: DetectPHI returns offsets; you redact.
- Transcribe: Content redaction with RedactionType=PII; output redacted and/or unredacted.
- Bedrock Guardrails: Sensitive information filters; can block or mask PII; custom regex; applies to input/output.
- DataBrew: Profile job surfaces sensitive columns; transformations include Mask, Replace, Shuffle, etc.
- Glue/EMR: Use Spark for generalization, suppression, perturbation, k-anonymity, differential privacy.
Critical Exam Rules
Redact/anonymize BEFORE training, fine-tuning, continued pre-training, distillation, or RAG ingestion. Anything left in training data can surface in model outputs.
Macie finds; Comprehend redacts. Do not use Macie to redact. Do not use Comprehend to discover an S3 estate.
Metadata is data. For RAG, sanitize document metadata (author, email, file path) before indexing; it can leak PII and affect retrieval.
Streaming state: Per-record masking runs in Lambda. Sliding-window or stateful anonymization requires a stream processor (Managed Service for Apache Flink, Spark Structured Streaming on EMR).
Feature Store stores post-masked features. Never store raw PII in Feature Store. Encrypt with KMS and restrict IAM.
When to Use X vs Y
| Decision | Use X | Use Y |
|---|---|---|
| Mask vs Redact | Mask when format/length must be preserved (last-4, fixed-width, joins). | Redact when sensitive value must be permanently removed. |
| Redact vs Anonymize | Redact removes direct identifiers. | Anonymize also handles indirect/quasi-identifiers (zip, age, gender) to prevent re-identification. |
| Comprehend vs Macie | Comprehend for text PII detection/redaction. | Macie for S3 discovery/classification. |
| Bedrock Guardrails vs Comprehend | Guardrails for runtime FM I/O. | Comprehend for batch/async text redaction. |
| DataBrew vs EMR | DataBrew for visual, low-code, profiling. | EMR for cluster-scale anonymization algorithms. |
| Pseudonymization vs Anonymization | Pseudonymization if you need to re-identify with a key. | Anonymization if re-identification must be impossible. |
Security & Compliance
- IAM: Least privilege; separate roles for discovery (Macie) vs transformation (Glue/Comprehend/Lambda). Use service-linked roles where required.
- KMS: SSE-KMS for S3, Glue, DataBrew, Feature Store; TLS in transit. Customer-managed keys for audit/control.
- VPC endpoints: S3 gateway endpoint for data; interface endpoints for Comprehend, Bedrock, Glue where supported. Keep PII off the public internet.
- CloudTrail: Log API calls and data access. Macie findings to EventBridge/Security Hub for automated response.
- Lake Formation: Column/row-level security for analytics; not a substitute for anonymization.
- Compliance: GDPR/CCPA/HIPAA. Anonymization must be irreversible. Pseudonymization is still personal data. Redact PHI before ML.
Anti-Patterns
- Using Macie as a redaction tool.
- Masking only direct identifiers and calling it anonymization.
- Leaving PII in metadata, logs, error messages, or model outputs.
- Redacting after training or after storing in Feature Store.
- Using reversible tokenization and calling it anonymization.
- Refitting preprocessing transforms on inference data after masking (training-serving skew).
- Ignoring indirect identifiers that allow re-identification.
- For RAG, chunking/embedding documents before redaction.