Skip to content

1.3

Study Notes: MLA_C01 Task 1.3 – Ensure Data Integrity and Prepare Data for Modeling

Overview and Key Concepts

This task focuses on the data preparation phase of the ML lifecycle: validating quality, detecting/mitigating bias (especially pre-training), handling class imbalance (CI), applying encryption/anonymization/masking for compliance (PII/PHI/data residency), splitting/shuffling/augmenting data, and configuring efficient loading into training resources (S3, EFS, FSx for Lustre).

Core AWS tools:

  • SageMaker Clarify: Pre-training bias metrics (model-agnostic) such as Class Imbalance (CI), Difference in Proportions of Labels (DPL), and others for numeric/text/image data. Also supports post-training bias and explainability. Generates governance reports.
  • Strategies for CI: Resampling (up/down-sample), synthetic data generation (does not use original samples), data augmentation (artificially creates variants from existing data, e.g., flips/rotations for images). SageMaker Ground Truth for labeling; SageMaker image classification algorithm (supports multi-label, transfer learning, RecordIO or raw .jpg/.png; data_augmentation hyperparameter).
  • Data quality: AWS Glue DataBrew (visual transforms, profiling) + AWS Glue Data Quality (rules via Data Quality Definition Language/DQDL; entry points = Glue Data Catalog or Glue ETL jobs). SageMaker Canvas built-in/no-code transforms + custom transforms + re-sample for time-series (to enforce regular intervals).
  • Preparation for modeling: Randomize/shuffle (critical for most cases to avoid biased weight updates; do not randomize pure time-series/ordered sequential data where order = context, e.g., RNNs). Split train/validation/test. Augment to reduce prediction bias.
  • Loading modes & storage: File mode (copies to EBS, slower) vs Pipe mode (streams RecordIO, faster). Training data locations: S3 (default), Amazon EFS, Amazon FSx for Lustre. Input modes control download-vs-stream.
  • Security/compliance: Encrypt with AWS KMS (S3 SSE-KMS, SageMaker notebook/volume/job/endpoint encryption by passing KMS key). Redact/mask/anonymize via Glue. Data classification. Store secrets in Secrets Manager. For Redshift → S3 subsets: Canvas import + prep flow, or Glue Spark job. Inference pipelines reuse scikit-learn/Spark ML transformers so real-time/batch endpoints receive preprocessed data without external steps.
  • Bias/variance foundation: Balance is required for accurate models. Pre-training metrics detect issues in raw data before any model is fit.

Exam traps & tips (high-yield):

  • Synthetic ≠ Augmented (synthetic = fully generated/no original; augmented = derived from existing).
  • Randomize unless order matters (time-series).
  • Clarify metrics are pre-training (raw data) and can continue post-training/monitoring.
  • Glue Data Quality has two entry points (Catalog + ETL).
  • Preferred image classifier input: RecordIO (or jpg/png); enable augmentation hyperparameter.
  • Encryption: Always prefer customer-managed KMS keys for SageMaker storage volumes + S3; redact before training.
  • Pipe mode + RecordIO for speed on large image sets; File mode is simpler but slower/costlier on copy.
  • Canvas/Glue for secure Redshift extracts; inference pipeline for consistent preprocess at serving time.
  • Small mistakes: Forgetting to shuffle → model “thinks everything is class A”; ignoring CI → poor minority-class recall; unencrypted PII/PHI → compliance failure.

Minimize cost everywhere: Use tiny public/sample CSVs or 10–50 row synthetic data, single small Glue/DataBrew jobs, Clarify on toy data only, delete all resources immediately, avoid long-running training/Studio instances, prefer S3 + short-lived processing jobs over persistent EFS/FSx unless testing the config itself. No large images or multi-GB datasets.

Hands-on Exercises (Low-Cost)

All exercises assume an IAM role/user with least-privilege access to S3, Glue, SageMaker, KMS, and (optionally) Secrets Manager. Create a single small S3 bucket (e.g., mla-c01-task13-<unique>). Delete everything at the end of each exercise. Expected wall-clock time per exercise: 15–40 min. Estimated cost: pennies (S3 storage + request fees + brief Glue/SageMaker processing).

Exercise 1: Data Quality Validation with Glue DataBrew + Glue Data Quality (Covers validating quality, profiling, basic transforms, compliance-ready redaction)

Goal: Profile a small dataset, apply quality rules, mask/redact PII-like columns, and publish cleaned data. Relates directly to “Validating data quality (DataBrew and Glue Data Quality)” and anonymization/masking.

Detailed steps:

  1. Prepare tiny data: Create a local CSV (or upload via console) with ~20–30 rows containing columns such as id, age, income, label (0/1), email (fake PII), notes. Introduce 2–3 missing values and 1 duplicate. Upload to s3://your-bucket/raw/sample.csv.
  2. DataBrew: In console → AWS Glue DataBrew → Datasets → Connect new dataset → S3 → select the CSV. Create a project on it (use default recipe). Run “Profile dataset” (job will be small). Inspect column statistics, missing %, unique counts, data types.
  3. Add transforms in the recipe (no-code):
  4. Fill missing age/income with median or constant.
  5. Remove duplicates.
  6. Mask/redact: Apply “Replace” or “PII redaction-style” transform on email (e.g., hash or replace with ***@***.com).
  7. Optionally bin numeric columns or one-hot a categorical.
  8. Publish the recipe version and create a DataBrew job that writes cleaned output to s3://your-bucket/cleaned/.
  9. Glue Data Quality: In Glue console → Data Catalog (create a simple crawler on the cleaned prefix if needed, or use ETL). Create a Data Quality ruleset using DQDL examples: IsComplete "age", ColumnValues "label" in [0,1], RowCount > 10, IsUnique "id". Run evaluation on the Data Catalog table or attach to a tiny Glue ETL job that reads the cleaned data. Review pass/fail + metrics.
  10. (Optional compliance) Note data residency: Confirm bucket is in your allowed Region; add a bucket policy denying non-approved principals.

Expected results:

  • Profile report shows original missing/dupe rates and distributions.
  • Cleaned S3 output has no missing critical values, redacted email, correct row count.
  • Data Quality run returns “Passed” (or clear failures you can fix). Job logs show rule evaluation.
  • Total resources: 1 short DataBrew job + 1 crawler/ETL evaluation. Delete project, job, crawler, and S3 objects immediately.

Exam traps/tips linked: Glue Data Quality entry points = Catalog and ETL (both work). Always redact/mask PII/PHI before any training data leaves the secure zone. DataBrew visual transforms map to Canvas-style no-code prep. Cost trap: Leaving DataBrew projects or frequent full-table crawls running.

Exercise 2: Detect Pre-Training Bias + Mitigate CI with SageMaker Clarify + Simple Resampling/Augmentation Simulation (Covers bias metrics CI/DPL, Clarify, CI strategies, shuffle/split)

Goal: Compute pre-training bias on a tiny imbalanced dataset, apply resampling, re-measure, and prepare shuffled/split data. Relates to Clarify, CI/DPL, synthetic vs augmented, shuffle importance.

Detailed steps:

  1. Tiny imbalanced data: Create CSV (~40 rows) with features + binary label that is heavily skewed (e.g., 85 % class 0). Upload to S3. (For image flavor you can note the concept; do not upload real images.)
  2. SageMaker processing (Clarify): Use SageMaker Studio or (cheaper) a one-off Processing job / notebook instance of type ml.t3.medium (start only for this exercise, stop immediately). Install/use the Clarify processor. Configure a BiasConfig with label_name="label", facet (e.g., a sensitive column if present), and run pre-training bias analysis. Request metrics including CI and DPL.
  3. Inspect Clarify report (JSON/HTML in S3 output): Note high CI and any DPL values indicating imbalance/fairness issues.
  4. Mitigate:
  5. Resample (down-sample majority or simple oversample minority via pandas in the same notebook).
  6. Conceptual synthetic: Generate 5–10 new minority rows with simple noise (document that true synthetic tools exist outside this tiny lab).
  7. For “augmentation” note: If this were images you would enable the image-classification augmentation hyperparameter or use Albumentations-style transforms; here just duplicate + slight numeric jitter a few minority rows.
  8. Shuffle the balanced data (df.sample(frac=1)), then split 70/15/15 train/val/test (stratified). Write three small CSVs/Parquet files back to S3. (Demonstrate “when not to shuffle” by keeping a separate ordered time-series toy file untouched.)
  9. Re-run Clarify pre-training bias on the new training split; confirm CI/DPL improved.
  10. Clean up: Stop/delete notebook instance or processing job, delete Clarify output, empty S3 prefixes.

Expected results:

  • First Clarify report flags high CI (and possibly DPL).
  • After resampling + shuffle + split you have three clean partitions and a second report showing reduced CI.
  • You can articulate: “Randomizing prevents the model from seeing long runs of one class and locking weights; time-series keeps order.”
  • Cost stays tiny because data is <1 MB and instance runs <15 min.

Exam traps/tips linked: Clarify pre-training metrics are model-agnostic and run on raw data. CI is a classic pre-training metric. Synthetic data does not copy originals; augmentation does. Always shuffle unless sequential context is required. After balancing you still split properly to avoid leakage. Trap: Running Clarify on huge data or leaving Studio apps open.

Exercise 3: Secure Encryption, Redaction & Efficient Data Loading Config (KMS + S3 + conceptual EFS/FSx + Pipe/File mode) (Covers encryption, compliance, loading into training resources)

Goal: Encrypt data at rest, redact, then configure a minimal training job input that demonstrates location + mode choices. Relates to KMS encryption, anonymization, EFS/FSx/S3, file vs pipe mode.

Detailed steps:

  1. KMS: Create a customer-managed CMK (symmetric) in KMS (enable key rotation). Note the key ARN.
  2. S3: Upload the cleaned CSV from Exercise 1. Enable default encryption = SSE-KMS using your CMK. Add a bucket key if desired. (Optional) Create a Glue ETL job or DataBrew step that further redacts any remaining sensitive columns and writes a new encrypted prefix.
  3. SageMaker side: In a short-lived notebook or via console/CLI, create a SageMaker Training job configuration (you do not need to run a full expensive training):
  4. Input DataConfig: S3DataSource pointing to your encrypted prefix, S3DataType=S3Prefix, S3DataDistributionType=FullyReplicated.
  5. Set InputMode="Pipe" (or “File” for comparison) and mention RecordIO conversion for images.
  6. VolumeKmsKeyId and (if using) OutputKmsKeyId = your CMK ARN. For notebooks/Studio also set KMS encryption.
  7. Optionally reference EFS or FSx for Lustre as alternative FileSystemDataSource (do not actually provision EFS/FSx unless you want the config practice—they incur ongoing cost; just document the JSON shape and delete any test mount immediately).
  8. (Bonus low-cost) Create a Secrets Manager secret holding a dummy Redshift credential; show (in comments or a 5-line script) how Canvas or a Glue job would retrieve it at runtime to pull a subset → S3.
  9. Verify: Attempt to read the object without KMS permissions (should fail); with proper role it succeeds. Confirm training job definition accepts the KMS key and Pipe mode.
  10. Tear-down: Schedule CMK deletion (7-day window), empty/delete bucket, delete any job definitions, secrets, and notebook instances.

Expected results:

  • Objects in S3 show SSE-KMS with your key.
  • Training job definition validates with KMS ARNs and chosen InputMode (Pipe preferred for streaming large sets).
  • You understand: “Pass KMS key to encrypt SageMaker storage volumes attached to notebooks, processing, training, tuning, batch transform, and endpoints.” Redaction happened before the data reached the training resource.
  • No long-running clusters; FSx/EFS avoided or deleted in minutes.

Exam traps/tips linked: Always encrypt SageMaker volumes + S3 with KMS when data is confidential. Redact with Glue then train. Pipe mode + RecordIO is faster than File mode (no full EBS copy). EFS/FSx are valid training data sources alongside S3. Canvas or Glue + Secrets Manager is the secure pattern for Redshift extracts. Trap: Using AWS-managed keys only, or forgetting to encrypt the ML storage volume.

Exercise 4 (Optional stretch, still cheap): End-to-End Tiny Prep → Inference Pipeline Sketch (Covers full prepare-for-modeling + consistent serving)

Combine pieces: Take the balanced/split data → apply a scikit-learn Pipeline (imputer + scaler + simple encoder) inside a SageMaker SKLearn processor or notebook → save the fitted transformers → create a SageMaker Inference Pipeline model that chains the preprocessor container + a dummy model container. Deploy nothing (or a very short serverless/real-time endpoint that you delete in <10 min). This shows “reuse the exact training transforms at inference so the model only ever sees preprocessed data.”

Expected result: Serialized preprocessor artifacts in S3; pipeline model definition that can be invoked with raw data. Reinforces the transcript point about scikit-learn + Spark ML Serving containers inside an inference pipeline.

Quick Review Checklist Before Exam

  • List 3–4 pre-training Clarify metrics and what CI/DPL mean.
  • Explain synthetic vs augmented and when to resample vs augment.
  • Know exactly when not to shuffle.
  • Glue Data Quality entry points and DQDL basics.
  • KMS encryption points for every SageMaker resource + S3.
  • File vs Pipe mode trade-offs; S3/EFS/FSx as data sources.
  • How Canvas/Glue help with Redshift and no-code transforms + time-series resample.
  • Compliance: PII/PHI redaction/masking/anonymization + data residency.

Perform the three core exercises once each, screenshot the Clarify report + Data Quality pass + KMS-encrypted object properties, then delete all resources. This gives durable muscle memory for the exact services and decision points tested in Task 1.3 while keeping your bill near zero. Good luck!