Task 4.1: Monitor Model Inference

MLA-C01 Domain 4 — Study Notes with Low-Cost Hands-on Exercises

1. Task Overview & Key Concepts

In the ML lifecycle, inference + model monitoring is the continuous sixth step. After deployment you must automatically:

Capture inference inputs/outputs
Compare them to a training/baseline distribution
Apply rules that detect data quality issues, model quality degradation, bias drift, and feature attribution drift
Alert and (optionally) trigger re-training

💡 Core principle (ML Well-Architected Lens) A model is only valid while production data remains statistically similar to training data. Drift is expected; monitoring + re-training is mandatory, not optional. “Deploy once and forget” is an anti-pattern that appears in many exam distractors.

Four Drift / Quality Categories You Must Distinguish

Type	What changes	Primary AWS tool	Needs ground truth?
Data quality drift	Schema, missing values, type mismatches, range violations	SageMaker Model Monitor (Data Quality)	No
Data / covariate drift	Feature distributions shift (same labels relationship may still hold)	Model Monitor (Data Quality stats) + Clarify	No
Model quality / concept drift	P(y	x) changes → accuracy, F1, MAE etc. degrade	Model Monitor (Model Quality)
Bias drift	Fairness metrics (DP, CDD, etc.) change over time	SageMaker Clarify + Model Monitor bias	Usually yes (for many metrics)
Feature attribution drift	SHAP / feature importance rankings change	Clarify model explainability monitor	No (uses predictions)

Monitoring Modes

Real-time endpoint — continuous capture → scheduled monitoring jobs
Batch transform — regular/scheduled jobs; cheaper for many exam scenarios
Asynchronous inference — same monitoring concepts apply

Supporting services that frequently appear together: CloudWatch (metrics + alarms), SNS / EventBridge (notifications), SageMaker Pipelines / Projects / Step Functions (retrain orchestration), A2I (human-in-the-loop quality checks), Lookout for Metrics / Glue Data Quality (anomaly detection on stats).

2. Knowledge & Skills Map

Knowledge

Data drift vs concept drift vs bias drift vs attribution drift
Baseline creation from training data
Statistics & constraints (mean, quantiles, completeness…)
ML Lens design principles for monitoring & continuous improvement
When ground truth is required
A/B (production variants) for live performance comparison

Skills

Configure SageMaker Model Monitor (data + model quality)
Use Clarify for bias & feature attribution drift
Wire CloudWatch alarms → SNS / Lambda / retrain pipeline
Detect anomalies in processing / inference workflows
A/B testing with production variants + weight traffic
Human review loops (A2I) as quality ground truth

3. Exercise 1 — Data Quality Baseline + Monitoring Job

SageMaker Model Monitor (Batch-style, lowest cost)

💡 Cost Control — Design goal: < $1–2 total. * No persistent real-time endpoint (biggest cost driver). * Use a short-lived ml.m5.large (or ml.t3.medium if available in region) processing job only. * Tiny CSV (≤ 5–10 MB). Delete all S3 outputs, jobs, and any temporary notebook immediately. * Prefer AWS CloudShell + boto3/CLI over a long-running Notebook/Studio instance. * Run in a cheap region you already use (e.g., us-east-1). Set billing alarm first.

What you learn

How Model Monitor builds a baseline (statistics.json + constraints.json) from training-like data, then runs a monitoring job on “production” data to emit a violation report. This is the foundation for data-quality and distribution-drift detection.

Detailed Steps

Step 0: Pre-flight (5 min)

Create (or reuse) an S3 bucket: s3://sm-monitor-lab-<account>/
Create an IAM role with AmazonSageMakerFullAccess (or tighter: S3 + SageMaker processing).
In CloudWatch → Billing, set an alarm at $5 absolute as safety net.
Upload two tiny CSVs (you can generate with Python locally):
train_baseline.csv — 2,000 rows, numeric + categorical columns (the “training” distribution)
prod_day1.csv — 500 rows drawn from same distribution
prod_day2_drift.csv — 500 rows where one numeric feature mean is shifted +2σ and 15 % nulls introduced

Step 1: Create the baseline (`DefaultModelMonitor.suggest_baseline`)

Use a short Python script in CloudShell (or a notebook you stop after 10 min):

import boto3, sagemaker
from sagemaker.model_monitor import DefaultModelMonitor
from sagemaker.model_monitor.dataset_format import DatasetFormat

role = "arn:aws:iam::ACCOUNT:role/SageMakerExec"
sess = sagemaker.Session()
bucket = "sm-monitor-lab-ACCOUNT"
prefix = "ex1"

monitor = DefaultModelMonitor(
    role=role,
    instance_count=1,
    instance_type="ml.m5.large",   # smallest reasonable
    volume_size_in_gb=5,
    max_runtime_in_seconds=1800,
    sagemaker_session=sess
)

monitor.suggest_baseline(
    baseline_dataset=f"s3://{bucket}/{prefix}/train_baseline.csv",
    dataset_format=DatasetFormat.csv(header=True),
    output_s3_uri=f"s3://{bucket}/{prefix}/baseline/",
    wait=True
)

This launches one Processing job (billable minutes only).

Expected result: Under s3://…/baseline/ you see statistics.json (means, stds, histograms, quantiles, completeness) and constraints.json (min/max, dtype, required columns, non-null fractions). Job status = Completed. Wall-clock usually 3–8 minutes.

Step 2: Run a monitoring job against “good” production data

from sagemaker.model_monitor import DefaultModelMonitor, DatasetFormat
# reuse monitor object or recreate with same role/instance

monitor.run_base_monitoring_job(
    baseline_statistics=f"s3://{bucket}/{prefix}/baseline/statistics.json",
    baseline_constraints=f"s3://{bucket}/{prefix}/baseline/constraints.json",
    dataset_format=DatasetFormat.csv(header=True),
    input_data=f"s3://{bucket}/{prefix}/prod_day1.csv",
    output_s3_uri=f"s3://{bucket}/{prefix}/reports/day1/",
    wait=True
)

Expected result: constraint_violations.json is empty (or only trivial warnings). statistics.json for the new batch is close to baseline. No CloudWatch alarm would fire.

Step 3: Run monitoring against drifted data

Repeat the same call pointing at prod_day2_drift.csv and a new output prefix …/reports/day2/.

Expected result: constraint_violations.json lists violations such as: * data_type_check or completeness_check (nulls) * baseline_drift_check (distribution distance exceeded threshold) * numeric range / quantile shifts You now have concrete evidence of data-quality / covariate drift.

Step 4: Inspect & clean up (mandatory)

Download the two violation files and compare them side-by-side.
Delete all S3 prefixes created, stop any notebook, cancel any leftover processing jobs.
Confirm in Cost Explorer / Billing that only Processing-job charges appear.

Exam link: Model Monitor always needs a baseline created before the model goes to production (or from a trusted historical window). On the exam, options that “just turn on monitoring without baseline” are wrong. Continuous capture is typically enabled on an endpoint with DataCaptureConfig; for batch you feed the CSV/Parquet directly as above — both patterns are fair game.

4. Exercise 2 — Bias Drift & Feature Attribution Drift with Clarify

💡 Cost Control Clarify runs as Processing jobs. Use ml.m5.large, 1 instance, max_runtime 1,200 s, dataset ≤ 3,000 rows. No endpoint required for offline bias/attribution analysis. Total usually < $1.

What you learn

How to compute pre-training and post-training bias metrics, how to generate SHAP feature attributions, and how Model Monitor + Clarify detect bias drift and feature attribution drift when those metrics move beyond thresholds.

Detailed Steps

Step 1: Prepare a small labeled dataset with a sensitive attribute

Example columns: age, income, credit_score, gender, loan_approved.

Upload clarify_train.csv and a slightly shifted clarify_prod.csv (e.g., different gender ratio + changed approval rate for one group).

Step 2: Run Clarify bias analysis (offline)

from sagemaker.clarify import (
    SageMakerClarifyProcessor, DataConfig, BiasConfig, ModelConfig
)

clarify = SageMakerClarifyProcessor(
    role=role,
    instance_count=1,
    instance_type="ml.m5.large",
    sagemaker_session=sess
)

data_cfg = DataConfig(
    s3_data_input_path=f"s3://{bucket}/ex2/clarify_train.csv",
    s3_output_path=f"s3://{bucket}/ex2/bias-out/",
    label="loan_approved",
    headers=["age","income","credit_score","gender","loan_approved"],
    dataset_type="text/csv"
)

bias_cfg = BiasConfig(
    label_values_or_threshold=[1],
    facet_name="gender",
    facet_values_or_threshold=["female"]
)

# For post-training bias you also need ModelConfig pointing to a model,
# but for pure pre-training bias you can skip model.
clarify.run_pre_training_bias(
    data_config=data_cfg,
    data_bias_config=bias_cfg,
    methods=["CI","DPL","KL"],   # class imbalance, difference in pos probs, KL
    wait=True
)

Expected result: Analysis file (JSON/HTML) containing bias metrics. Note the baseline values (e.g., DPL = 0.04). You will later compare production runs against these numbers.

Step 3: Feature attribution (SHAP) baseline

If you already have a trained model artifact in S3 (even a tiny XGBoost/SKLearn model from a previous lab), configure ModelConfig + SHAPConfig and call run_explainability. Save the global SHAP ranking as your attribution baseline.

No model? Skip live SHAP and simply study the output schema from AWS docs / sample notebooks so you can recognize “feature attribution drift” questions. (Still valuable for the exam.)

Step 4: Simulate drift detection

Re-run the same bias job on clarify_prod.csv.
Manually (or with a few lines of Python) compute delta of DPL / CDD / etc.
Decide a threshold (e.g., |ΔDPL| > 0.1) → “bias drift detected”.

In production this comparison is automated by Model Monitor’s bias monitoring job schedule.

Expected result: Clear numeric change in at least one bias metric and/or a reordered top-k SHAP features. You understand why Clarify is cited whenever the exam mentions “bias drift” or “feature attribution drift”.

⚠️ Trap: Clarify bias monitoring and Model Monitor data-quality monitoring are complementary, not interchangeable. Data-quality jobs do not compute fairness metrics; you need Clarify integration (or bias-specific Model Monitor config) for bias drift. Also, many bias metrics need ground-truth labels — options that claim bias drift can be detected from features alone are often wrong.

5. Exercise 3 — Model Quality Monitoring + CloudWatch Alert + Retrain Signal

💡 Cost Control Still avoid a 24/7 endpoint. Use: * Batch-style model quality job (predictions CSV + ground-truth CSV). * CloudWatch alarm on a custom metric you publish from a Lambda (free tier usually covers the alarm + a few Lambda invocations; if no free tier, Lambda is still cents). * Optional: EventBridge rule that starts a SageMaker Pipeline execution (Pipeline definition itself is free; only the pipeline run costs money — do not actually run training unless you use a free/local mock step). Target: < $1.

What you learn

Model quality monitoring compares predictions vs ground truth (accuracy, F1, MAE…). You emit metrics to CloudWatch, alarm on threshold breach, and wire a retrain trigger — exactly the pattern described in the exam guide (“analyze performance against defined metrics → alarm or re-training event”).

Detailed Steps

Step 1: Create prediction & ground-truth files

predictions.csv — columns: id, probability, predicted_label
groundtruth.csv — columns: id, label
Make a “good” pair (high accuracy) and a “degraded” pair (accuracy drops 15 points).

Step 2: Run Model Quality baseline + monitoring job

from sagemaker.model_monitor import ModelQualityMonitor
from sagemaker.model_monitor import DatasetFormat

mq = ModelQualityMonitor(
    role=role,
    instance_count=1,
    instance_type="ml.m5.large",
    max_runtime_in_seconds=1500,
    sagemaker_session=sess
)

# suggest_baseline with ground truth joins predictions
mq.suggest_baseline(
    baseline_dataset=f"s3://{bucket}/ex3/baseline_joined.csv",  # pred+label
    dataset_format=DatasetFormat.csv(header=True),
    problem_type="BinaryClassification",
    inference_attribute="predicted_label",
    probability_attribute="probability",
    ground_truth_attribute="label",
    output_s3_uri=f"s3://{bucket}/ex3/mq-baseline/",
    wait=True
)

Then execute a monitoring job on the degraded day.

Expected result: Baseline constraints contain thresholds for accuracy / F1 / AUC etc. The degraded run produces violations in constraint_violations.json. This is “model quality drift / concept drift” evidence.

Step 3: Publish a custom CloudWatch metric & create an alarm

Simple Lambda (or even a one-off CLI call) that parses the violation file and does:

aws cloudwatch put-metric-data \
  --namespace "MLA/ModelMonitor" \
  --metric-name "AccuracyViolation" \
  --value 1 \
  --dimensions ModelName=RecModel,Stage=prod

Create alarm:

aws cloudwatch put-metric-alarm \
  --alarm-name "ModelAccuracyDrift" \
  --metric-name "AccuracyViolation" \
  --namespace "MLA/ModelMonitor" \
  --statistic Maximum \
  --period 300 \
  --evaluation-periods 1 \
  --threshold 0.5 \
  --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:REGION:ACCOUNT:model-drift-topic

Subscribe your email to the SNS topic (confirm subscription).

Expected result: When you publish value=1 the alarm goes ALARM and you receive the SNS email. This mirrors the “configure alerts … initiate actions if drift observed” exam requirement.

Step 4: Wire a retrain signal (do not burn training $)

Create an EventBridge rule: source = CloudWatch Alarm state change → target = Lambda that only logs “Would start SageMaker Pipeline X” (or starts a Pipeline whose first step is a no-op / cheap Processing step that exits immediately).
Alternatively document the pattern: S3 new-data event → CloudTrail/EventBridge → Step Functions → SageMaker Training step. Know the pattern cold; you do not need a full training run for learning.

Exam link: Model quality monitoring requires ground truth. Data quality monitoring does not. Many questions test exactly this distinction. Re-training triggers can be: (a) schedule, (b) drift threshold, (c) new data arrival. SageMaker Pipelines, Projects, Step Functions Data Science SDK, EventBridge + Lambda are all valid orchestration answers.

💡 Human-in-the-loop extension (zero extra infra cost to understand) Amazon A2I can send a sample of predictions for human review. Comparing human labels vs model predictions gives a reliable estimate of degradation and can itself feed ground truth into Model Quality Monitor. Remember A2I for questions about “reliable quality test for model inferences”.

6. Exercise 4 (Optional Micro-lab) — A/B Testing & Endpoint Metrics

💡 Cost Control — Critical Real-time endpoints are the most expensive part of these labs. If you do this exercise: * Use the absolute smallest instance (ml.t2.medium / ml.t3.medium). * Deploy → invoke 20–50 times → capture metrics screenshot → delete the endpoint within 15–20 minutes. * Never leave an endpoint overnight. Set a calendar reminder + CloudWatch billing alarm. * One endpoint with two production variants is enough; no autoscaling needed for the lab. Expected cost if disciplined: A few dollars at most; if left running = tens of dollars/day.

What you learn

Production variants let you shift traffic (e.g., 90/10 or 50/50) between model-A and model-B. CloudWatch emits Invocations, ModelLatency, Invocation4XXErrors, Invocation5XXErrors, etc. per variant. This is how you “monitor model performance in production by using A/B testing”.

Minimal Steps

Create two model objects (same architecture, different model.tar.gz or different hyperparameters).
create_endpoint_config with two ProductionVariants (InitialVariantWeight 0.9 and 0.1, same instance type).
create_endpoint → wait InService.
Invoke with TargetVariant header (or let weights decide).
In CloudWatch → Metrics → SageMaker → Endpoints, compare latency and error rates of VariantA vs VariantB.
Update weights via update_endpoint_weights_and_capacities (no downtime) to finish the A/B test.
Delete endpoint + endpoint config + models immediately.

Expected result: You see per-variant metrics. You understand that A/B testing is a live performance comparison technique, distinct from offline Model Monitor jobs, and that SageMaker automatic scaling can be attached to the same endpoint later if load requires it.

⚠️ Trap: A/B testing ≠ shadow deployment ≠ blue/green. A/B (production variants) sends live traffic to both and compares business/model metrics. Shadow sends duplicate traffic but only the production model’s answers are returned to the user. Exam questions often mix these terms.

7. Exam Traps & Tips (Mapped to Exercises)

Trap / Misconception	Correct Mental Model	Which exercise proves it
“Just retrain on the original data” or “tune hyperparameters to fix drift”	You must incorporate new/updated data. Regularization or hyperparameter tuning on stale data does not fix distribution shift.	Ex 1 (day2 drift)
Monitoring without a baseline	Baseline (statistics + constraints) is mandatory; created pre-production or from trusted window.	Ex 1 step 1
Confusing data-quality monitor with model-quality monitor	Data quality = features/inputs (no labels). Model quality = predictions vs ground truth (labels required).	Ex 1 vs Ex 3
Claiming bias drift can be seen from Model Monitor data-quality alone	Bias & feature attribution require Clarify (integrated into Model Monitor bias/explainability jobs).	Ex 2
Forgetting alerts / actions	Detection without CloudWatch alarm → SNS / EventBridge / Lambda / Pipeline is incomplete.	Ex 3 steps 3–4
Using only accuracy on imbalanced data	Model quality jobs support F1, AUC, precision, recall, MAE, RMSE… choose metric that matches business.	Ex 3
Leaving real-time endpoints as the only monitoring option	Batch transform + scheduled monitoring jobs are first-class and often cheaper; exam loves both.	Ex 1 & 3 (batch style)
No human oversight	A2I human review of a sample is a valid quality gate and ground-truth source.	Ex 3 note
Ignoring endpoint health metrics	Invocation errors, ModelLatency, CPU/Memory utilization are CloudWatch metrics you should alarm on in addition to drift.	Ex 4
Autoscaling confuses with monitoring	Autoscaling reacts to load (Invocations, CPU…). Monitoring reacts to data/model quality. Both can coexist on same endpoint.	Ex 4

🎯 High-yield exam checklist * [ ] I can name the four issue types: data quality, model quality, bias drift, feature attribution drift. * [ ] I know which of them need ground truth. * [ ] I know the artifacts: statistics.json, constraints.json, constraint_violations.json. * [ ] I can describe a retrain trigger path: Monitor → CloudWatch Alarm → SNS/EventBridge → Pipeline/Step Functions. * [ ] I can explain A/B via production variants and how to read per-variant CloudWatch metrics. * [ ] I remember Glue Data Quality anomaly detection and Lookout for Metrics as alternatives/complements for anomaly detection on data stats. * [ ] I never recommend “retrain on original data only” when customer data has drifted.

8. Quick Reference Cheatsheet

Service / Feature	Use for	Ground truth?
Model Monitor - Data Quality	Schema, missing, distribution drift	No
Model Monitor - Model Quality	Accuracy, F1, MAE… vs labels	Yes
Clarify + MM Bias	Bias drift (DPL, CDD, …)	Usually Yes
Clarify Explainability	Feature attribution (SHAP) drift	No
CloudWatch Endpoint metrics	Latency, errors, invocations	N/A
CloudWatch + SNS / EventBridge	Alerting & triggering retrain	N/A
SageMaker Pipelines / Projects	Orchestrate retrain & redeploy	N/A
Step Functions Data Science SDK	Alternative orchestration	N/A
A2I	Human review → quality / GT labels	Creates GT
Lookout for Metrics	Anomaly detection on time-series	No
Glue Data Quality	ML-based anomaly on data stats	No
Production Variants (A/B)	Live traffic split performance test	Business GT
DataCaptureConfig	Capture req/resp on real-time ep	N/A

Minimal Cost Command Reminders

# Always run cleanups after labs:
aws s3 rm s3://sm-monitor-lab-ACCOUNT/ --recursive
aws sagemaker list-endpoints --status-equals InService
aws sagemaker delete-endpoint --endpoint-name ...
aws sagemaker stop-notebook-instance --notebook-instance-name ...  # if used
aws sagemaker list-processing-jobs --status-equals InProgress

💡 Standing cost-hygiene rules for all four exercises 1. Create a $5 billing alarm before any SageMaker call. 2. Prefer CloudShell + boto3 over Studio/Notebook when possible. 3. Cap every Processing job: max_runtime_in_seconds ≤ 1800, volume 5 GB, 1 × ml.m5.large. 4. Never leave an endpoint running unattended; delete within minutes of finishing Ex 4. 5. Delete S3 monitor outputs after you have copied the JSON reports locally for study. 6. Use datasets of a few thousand rows max — statistical concepts still hold.

MLA-C01 Task 4.1 study notes — Monitor Model Inference. Exercises deliberately use short-lived Processing jobs and optional micro-endpoints so concepts map 1:1 to exam objectives while keeping AWS spend near zero. After each lab, verify in the Billing console that only expected SageMaker Processing (and optionally a few minutes of Hosting) charges appear.

Task 4.1: Monitor Model Inference

Contents

1. Task Overview & Key Concepts

Four Drift / Quality Categories You Must Distinguish

Monitoring Modes

2. Knowledge & Skills Map

Knowledge

Skills

3. Exercise 1 — Data Quality Baseline + Monitoring Job

What you learn

Detailed Steps

Step 0: Pre-flight (5 min)

Step 1: Create the baseline (DefaultModelMonitor.suggest_baseline)

Step 2: Run a monitoring job against “good” production data

Step 3: Run monitoring against drifted data

Step 4: Inspect & clean up (mandatory)

4. Exercise 2 — Bias Drift & Feature Attribution Drift with Clarify

What you learn

Detailed Steps

Step 1: Prepare a small labeled dataset with a sensitive attribute

Step 2: Run Clarify bias analysis (offline)

Step 3: Feature attribution (SHAP) baseline

Step 4: Simulate drift detection

5. Exercise 3 — Model Quality Monitoring + CloudWatch Alert + Retrain Signal

What you learn

Detailed Steps

Step 1: Create prediction & ground-truth files

Step 2: Run Model Quality baseline + monitoring job

Step 3: Publish a custom CloudWatch metric & create an alarm

Step 4: Wire a retrain signal (do not burn training $)

6. Exercise 4 (Optional Micro-lab) — A/B Testing & Endpoint Metrics

What you learn

Minimal Steps

7. Exam Traps & Tips (Mapped to Exercises)

8. Quick Reference Cheatsheet

Minimal Cost Command Reminders

Step 1: Create the baseline (`DefaultModelMonitor.suggest_baseline`)