Skip to content

Study Notes: MLA_C01 Task 2.3 – Analyze Model Performance

Task Overview

After training, you must rigorously evaluate how well a model performs before (and after) production use. This includes selecting the right metrics for the problem type, establishing baselines, detecting overfitting/underfitting/bias/convergence problems, comparing variants (including shadow deployments), and ensuring experiments are reproducible—while always balancing accuracy vs. training time vs. cost.

Core AWS services involved (low-cost focus):
SageMaker Model Monitor, SageMaker Clarify, SageMaker Debugger, SageMaker Experiments, SageMaker endpoints (shadow testing), CloudWatch, S3 (tiny objects only).
Avoid EMR, large training clusters, persistent endpoints, or Canvas/QuickSight unless absolutely necessary (they drive cost).

Key knowledge & skills covered

  • Metrics: confusion matrix, precision/recall/F1/accuracy, ROC-AUC, RMSE/MAE/MAPE, heatmaps.
  • Baselines for Model Monitor.
  • Overfit/underfit diagnosis & remedies.
  • Clarify bias & explainability metrics.
  • Debugger rules for vanishing gradients, saturated activations, loss not decreasing, etc.
  • Shadow variant vs. production variant comparison.
  • Reproducible tracking with Experiments.
  • Trade-off decisions (performance / time / cost).

Essential Concepts (Exam-Ready Summary)

Problem Type Primary Metrics Secondary / Visualization Baseline / Monitor Notes
Binary Classification Precision, Recall, F1, Accuracy, ROC-AUC, PR-AUC Confusion matrix, ROC/PR curves Model Monitor uses F1, AUC, etc.
Multi-class Macro/Micro F1, Accuracy, Confusion matrix (heatmap) Per-class precision/recall Same as binary + multiclass support
Regression RMSE, MAE, MAPE, R² Residual plots Model Monitor uses MSE/RMSE/MAE
Ranking / Other NDCG, etc. – Custom metrics via Scripts

Overfit vs Underfit (classic trap)

  • Overfit = low train error, high validation error → reduce flexibility (fewer features/n-grams, stronger regularization, more dropout, early stop).
  • Underfit = high error on both → increase flexibility (more features/Cartesian products, larger n-grams, less regularization, more capacity).
  • Not enough data → more examples or more epochs/passes.

Clarify – post-training bias metrics (DPPL, DI, AD, etc.) + feature attributions (SHAP).
Debugger – built-in rules (vanishing_gradient, overfit, loss_not_decreasing…). Hooks save tensors to S3; integrate with EventBridge → Lambda/SNS.
Shadow testing – send % of live traffic to new variant, compare latency/error/accuracy without impacting users.
Experiments – log params, metrics, artifacts; compare runs visually; enable full reproducibility.

Cost-conscious principles for all exercises

  • Use the smallest possible instance (ml.t3.medium or ml.m5.large only when required).
  • Prefer local mode or single short training jobs (<5–10 min).
  • Delete endpoints, monitoring schedules, and Debugger profiling immediately.
  • Keep datasets <1–2 MB (Iris, synthetic, or 1–2 k rows of public data).
  • Stop notebook instances when idle.
  • Use Spot for any training (EnableManagedSpotTraining=True).
  • Never leave Model Monitor or Clarify jobs running overnight.

Hands-on Exercises (Minimized Cost)

Exercise 1: Compute & Interpret Core Metrics + Detect Over/Underfitting (Local + Tiny SageMaker Job)

Goal: Practice metric selection, confusion-matrix/ROC interpretation, and overfit diagnosis—exactly the calculations the exam loves.
Est. cost: < $0.50 (notebook ~30–45 min + one 3-min training job).

Detailed Steps

  1. Launch a SageMaker Notebook instance: ml.t3.medium, volume 5 GB. Stop it the moment you finish.

  2. In a notebook cell, create a tiny synthetic binary-classification dataset (or load Iris and binarize): python from sklearn.datasets import make_classification from sklearn.model_selection import train_test_split X, y = make_classification(n_samples=2000, n_features=20, n_informative=8, weights=[0.8,0.2], random_state=42) X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3, random_state=42)

  3. Train two quick models with scikit-learn (or XGBoost in local mode):

  4. Model A: high-capacity (max_depth=12, no regularization) → expect overfit.
  5. Model B: constrained (max_depth=3, strong reg) → expect underfit or good fit.

  6. Generate predictions and compute: python from sklearn.metrics import (confusion_matrix, classification_report, roc_auc_score, precision_recall_curve, f1_score) print(confusion_matrix(y_val, y_pred)) print(classification_report(...)) print("ROC-AUC:", roc_auc_score(y_val, y_prob)) # Plot PR curve and ROC (save as image, do not need QuickSight)

  7. Manually calculate F1 from the confusion matrix to lock the formula in memory:
    F1 = 2 * (P*R)/(P+R).

  8. (Optional 3-min SageMaker job) Upload the same tiny data to S3, launch a built-in XGBoost training job on ml.m5.large Spot with max_runtime=300. Enable SageMaker Experiments to log train vs validation loss.

  9. Compare train-loss vs val-loss curves. Document whether you see overfit/underfit and list the exact remedies you would apply (feature reduction, reg strength, more data…).

Expected Results

  • Confusion matrix shows class imbalance effect (high accuracy but low recall on minority class—classic trap).
  • Overfit model: train F1 ≈ 0.98, val F1 ≈ 0.75; underfit: both ≈ 0.65.
  • Experiments UI shows side-by-side metric charts and hyper-parameter table.
  • You can articulate “I would decrease n-gram size / increase L2 / add early stopping”.

Exam Traps & Tips Related to This Exercise

  • Trap: Choosing accuracy on imbalanced data → always look at F1 / PR-AUC / confusion matrix.
  • Trap: Forgetting that RMSE is for regression, F1 for classification.
  • Tip: Exam loves the exact remedy list for over/underfit (see transcript). Memorize the three levers: data volume, feature processing, model flexibility/regularization.
  • Tip: “Visualize PR curve” question often points to S3 + QuickSight or CloudWatch; know both, but cheapest is notebook + matplotlib (still valid knowledge).

Exercise 2: Create a Model-Quality Baseline + Run a Short Model Monitor Job + Clarify Bias Check

Goal: Practice baseline creation (Ground Truth style), Model Monitor metrics, and Clarify bias detection—directly maps to “methods to create performance baselines” and “metrics available in SageMaker Clarify”.
Est. cost: < $1.00 (one short processing job + one monitoring job that you stop after first execution).

Detailed Steps

  1. Reuse the notebook from Exercise 1 (or restart ml.t3.medium).

  2. Generate a small “baseline” dataset (CSV) that contains: features + probability score + true label. Upload to s3://your-tiny-bucket/baseline/.
    (Simulate Ground Truth labels—you do not need real Ground Truth jobs.)

  3. Create a Model Quality baseline job: python from sagemaker.model_monitor import ModelQualityMonitor, EndpointInput quality_monitor = ModelQualityMonitor(role=role, instance_count=1, instance_type="ml.m5.large") job = quality_monitor.suggest_baseline( baseline_dataset="s3://.../baseline.csv", dataset_format={"csv": {"header": True}}, problem_type="BinaryClassification", # or Regression ground_truth_attribute="label", inference_attribute="score", output_s3_uri="s3://.../baseline-results/" ) job.run(wait=True) # finishes in 3–5 min

  4. Inspect the generated statistics.json and constraints.json (contain F1, AUC, precision… thresholds).

  5. (Optional but recommended) Create a tiny Clarify bias job on the same data: python from sagemaker.clarify import SageMakerClarifyProcessor, DataConfig, BiasConfig clarify = SageMakerClarifyProcessor(role=role, instance_count=1, instance_type="ml.m5.large") # configure DataConfig + BiasConfig for a sensitive attribute (e.g. synthetic “gender”) clarify.run_bias(...)

  6. Immediately delete any schedule you created (monitor.delete_monitoring_schedule()). Terminate the notebook.

Expected Results

  • constraints.json contains the exact metric names Model Monitor will track (F1, AUC, MSE…).
  • Clarify report shows bias metrics (CI, DPL, etc.) and a simple bar chart of feature attribution.
  • You understand that baselines are just S3 JSON files produced once and then used for continuous comparison.

Exam Traps & Tips Related to This Exercise

  • Trap: Thinking you must use real Ground Truth labeling jobs—any S3 dataset with labels works for the baseline.
  • Trap: Forgetting problem_type (BinaryClassification vs Regression) changes the metrics Model Monitor emits.
  • Tip: Clarify is the answer whenever the question mentions “bias”, “fairness”, or “feature importance / SHAP”.
  • Tip: Model Monitor + CloudWatch alarms is the standard pattern for production drift; know you can also trigger Lambda/SNS.

Exercise 3: Shadow Variant Comparison + Debugger Convergence Check (Very Short-Lived Endpoint)

Goal: Practice shadow testing skill and Debugger rules for convergence—covers “comparing shadow vs production” and “using Model Debugger”.
Est. cost: < $1.50 if you are disciplined (endpoint lives ≤ 15 min).

Detailed Steps

  1. Train two tiny XGBoost models (or reuse previous) and create two Model packages / model objects: model-A (production) and model-B (candidate).

  2. Deploy a real-time endpoint with production variant + shadow variant: python from sagemaker.predictor import Predictor # create EndpointConfig with two variants # ProductionVariant: InitialVariantWeight=1.0, InstanceType="ml.t2.medium" # ShadowVariant: InitialVariantWeight=0.0 (or 0.1), same instance type predictor = model-A.deploy(..., endpoint_name="lowcost-shadow-demo") # then update endpoint to add shadow

  3. Send 50–100 synthetic inference requests (boto3 or predictor).

  4. Immediately open CloudWatch Metrics → SageMaker → Endpoints → compare ModelLatency, Invocation4XX/5XX, and any custom metrics you emitted.

  5. In parallel, re-run one of the training jobs with Debugger enabled: python from sagemaker.debugger import Rule, rule_configs, DebuggerHookConfig debugger_hook = DebuggerHookConfig(s3_output_path="s3://.../debug") rules = [Rule.sagemaker(rule_configs.vanishing_gradient()), Rule.sagemaker(rule_configs.loss_not_decreasing()), Rule.sagemaker(rule_configs.overfit())] estimator = xgb.estimator(..., debugger_hook_config=debugger_hook, rules=rules) estimator.fit(...)

  6. Check the Debugger insights in Studio or download the tensors. Delete the endpoint the moment you finish (predictor.delete_endpoint()).

Expected Results

  • CloudWatch shows side-by-side latency and error rates for production vs shadow (even with tiny traffic you see the metric names).
  • Debugger rule evaluation status appears as “IssuesFound” or “NoIssuesFound”; you can open the saved tensors.
  • You experience the exact workflow for “safe canary/shadow promotion”.

Exam Traps & Tips Related to This Exercise

  • Trap: Confusing shadow testing with A/B testing—shadow does not serve responses to users; it only mirrors traffic.
  • Trap: Forgetting that Debugger rules can emit CloudWatch Events → automated Lambda rollback.
  • Tip: When a question asks “compare new model infrastructure without impacting production”, answer = Shadow Testing.
  • Tip: Convergence issues → Debugger (or TensorBoard hosted in SageMaker Domain). Know the built-in rule names.

Additional Low-Cost Practice Ideas (Optional)

  • Use SageMaker Experiments exclusively (no extra infra) to log 4–5 hyper-parameter runs and generate the comparison table/chart the exam expects.
  • CloudWatch dashboard for notebook metrics (CPU, memory, GPU) while doing a quick load-test loop—answers the “visualize load tests” sample question.
  • Download Debugger or Clarify reports and practice reading them offline (zero cost).

Final Exam Tips for Task 2.3

  • Always match metric to problem type first.
  • Baseline = Model Monitor suggest_baseline on labeled S3 data.
  • Overfit/underfit remedies are high-yield—memorize the transcript list.
  • Clarify = bias + explainability; Debugger = convergence/tensors; Experiments = reproducibility & comparison; Shadow = safe infra/model comparison.
  • Cost/performance trade-off questions: smaller instance + Spot + early stopping + fewer features.
  • Delete everything—endpoints and schedules are the #1 surprise bill.

Perform the three exercises above in a single afternoon; they cover virtually every knowledge and skill bullet in Task 2.3 while keeping the bill under a few dollars. Good luck!