Skill 2.3.2: Create model performance baselines and implement drift detection

Question	Answer & Explanation
### Question 1 - Establishing Data Drift Baselines with SageMaker Model Monitor A retail company has deployed an XGBoost model on an Amazon SageMaker real-time endpoint to predict customer churn. The data science team needs to establish a baseline of the incoming traffic to detect data drift over time. The solution must capture 100% of the inference requests and responses and generate baseline statistics automatically with the least operational overhead. Which approach meets these requirements? - A) Use SageMaker Ground Truth to label incoming data and calculate baseline statistics. - B) Enable data capture on the SageMaker endpoint, then use SageMaker Model Monitor to suggest baselines from the captured data. - C) Stream inference data to Kinesis Data Firehose and compute statistics using a custom AWS Lambda function. - D) Export CloudWatch metrics from the endpoint to an S3 bucket and use Amazon QuickSight to visualize the baseline.	Correct Answer: B Correct Option Analysis: - B is Correct: SageMaker Model Monitor natively supports data capture on real-time endpoints. By enabling data capture and using the `DefaultModelMonitor.suggest_baseline()` API, the service automatically computes and saves baseline statistics and constraints to Amazon S3 with minimal operational overhead. Distractor Analysis: - A is Incorrect: SageMaker Ground Truth is used for labeling raw datasets for training, not for monitoring live inference traffic or generating statistical baselines for drift detection. - C is Incorrect: Building a custom Kinesis and Lambda pipeline introduces unnecessary operational overhead and custom code, whereas Model Monitor provides this functionality natively. - D is Incorrect: CloudWatch operational metrics (like CPUUtilization) do not contain the actual feature payloads required to compute data distribution baselines. > Key Takeaway: SageMaker Model Monitor automates baseline creation by capturing live endpoint traffic and generating statistical constraints, eliminating the need for custom data pipeline code.
### Question 2 - Monitoring Prediction Drift for Fraud Detection A financial institution uses a SageMaker endpoint for real-time fraud detection. The team wants to monitor model predictions for drift, ensuring the model's output distribution does not significantly change from the established baseline. They need to set up an alert if the prediction drift exceeds a specific threshold. Which AWS service configuration will achieve this? - A) Configure a SageMaker Model Monitor monitoring schedule comparing prediction distributions against the baseline constraints. - B) Use SageMaker Shadow Testing to compare live predictions against a new candidate model. - C) Enable Amazon Bedrock Guardrails to filter anomalous predictions. - D) Use AWS Config rules to monitor changes in the endpoint configuration.	Correct Answer: A Correct Option Analysis: - A is Correct: SageMaker Model Monitor can evaluate model quality drift (including prediction drift) by comparing the distribution of live predictions against the baseline constraints derived from a known-good period. Violations are sent to CloudWatch, where alarms can be configured. Distractor Analysis: - B is Incorrect: Shadow Testing is used for safely validating a new model variant against live traffic, not for continuously monitoring the production model's prediction distribution against a historical baseline. - C is Incorrect: Amazon Bedrock Guardrails are designed for GenAI content filtering and safety, not for monitoring numerical prediction drift on traditional ML endpoints. - D is Incorrect: AWS Config tracks infrastructure and configuration changes, not runtime statistical distributions of model predictions. > Key Takeaway: Model Monitor evaluates both data drift (input features) and model quality drift (predictions, accuracy) by comparing live traffic against saved baseline constraints.
### Question 3 - Custom Drift Detection for Skewed Data An ML team has a deployed SageMaker model predicting housing prices. The team wants to detect data drift on a specific feature, `square_footage`, which is heavily skewed. The standard built-in Model Monitor drift detection algorithms are not sufficient for this feature, and the team needs to apply a Kolmogorov-Smirnov (KS) test for accurate drift detection. Which solution meets this requirement with the lowest operational overhead while remaining within AWS native services? - A) Use SageMaker Model Monitor custom preprocessing and postprocessing scripts to compute the KS statistic. - B) Rely on the default Model Monitor baseline constraint violations for skewed data, as it handles all statistical distributions natively. - C) Retrain the model weekly regardless of drift using SageMaker Pipelines. - D) Deploy an AWS Lambda function triggered by S3 events to run the KS test on captured data natively.	Correct Answer: A Correct Option Analysis: - A is Correct: SageMaker Model Monitor allows the use of custom preprocessing and postprocessing scripts. The team can write a custom script to apply the Kolmogorov-Smirnov test and run it within the Model Monitor container, leveraging the native scheduling and data capture infrastructure. Distractor Analysis: - B is Incorrect: While Model Monitor has built-in algorithms, they may not handle all specific statistical tests or skewed distributions optimally without customization. - C is Incorrect: Retraining blindly on a schedule ignores the actual presence of drift and wastes compute resources. - D is Incorrect: Writing a custom Lambda function to process S3 data introduces custom infrastructure and operational overhead compared to extending Model Monitor's existing container. > Key Takeaway: When built-in Model Monitor constraints are insufficient, you can provide custom preprocessing/postprocessing scripts to the Model Monitor execution environment to calculate specialized statistical metrics.
### Question 4 - Monitoring GenAI Output Characteristics Drift A healthcare company is using a foundation model provided by Amazon Bedrock to summarize patient transcripts. Over the last month, the output summaries have become longer and less concise. The team needs to detect this drift in output characteristics (e.g., average response length, toxicity, semantic similarity) on a continuous basis with minimal operational overhead. Which approach should the team use? - A) Deploy the LLM on SageMaker and use Model Monitor's default baseline constraints for text data. - B) Create an Amazon Bedrock Evaluation job using programmatic metrics to run periodically against a held-out set of transcripts. - C) Use SageMaker Ground Truth to have human raters manually check every summary. - D) Monitor the CloudWatch invocations metric for the Bedrock API.	Correct Answer: B Correct Option Analysis: - B is Correct: Amazon Bedrock Evaluations supports programmatic evaluation jobs that can automatically compute metrics like toxicity, semantic similarity, and length. Scheduling this job periodically allows the team to baseline and monitor GenAI output drift without manual intervention. Distractor Analysis: - A is Incorrect: Model Monitor is primarily designed for tabular data drift and model quality, not for evaluating generative text output characteristics of Bedrock foundation models. - C is Incorrect: Manual human rating of every summary does not scale and introduces massive operational overhead. - D is Incorrect: CloudWatch invocations only track API call volume, not the content or quality of the model's text output. > Key Takeaway: For foundation models and GenAI applications, use Amazon Bedrock Evaluations to create programmatic baselines for text quality metrics rather than traditional SageMaker Model Monitor.
### Question 5 - Feature Attribution Drift in Computer Vision An e-commerce platform uses a computer vision model deployed on SageMaker to classify product images. The team wants to detect feature attribution drift to understand if the model is starting to rely on different image pixels or features than it did during training. The solution must use AWS native services that natively support explainability monitoring. Which configuration achieves this? - A) Use Amazon SageMaker Clarify to generate SHAP values for the baseline and monitor feature attribution drift with Model Monitor. - B) Use Amazon Rekognition to detect labels in the images and compare them over time. - C) Deploy a custom PyTorch container and calculate Integrated Gradients manually in real-time using AWS Lambda. - D) Use AWS Glue to run Spark jobs calculating image similarity over time.	Correct Answer: A Correct Option Analysis: - A is Correct: SageMaker Clarify can generate SHAP (SHapley Additive exPlanations) values to establish a baseline of feature importance. Model Monitor can then use this Clarify baseline to continuously monitor for feature attribution drift in production. Distractor Analysis: - B is Incorrect: Amazon Rekognition identifies objects and scenes, but it does not explain why a specific custom model makes a prediction or monitor feature attribution drift. - C is Incorrect: Running custom Integrated Gradients in Lambda introduces heavy operational overhead and is not an integrated AWS drift monitoring solution. - D is Incorrect: AWS Glue image similarity measures generic data drift, not feature attribution drift (the specific features the model relies on for its predictions). > Key Takeaway: SageMaker Clarify establishes feature attribution baselines (e.g., SHAP values), which SageMaker Model Monitor uses to detect if a model's reliance on specific features changes over time in production.
### Question 6 - Latency Drift vs. Data Drift Monitoring An autonomous vehicle startup is deploying a real-time object detection model on a SageMaker endpoint. The team needs to monitor for drift in the model's inference latency, ensuring that P99 latency does not exceed 50 milliseconds. If it does, they want an automated alert. The constraint is to use the most operationally efficient AWS service for this metric. Which solution is the MOST appropriate? - A) Use SageMaker Model Monitor to track the latency metric and send constraint violations to an SNS topic. - B) Configure a CloudWatch Alarm on the SageMaker endpoint's `OverheadLatency` metric with a threshold of 50ms. - C) Set up an AWS Lambda function to ping the endpoint and log the response time to DynamoDB. - D) Use SageMaker Pipelines to retrain the model automatically if the P99 latency increases.	Correct Answer: B Correct Option Analysis: - B is Correct: Latency is an operational performance metric. SageMaker endpoints automatically emit operational metrics like `OverheadLatency` and `ModelLatency` to CloudWatch. Setting a CloudWatch Alarm is the most operationally efficient and native way to alert on latency thresholds. Distractor Analysis: - A is Incorrect: While Model Monitor handles data, model quality, bias, and feature attribution drift, operational metrics like latency are natively handled by CloudWatch, making Model Monitor the wrong tool for this specific constraint. - C is Incorrect: Setting up a custom Lambda function to ping the endpoint introduces unnecessary operational overhead and is redundant given SageMaker's native CloudWatch metrics. - D is Incorrect: Retraining the model will not necessarily fix an infrastructure or endpoint configuration-induced latency issue, and it doesn't fulfill the requirement to "alert" on the threshold. > Key Takeaway: Differentiating between statistical drift (Model Monitor) and operational performance drift (CloudWatch) is critical. Latency, CPU, and memory metrics are monitored via CloudWatch, not Model Monitor.
### Question 7 - Automating Baseline Creation and Monitoring Schedules A machine learning operations (MLOps) team needs to automate the process of capturing inference data, creating a baseline, and continuously monitoring a SageMaker endpoint for data drift. They want to ensure the baseline constraints are generated from a representative sample of the production traffic and stored in Amazon S3. Which sequence of actions fulfills these requirements? - A) Use SageMaker Experiments to track the baseline, and use a Step Functions workflow to compare new data. - B) Enable data capture on the endpoint, use `DefaultModelMonitor.suggest_baseline()` to create the baseline, and schedule monitoring jobs using `create_monitoring_schedule()`. - C) Build a custom Apache Spark pipeline on Amazon EMR to compute statistics and save to S3. - D) Use AWS CloudTrail to log API calls and analyze the distribution of input parameters using Athena.	Correct Answer: B Correct Option Analysis: - B is Correct: This is the exact native SageMaker Model Monitor workflow. Enabling data capture records traffic. `suggest_baseline()` analyzes a sample of this traffic to create baseline statistics and constraints in S3. `create_monitoring_schedule()` runs periodic analysis against this baseline. Distractor Analysis: - A is Incorrect: SageMaker Experiments tracks training parameters and metrics, not live inference data drift, and Step Functions is not the native tool for this baseline comparison. - C is Incorrect: Building an EMR pipeline is highly complex and incurs massive operational overhead compared to the fully managed Model Monitor. - D is Incorrect: CloudTrail logs API metadata, not the actual payload contents required to calculate data distribution statistics. > Key Takeaway: The standard SageMaker Model Monitor lifecycle is: Enable Data Capture -> Suggest Baseline (using captured data) -> Create Monitoring Schedule (comparing new data to baseline).
### Question 8 - Continuous Bias Drift Monitoring A loan approval model has been in production for six months. The compliance team requires continuous monitoring to ensure the model is not developing bias drift—specifically, that the parity in approval rates between different demographic groups does not deviate from the baseline established during training. The team must use an AWS native service with the least operational overhead. Which configuration will achieve this? - A) Configure a SageMaker Model Monitor bias drift monitoring schedule using the baseline config generated by SageMaker Clarify. - B) Run a daily batch transform job and manually calculate bias metrics using pandas in a SageMaker Processing job. - C) Deploy an Amazon Bedrock Evaluation job with human raters to assess fairness. - D) Use Amazon Macie to detect and redact demographic data before inference.	Correct Answer: A Correct Option Analysis: - A is Correct: SageMaker Clarify is used to generate bias metrics (like post-training bias) as a baseline. SageMaker Model Monitor can then use this baseline to automatically schedule and run bias drift monitoring jobs on live endpoint data. Distractor Analysis: - B is Incorrect: Manually calculating bias metrics using custom pandas scripts in a Processing job introduces unnecessary operational overhead and isn't automated natively like Model Monitor. - C is Incorrect: Bedrock Evaluations are for GenAI text/foundation models, not traditional tabular loan approval models. Human raters also introduce high overhead. - D is Incorrect: Amazon Macie protects sensitive data by redacting it, but you cannot monitor bias if the sensitive demographic features are removed before the model evaluates them. > Key Takeaway: Bias drift is monitored by passing a SageMaker Clarify bias baseline configuration to a SageMaker Model Monitor monitoring schedule.
### Question 9 - Automating Retraining Upon Drift Detection A manufacturing company predicts equipment failure using a SageMaker endpoint. The team has set up SageMaker Model Monitor, which recently triggered a CloudWatch alarm for severe data drift in the `temperature` and `vibration` features. The company requires the model to be automatically retrained using the newly captured data when such drift is detected, with minimal manual intervention. Which AWS native solution meets this requirement? - A) Trigger an AWS Lambda function from the CloudWatch alarm that starts a SageMaker Pipeline execution for retraining. - B) Configure SageMaker Model Monitor to automatically retrain the model in-place when a constraint violation occurs. - C) Use Amazon EventBridge to send an SNS notification, requiring an admin to manually trigger the pipeline. - D) Delete the endpoint and deploy a new one using the captured data directly without retraining.	Correct Answer: A Correct Option Analysis: - A is Correct: CloudWatch alarms triggered by Model Monitor constraint violations can natively invoke AWS Lambda. The Lambda function can then programmatically trigger a SageMaker Pipeline execution to start an automated, version-controlled retraining process using the newly captured data. Distractor Analysis: - B is Incorrect: SageMaker Model Monitor detects and alerts on drift, but it does not natively possess an "auto-retrain" feature; it requires an external trigger (like Lambda) to initiate a pipeline. - C is Incorrect: Sending an SNS notification for manual intervention violates the "minimal manual intervention" and "automatically retrained" constraints. - D is Incorrect: Deploying an endpoint with new data without actually retraining the model weights makes no logical sense and will not solve the underlying drift issue. > Key Takeaway: Model Monitor does not auto-retrain. To automate retraining on drift, configure Model Monitor to emit CloudWatch alarms, and use those alarms to trigger AWS Lambda, which starts a SageMaker Pipeline.
### Question 10 - RAG Retrieval Drift Evaluation A legal firm uses a Retrieval-Augmented Generation (RAG) application built on Amazon Bedrock Knowledge Bases. The application retrieves relevant case laws and generates summaries. Over time, the retrieval accuracy seems to be dropping as new case laws are added to the S3 data source, but the generation quality remains stable. The team wants to evaluate and monitor the retrieval component independently to detect drift in retrieval accuracy. Which approach isolates and monitors the retrieval drift? - A) Configure Amazon Bedrock Evaluations to run a retrieval evaluation job assessing context precision and context recall against a held-out dataset periodically. - B) Evaluate the final generated summaries using ROUGE scores to detect retrieval drift. - C) Use SageMaker Model Monitor to track the token count of the LLM responses. - D) Monitor the S3 bucket size in CloudWatch to detect when enough new documents have been added to cause drift.	Correct Answer: A Correct Option Analysis: - A is Correct: Bedrock Evaluations allows for RAG-specific retrieval evaluation. By measuring metrics like Context Precision and Context Recall independently against a known dataset, the team can establish a baseline and monitor for degradation strictly in the retrieval step. Distractor Analysis: - B is Incorrect: ROUGE scores evaluate the final generated text, which conflates retrieval and generation quality. If generation quality is stable but retrieval is dropping, ROUGE might not accurately reflect the root cause. - C is Incorrect: SageMaker Model Monitor tracking token counts is an operational metric and does not measure the semantic accuracy of the retrieved context. - D is Incorrect: S3 bucket size indicates data volume, not retrieval performance or accuracy drift. > Key Takeaway: To detect retrieval drift in a RAG system, use Bedrock Evaluations to monitor retrieval-specific metrics (Context Precision, Context Recall) separately from generation metrics.