Skip to content

Skill 4.1.3: Detect changes in data distribution that can affect model performance

Question 1 - Drift Detection Pipeline Architecture

A financial services company has deployed a fraud detection model on Amazon SageMaker that processes real-time transaction data. The model's F1 score has dropped from 0.94 to 0.71 over the past month, and the ML engineering team suspects that transaction patterns have shifted. The team needs to implement an automated monitoring solution that captures a statistical baseline of input features, continuously compares incoming traffic against that baseline, and alerts when significant deviation occurs. The solution must minimize operational overhead and integrate natively with SageMaker.

Which TWO steps should the team implement to detect this data distribution change? (Select TWO)

  • A) Enable SageMaker Model Monitor with a data quality monitoring job that uses a baseline dataset and applies statistical rules (Kolmogorov-Smirnov, mean/std deviations) to detect drift.
  • B) Deploy an AWS Lambda function that manually inspects a random sample of 100 transactions daily and logs summary statistics to CloudWatch.
  • C) Configure AWS X-Ray tracing on the SageMaker endpoint to detect changes in input feature distributions.
  • D) Configure Amazon CloudWatch Synthetics canaries to replay a fixed set of test transactions hourly and compare the model's confidence scores.
  • E) Set up a CloudWatch alarm on the drift metric generated by the SageMaker Model Monitor job to notify the team when violations exceed a pre-defined threshold.
Click to reveal Answer & Explanation

Correct Answers: A and E

Correct Options Analysis

  • A is Correct: SageMaker Model Monitor's data quality monitoring automatically creates a statistical baseline from a reference dataset and applies rules (KS-test, mean/std, percentile shifts) to detect data distribution drift on incoming traffic.
  • E is Correct: Model Monitor emits drift metrics to CloudWatch; adding an alarm on these metrics completes the automated detection-to-alerting loop, enabling proactive notification without manual inspection.

Distractor Analysis

  • B is Incorrect: Manually sampling 100 transactions with a Lambda function introduces significant operational overhead and lacks statistical rigor—it cannot detect subtle distributional shifts with confidence.
  • C is Incorrect: AWS X-Ray traces request latency and component dependencies for distributed applications; it does not analyze input feature distributions or detect statistical data drift.
  • D is Incorrect: Synthetics canaries replay predetermined transactions (which are constant and therefore cannot reflect evolving data drift), and they validate endpoint availability rather than statistical changes in the overall input distribution.

Key Takeaway: SageMaker Model Monitor with CloudWatch alarms is the native, low-overhead pattern for automated drift detection; X-Ray and Synthetics are observability tools, not statistical distribution comparators.

Question 2 - Concept Drift vs. Data Drift Identification

A customer support chatbot team uses an embedding-based retrieval model backed by Amazon OpenSearch Serverless. The team has been capturing a daily summary of incoming user queries in near real-time and storing them in Amazon S3. Recently, the retrieval accuracy has declined, but the distribution of user intents and query lengths has remained statistically identical to the baseline captured two months ago. Customer satisfaction surveys indicate users find answers technically correct but not aligned with their current expectations.

Which condition does this scenario describe, and which monitoring approach should the team add to detect it? (Select TWO)

  • A) Data drift, because the input embeddings have shifted.
  • B) Concept drift, because the relationship between input queries and expected answers has changed.
  • C) Introduce human-in-the-loop review of the retrieval results to capture changing user feedback and retrain the model accordingly.
  • D) Add a CloudWatch alarm on the OpenSearch query latency to detect when the retrieval index becomes outdated.
  • E) Enable an anomaly detection job in Amazon QuickSight on query counts to identify when new topics emerge.
Click to reveal Answer & Explanation

Correct Answers: B and C

Correct Options Analysis

  • B is Correct: Concept drift occurs when the mapping from inputs (X) to outputs (Y), or P(Y|X), changes without a shift in P(X). The queries look the same, but the "correct" answers users now expect have evolved.
  • C is Correct: Human feedback loops (e.g., ratings, A/B testing, or periodic labeling of production predictions) are the primary way to surface concept drift, as statistical distribution monitors on input data alone cannot detect it.

Distractor Analysis

  • A is Incorrect: Data drift (covariate shift) describes changes in the distribution of the inputs P(X) itself—here the query patterns remain statistically unchanged.
  • D is Incorrect: OpenSearch query latency measures retrieval speed, not answer relevance or changing user expectations; latency can stay flat while concept drift silently degrades satisfaction.
  • E is Incorrect: QuickSight anomaly detection on query counts identifies volumetric anomalies, not semantic shifts in what constitutes a correct answer.

Key Takeaway: Input distribution monitors (like embeddings-based data drift) cannot detect concept drift—that requires downstream business metrics, labels, or user feedback.

Question 3 - Embedding-Based Drift Detection for Generative AI

A media company is operating a generative AI summarization service powered by a foundation model on Amazon Bedrock. The service processes news articles and generates executive summaries. The team wants to detect drift in the semantic content of incoming articles (e.g., a sudden shift from financial reports to sports coverage) so they can retrain their prompt templates. The pipeline must detect semantic drift without needing labeled ground truth for each article.

Which TWO AWS-native approaches can the team implement to detect semantic drift in incoming article embeddings? (Select TWO)

  • A) Configure an Amazon Bedrock continuous pre-training job to automatically adjust the embedding model whenever new articles arrive.
  • B) Use an AWS Glue job that computes a baseline distribution of input embeddings and periodically runs a multivariate drift test (like Maximum Mean Discrepancy) on current embeddings.
  • C) Create a SageMaker Clarify job to generate Shapley values for each article and alert when feature attribution changes.
  • D) Implement a SageMaker Pipeline that triggers a Processing job to compare current embeddings against a stored baseline using a distance metric (e.g., cosine similarity distribution) and logs the drift score to CloudWatch.
  • E) Enable account-level Amazon Macie to scan S3 buckets for sensitive data and flag articles that contain unexpected content categories.
Click to reveal Answer & Explanation

Correct Answers: B and D

Correct Options Analysis

  • B is Correct: AWS Glue can run scheduled ETL jobs that compute statistical baselines on embedding vectors and apply multivariate drift statistics such as Maximum Mean Discrepancy (MMD) to detect semantic distribution shifts without labels.
  • D is Correct: A SageMaker Pipeline with a Processing job is a serverless, orchestrated way to compare current embeddings against a recorded baseline using appropriate distance metrics (e.g., cosine similarity across the corpus) and publish drift scores to CloudWatch for alerting.

Distractor Analysis

  • A is Incorrect: Bedrock continuous pre-training modifies foundation model weights, not drift detection. It does not quantify distribution shifts between a baseline and current traffic.
  • C is Incorrect: SageMaker Clarify generates feature attributions (SHAP values) to explain individual predictions—it is a model interpretability tool, not a statistical drift detector across a population of inputs.
  • E is Incorrect: Amazon Macie is a data security and privacy service that discovers and classifies sensitive data (e.g., PII) in S3; it does not perform statistical drift analysis on embeddings or model inputs.

Key Takeaway: For LLM input drift, embed the incoming text and use multivariate drift tests (MMD, cosine distribution shift) in a scheduled pipeline—Clarify and Macie solve completely different problems.

Question 4 - Monitoring Upstream Data Quality Failures

A retail ML team runs a nightly AWS Glue job that aggregates customer transactions and writes a Parquet dataset to S3, which is then used by a SageMaker-hosted recommendation model. At 3 AM, the Glue job completes successfully. At 9 AM, the model endpoint begins throwing deserialization errors and its invocation success rate falls from 99.9% to 67%. The team's CloudWatch dashboards show the Glue job completed with exit code 0 and no error logs.

Which monitoring gap does this pattern reveal, and which action should the team take? (Select TWO)

  • A) The pipeline monitors completion status but does not validate output partition counts, row counts, or schema before downstream consumption.
  • B) The SageMaker endpoint is overloaded and needs autoscaling to handle peak traffic.
  • C) Add a post-processing validation step in the Glue job (or a separate SageMaker Pipeline step) to run data quality checks using AWS Glue Data Quality or SageMaker Model Monitor model-quality reports.
  • D) The IAM role attached to the Glue job lacks permissions to invoke the SageMaker endpoint.
  • E) Enable AWS X-Ray tracing on the Glue job to capture ETL execution traces for error isolation.
Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: The Glue job reports success (exit code 0) but its output is malformed—empty partitions, schema mismatches, or corrupt files—triggering downstream inference errors. Completion status alone is insufficient as a success signal.
  • C is Correct: Adding explicit data quality validations (row counts, schema checks, null constraints, partition completeness) after the ETL step ensures that the model endpoint only consumes well-formed data.

Distractor Analysis

  • B is Incorrect: The error type (deserialization errors, not throttling or latency spikes) points to malformed input data, not insufficient serving capacity. Autoscaling addresses request volume, not bad payloads.
  • D is Incorrect: IAM role issues would prevent the Glue job itself from running or writing to S3—not cause downstream deserialization errors at the endpoint at 9 AM.
  • E is Incorrect: X-Ray traces ETL job internals and dependencies; it does not validate the statistical or structural quality of data written to S3.

Key Takeaway: A successful ETL job exit code is not a data-quality guarantee—always validate output partitions, schema, and row counts before the next pipeline stage consumes it.

Question 5 - Real-Time Anomaly Detection on Model Invocation Patterns

A healthcare analytics platform hosts an inference endpoint on Amazon SageMaker that predicts patient readmission risk. The endpoint typically serves 1,500–2,000 invocations per hour with a p95 latency of 180 ms. One afternoon, the invocation count drops to 8 per hour, but the endpoint remains healthy (no container errors, no throttling). The team needs to detect this type of silent failure automatically.

Which monitoring configuration will detect this anomaly? (Select ONE)

  • A) Create a CloudWatch anomaly detection band on the Invocations metric for the SageMaker endpoint and alarm if the observed value falls below the lower band threshold.
  • B) Set up a CloudWatch composite alarm that triggers when CPU utilization of the instance exceeds 85%.
  • C) Configure AWS CloudTrail to log every inference call and raise an event if fewer than 100 calls are recorded in a given hour.
  • D) Use Amazon Inspector to scan the endpoint for vulnerabilities that may have caused callers to be blocked.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: CloudWatch anomaly detection learns the normal pattern of the Invocations metric over time and creates an upper/lower band. When the actual invocation count drops below the lower band (a flat-line anomaly), the alarm fires—detecting "silent" failures where the endpoint is up but not receiving expected traffic.

Distractor Analysis

  • B is Incorrect: A CPU utilization alarm only detects resource contention or high-load conditions. A drop in invocations to near zero would lower CPU usage and never trigger the 85% threshold.
  • C is Incorrect: CloudTrail logs API activity for governance/auditing, not real-time operational metric monitoring. It lacks anomaly detection capabilities for time-series metrics.
  • D is Incorrect: Amazon Inspector is a vulnerability management service (CVE scanning for EC2/Lambda), unrelated to detecting volumetric anomalies in inference traffic.

Key Takeaway: CloudWatch anomaly detection is critical for catching "flat-line" anomalies (e.g., a service that stops receiving traffic) where only high-value threshold alarms would miss the failure.

Question 6 - SageMaker Model Monitor Execution Role and Reporting

A data science team is enabling SageMaker Model Monitor data quality monitoring for their production endpoint. They want the monitoring job to (1) write drift reports to an S3 bucket, (2) emit drift metrics to CloudWatch, and (3) create a visualization of drift over time. The team is unsure about where to view the generated reports and metrics.

Which TWO statements are TRUE about SageMaker Model Monitor output and configuration? (Select TWO)

  • A) Drift reports (statistics and constraint violations) are stored as JSON files in the S3 bucket specified in the monitoring schedule configuration.
  • B) Model Monitor automatically generates CloudWatch dashboards with drift-trend visualizations without any additional configuration.
  • C) Drift metrics are published to CloudWatch metrics under the aws/sagemaker/Endpoints namespace and can be used to create alarms.
  • D) Model Monitor requires the SageMaker execution role to have write access only to CloudWatch Logs, not S3.
  • E) Model Monitor can only evaluate model quality (prediction vs. ground truth) and does not compute feature distribution statistics.
Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: Model Monitor writes violation reports and statistical summaries to the S3 output path defined in the schedule; these JSON reports are queryable with tools like Athena.
  • C is Correct: Drift metrics (e.g., data_drift, violations) are emitted under the aws/sagemaker/Endpoints CloudWatch namespace, enabling alarm creation and dashboard widget configuration.

Distractor Analysis

  • B is Incorrect: Model Monitor does not automatically generate pre-built CloudWatch dashboards; users must create dashboards or alarm widgets referencing the emitted metrics.
  • D is Incorrect: Model Monitor requires both S3 write permissions (for reports) and CloudWatch PutMetricData permissions (for metrics), plus read access to input data paths.
  • E is Incorrect: Model Monitor supports both "data quality monitoring" (input feature distribution statistics and constraints) and "model quality monitoring" (requires ground truth labels).

Key Takeaway: Model Monitor emits drift metrics to CloudWatch under aws/sagemaker/Endpoints and writes JSON violation reports to S3—but dashboard creation is a manual step.

Question 7 - Embedding-Based Data Drift in a RAG Architecture

An enterprise has deployed a Retrieval-Augmented Generation (RAG) system using Amazon Bedrock Knowledge Bases and Amazon OpenSearch Serverless. The system serves internal HR policy questions. The ML team wants to detect when the semantic distribution of incoming employee questions drifts significantly from the baseline (e.g., employees start asking about new benefit types that were not in the training set). The solution must be fully managed, low-code, and must not require provisioning any additional compute instances.

Which approach should the team use? (Select ONE)

  • A) Set up an Amazon SageMaker Processing job that runs a custom Python script on a scheduled basis to batch-embed incoming questions and run a two-sample statistical test against stored baseline embeddings.
  • B) Use Amazon OpenSearch Serverless's built-in anomaly detection on the _score field of queries to detect semantic drift.
  • C) Enable CloudWatch generative AI observability, which automatically detects semantic drift in model inputs and alerts the team.
  • D) Launch a self-hosted Prometheus instance on Amazon EKS to scrape Bedrock invocation logs, compute TF-IDF vectors, and compare drift using a Kolmogorov-Smirnov test.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: A scheduled SageMaker Processing job (serverless compute with a short-lived instance) is a fully managed way to embed questions and run statistical distribution tests against a stored baseline. It is scalable, requires no long-lived infrastructure, and gives full control over the drift statistic (e.g., MMD, cosine shift).

Distractor Analysis

  • B is Incorrect: OpenSearch anomaly detection works on numeric and categorical fields in the index, not on query embedding distributions. It cannot compute semantic drift between incoming questions and a recorded baseline.
  • C is Incorrect: CloudWatch generative AI observability provides token metrics, latency, error rates, and prompt tracing—but no built-in semantic distribution comparison or drift detection.
  • D is Incorrect: Self-hosting Prometheus on EKS is high operational overhead (cluster management, upgrades, monitoring) and violates the "no additional compute instances" constraint—SageMaker Processing is the managed, low-code alternative.

Key Takeaway: For semantic drift detection in a RAG system, run scheduled embedding comparisons in a managed SageMaker Processing job; OpenSearch anomaly detection and CloudWatch GenAI observability do not compute distributional drift.

Question 8 - Shadow Variant for Safe Model Validation

A fraud detection team has trained a new model version and wants to validate its performance on live production traffic without returning its predictions to end users. The new model must run on the same endpoint as the current model and receive a copy of every request. The results from the shadow variant must be logged for offline comparison against the production variant's predictions, and zero user-visible output may change during the validation period.

Which configuration supports this? (Select ONE)

  • A) Deploy a second SageMaker endpoint with the new model and route 100% traffic to both endpoints using Route 53 weighted routing.
  • B) Deploy the new model as a shadow production variant on the same SageMaker endpoint, assign it a weight of zero, and configure data capture to log its responses.
  • C) Deploy the new model as a second production variant on the same endpoint and assign it 10% traffic weight while the incumbent retains 90%.
  • D) Use AWS Step Functions to invoke both models in parallel for each request and return only the current model's output.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: A shadow variant (weight = 0) on a SageMaker endpoint receives a copy of all live traffic, executes inference, and records its predictions (via data capture logs) without any of its outputs being returned to users. This enables validation of the full serving stack under real traffic conditions.

Distractor Analysis

  • A is Incorrect: Route 53 weighted routing sends some users to the new model (violating the "zero user-visible output may change" constraint) and introduces a second endpoint to manage, increasing operational overhead.
  • C is Incorrect: Assigning 10% traffic weight means the new model's predictions are returned to real users—this is a canary deployment, not a shadow test.
  • D is Incorrect: Step Functions orchestration adds latency to synchronous inference requests (violating the "same endpoint" constraint) and creates a tight coupling that is harder to revert.

Key Takeaway: A shadow variant with weight = 0 is the correct pattern for testing a serving stack against live traffic without exposing users to the new model's outputs.

Question 9 - Customer Churn Prediction Baseline Drift Alerting

A telecommunications company uses a SageMaker endpoint to predict customer churn from monthly usage features. The ML team has configured SageMaker Model Monitor with data quality monitoring. The team needs to detect when a single numerical feature (e.g., monthly_data_usage_gb) has shifted significantly compared to the baseline (e.g., after a new data plan promotion). The monitoring schedule runs hourly.

Which TWO configurations satisfy the requirement? (Select TWO)

  • A) Define a constraint on the monthly_data_usage_gb column using a Kolmogorov-Smirnov test with a comparison_threshold and quantile defining the acceptable drift margin.
  • B) Configure a CloudWatch alarm on the Model Monitor drift metric with a threshold that triggers a notification when the combined drift score exceeds 0.45.
  • C) Enable SageMaker Clarify's pre-training bias report to detect if the feature distribution has shifted from the training dataset.
  • D) Set up an AWS Glue Data Quality rule set that validates the column exists but does not perform drift comparisons.
  • E) Use a custom NLP model on the raw feature values to generate a sentiment score, and alert when this score drops below a threshold.
Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: SageMaker Model Monitor allows column-level constraints using the KS test to detect univariate feature drift between the baseline and current data. The comparison_threshold (e.g., 0.1) and quantile (e.g., 0.05) define statistical significance.
  • B is Correct: Model Monitor publishes a combined drift metric to CloudWatch; alarming when it exceeds a pre-agreed threshold (e.g., 0.45) ensures the team is notified when drift detection fires.

Distractor Analysis

  • C is Incorrect: SageMaker Clarify's pre-training bias report detects demographic bias in training data, not real-time drift between production traffic and a recorded baseline.
  • D is Incorrect: Glue Data Quality rule sets validate data quality (e.g., not-null, length checks, column counts), not statistical distribution comparison against a baseline.
  • E is Incorrect: Applying an NLP sentiment model to numerical usage features is non-sensical and does not fulfill the requirement of distributional drift detection.

Key Takeaway: For univariate numerical drift detection, use KS-test constraints in Model Monitor and pair it with a CloudWatch alarm on the drift metric.

Question 10 - Lambda-Based Drift Detection for Serverless Bedrock Applications

A productivity SaaS startup runs a serverless generative AI application that calls Amazon Bedrock directly from AWS Lambda. The startup has no EC2 or SageMaker endpoints. The team needs to detect data drift in the text prompts sent to Bedrock with minimal overhead and cost. They want to use AWS-native features that scale to zero when not in use.

Which THREE options are achievable in a serverless architecture for drift detection? (Select THREE)

  • A) Store prompt embeddings in Amazon S3 and run a scheduled AWS Glue Python Shell job that computes a drift score (e.g., cosine distance from baseline centroids) and writes to CloudWatch.
  • B) Configure SageMaker Model Monitor on the Lambda function's execution role to automatically detect data drift in Bedrock invocations.
  • C) Use an Amazon EventBridge rule to trigger a Lambda function hourly that samples recent prompt logs from CloudWatch Logs, embeds them with Bedrock, and computes a drift statistic.
  • D) Use AWS Step Functions with a Serverless Map state to orchestrate embedding generation and drift score computation on a scheduled basis without managing compute instances.
  • E) Deploy a self-hosted monitoring stack using Prometheus and Grafana on EC2 reserved instances to track Bedrock invocation rates.
Click to reveal Answer & Explanation

Correct Answers: A, C, and D

Correct Options Analysis

  • A is Correct: A scheduled Glue Python Shell job is a serverless ETL capability that can read stored embeddings, compute drift statistics (e.g., distance to baseline centroids), and publish metrics to CloudWatch—all without a persistent server.
  • C is Correct: EventBridge + Lambda + Bedrock embeddings is a fully serverless, event-driven drift detection pattern that only runs for seconds and scales to zero.
  • D is Correct: Step Functions supports serverless orchestration with a Map state to iterate over prompt batches, invoke embedding logic, and aggregate drift scores—ideal for maintaining a scheduled drift-check workflow without managing instances.

Distractor Analysis

  • B is Incorrect: SageMaker Model Monitor monitors SageMaker endpoints (invocations, data capture), not Lambda-executed Bedrock calls. It has no mechanism to hook into Bedrock invocations outside of a SageMaker endpoint context.
  • E is Incorrect: Prometheus/Grafana on EC2 introduces significant operational overhead (patching, scaling, storage management) and incurs continuous costs even when idle, violating the "scale to zero" and "minimal overhead" constraints.

Key Takeaway: Serverless drift detection patterns leverage EventBridge, Lambda, Glue Python Shell, and Step Functions; SageMaker Model Monitor is endpoint-specific, and self-hosted stacks violate the cost/overhead constraint.