Skip to content

Skill 2.3.2: Create model performance baselines and implement drift detection

Question 1 - Establishing Model Performance Baselines with SageMaker Model Monitor

A financial services company has deployed a fraud detection model on Amazon SageMaker to score real-time credit card transactions. The compliance team requires the ML engineering team to track model quality over time and detect when prediction accuracy degrades.

The team wants to implement a solution with minimal operational overhead that automatically captures ground truth labels from a post-transaction database, compares them against model predictions, and generates alerts when model accuracy drops below a configured threshold.

Which ONE approach should the ML engineer implement?

  • A) Deploy the model to a SageMaker endpoint and enable SageMaker Model Monitor with a model quality monitoring schedule. Configure the monitoring job to merge inference inputs with ground truth labels from Amazon S3, compute accuracy metrics against a stored baseline, and publish violations to Amazon CloudWatch alarms.

  • B) Create an AWS Lambda function that runs every hour, queries the post-transaction database, manually computes precision and recall using scikit-learn, and writes the results to a custom DynamoDB table. Configure a CloudWatch alarm on the DynamoDB metric.

  • C) Enable SageMaker Model Monitor data quality monitoring to track drift in the input feature distribution. Set up a CloudWatch alarm that triggers when the Kullback-Leibler divergence exceeds a static threshold.

  • D) Use Amazon Bedrock evaluations with a human-based evaluation job to have fraud analysts manually review sampled predictions weekly and record their judgments in a CSV file uploaded to S3.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: SageMaker Model Monitor's model quality monitoring job is purpose-built for this use case. It automatically merges captured inference data with ground truth labels (provided via S3), computes standard classification metrics (accuracy, precision, recall, F1, AUC), compares them against a statically computed baseline (from a baseline job run on a known-good dataset), and emits CloudWatch metrics that can trigger alarms when violations occur. This meets the "minimal operational overhead" constraint because Model Monitor is a fully managed service.

Distractor Analysis

  • B is Incorrect: While functionally possible, building a custom Lambda function with manual metric computation introduces significant operational overhead (code maintenance, error handling, scheduling, state management) compared to Model Monitor's native capabilities. The question explicitly requires minimal operational overhead.
  • C is Incorrect: Data quality monitoring detects drift in the input feature distribution (e.g., new categories, missing values, statistical shifts), not degradation in prediction accuracy. The requirement is specifically about tracking model quality against ground truth labels.
  • D is Incorrect: Amazon Bedrock evaluations is designed for evaluating generative AI models (LLMs), not traditional ML classification models like fraud detection. Additionally, human-based evaluation jobs introduce manual overhead and are not automated. The scenario is about a traditional supervised learning model, not generative AI.

Key Takeaway: For traditional ML models deployed on SageMaker endpoints, Model Monitor's model quality monitoring is the managed service to automatically detect prediction quality degradation using ground truth labels and baseline statistics.

Question 2 - Drift Detection Baseline Selection for RAG System

A healthcare technology company has deployed a Retrieval-Augmented Generation (RAG) system using Amazon Bedrock Knowledge Bases and Amazon OpenSearch Serverless to answer clinical questions from indexed medical documents.

The ML team needs to implement drift detection for the retrieval component. The company's compliance policy requires that all evaluation data must be de-identified to protect patient privacy, and the team must avoid storing any customer data for evaluation purposes beyond what's already in the vector index.

The team decides to use Amazon Bedrock evaluations to create a baseline for retrieval accuracy. Which TWO configuration choices should the team make to satisfy the compliance and operational requirements? (Select TWO)

  • A) Package a de-identified prompt dataset into an Amazon S3 bucket, configure the evaluation job with the custom dataset, and select a judge model to evaluate retrieval results against expected contexts.

  • B) In the Amazon Bedrock evaluations job, choose the retrieval source evaluation type and point to the existing Bedrock Knowledge Base.

  • C) Package the original (non-de-identified) dataset from the production database directly into the evaluation job, because Bedrock evaluations automatically handles PHI redaction for evaluation datasets.

  • D) Configure the evaluation job in programmatic mode using a custom-built retrieval accuracy metric (e.g., Context Precision), so no human review or additional data storage is needed.

  • E) Enable Amazon Macie on the evaluation results S3 bucket to scan for any PHI and automatically redact/encrypt findings.

Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: Using a de-identified prompt dataset stored in S3 ensures compliance with the privacy policy. The judge model in Bedrock evaluations can score the retrieval results from the knowledge base against expected contexts to compute retrieval accuracy metrics like Context Precision and Context Recall.
  • B is Correct: Amazon Bedrock evaluations supports a dedicated evaluation type for retrieval sources (Bedrock Knowledge Bases). This is the correct configuration to measure retrieval accuracy separately from generation quality, as the requirement focuses on the retrieval component's drift detection.

Distractor Analysis

  • C is Incorrect: Bedrock evaluations does not automatically perform PHI redaction on custom evaluation datasets. The team must ensure the dataset is de-identified before uploading it to S3. Uploading raw patient data violates the compliance constraint directly.
  • D is Incorrect: The team needs a baseline for retrieval accuracy, which typically requires a judge model or reference answers to evaluate signal quality. A purely programmatic evaluation with a custom metric would not evaluate the retrieval component's actual accuracy against expected contexts unless paired with reference contexts and semantic scoring methods. The question specifies leveraging Bedrock evaluations, not building a custom evaluation pipeline.
  • E is Incorrect: While Macie is a valid service for detecting sensitive data (PHI/PII) in S3 buckets, it does not provide automatic redaction of data or encryption for evaluation purposes. Moreover, the requirement is to avoid storing additional customer data — using Macie to scan already-stored data doesn't solve the root issue of compliance.

Key Takeaway: For RAG systems, evaluate retrieval quality separately from generation quality using Amazon Bedrock evaluations' retrieval source type, and ensure datasets are de-identified before uploading to S3 for compliance.

Question 3 - Data Drift Detection for Feature Store Pipeline

A retail company uses Amazon SageMaker Feature Store to store customer features for a real-time product recommendation model. The company has experienced a drop in model performance and suspects that the distribution of features in production has shifted from the training data.

The ML team needs to set up continuous drift detection on the feature values, with the constraint that the solution must use managed services with minimal code, and alerts must be routed to the existing on-call rotation via Amazon SNS.

Which ONE solution should the team implement?

  • A) Configure SageMaker Model Monitor data quality monitoring on a scheduled basis, pointing to the baseline statistics job. Select the features to monitor across the test dataset or newer datasets stored in S3. Route violations to Amazon CloudWatch alarms that publish to the SNS topic.

  • B) Write a custom Python job using the scipy.stats.ks_2samp test, schedule it with Amazon EventBridge, and send SNS notifications directly from the Python code. Deploy the job as a container on Amazon ECS.

  • C) Use AWS Glue Data Quality rules to compare feature distributions between the training dataset and production feature group, and configure Glue to send job failure notifications to the SNS topic.

  • D) Enable Amazon SageMaker Clarify to perform bias detection on the inference results, and configure its built-in drift detection capability to monitor feature shifts and alert via CloudWatch alarm to SNS.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: SageMaker Model Monitor's data quality monitoring is a fully managed service specifically designed for continuous drift detection. It computes statistical metrics (mean, standard deviation, KL divergence, etc.) for features and compares them against a stored baseline created from the training dataset. Scheduled monitoring jobs can pull data from S3 or directly from SageMaker endpoints, making it suitable for monitoring production feature distributions with minimal custom code. Violations automatically produce CloudWatch metrics, which can trigger alarms routed to SNS.

Distractor Analysis

  • B is Incorrect: This solution introduces significant operational overhead — building a custom container, writing statistical test code, managing ECS infrastructure, and handling error cases manually. The constraint explicitly requires managed services with minimal code, making this the wrong choice despite being statistically valid.
  • C is Incorrect: AWS Glue Data Quality is primarily designed for data quality rules (null check, uniqueness, etc.) on data catalog tables, not for continuous statistical drift detection between training data and production feature distributions. It doesn't provide the same baseline-vs-current statistical comparison features as Model Monitor.
  • D is Incorrect: Amazon SageMaker Clarify is designed for bias detection and explainability of ML predictions (pre-training and post-training bias metrics, SHAP feature attribution, partial dependence plots), not for statistical data drift detection over time. Clarify does not provide drift detection capability.

Key Takeaway: SageMaker Model Monitor's data quality monitoring is the managed drift detection solution for features, comparing live data statistics against a training baseline and integrating with CloudWatch alarms for alerting.

Question 4 - Baseline Comparison for Prompt Engineering

A customer support team uses Amazon Bedrock with a prompt template to generate responses to customer queries. The team wants to establish a performance baseline for the current prompt and compare a new prompt variant against it, using a systematic evaluation process.

The constraint is that the evaluation must be automated without involving human workers, and the team needs to use metrics that can be computed programmatically to compare the two prompts.

Which TWO steps should the team take to create the baseline and compare prompt variants? (Select TWO)

  • A) Use Amazon Bedrock Prompt Management to version the current prompt as the baseline and create a variant with the updated wording. Run a programmatic evaluation job comparing both prompts against a shared evaluation dataset.

  • B) Configure the evaluation job to use ROUGE or BLEU scores on the generated responses, comparing against reference answers in the evaluation dataset.

  • C) Use Amazon SageMaker Clarify to generate explainability reports for both prompts and visually compare the feature attributions to determine which prompt is better.

  • D) Run a human-based evaluation job in Bedrock, inviting the customer support team (as internal reviewers) to rate responses from both prompts on a 1-5 scale.

  • E) Deploy both prompts to production and use Amazon CloudWatch Synthetics canaries to measure real-time latency and decide which prompt is better.

Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: Amazon Bedrock Prompt Management supports versioning prompt templates and comparing prompt variants. This allows the team to systematically track the baseline and the new variant, ensuring the evaluation is reproducible and auditable.
  • B is Correct: ROUGE and BLEU are programmatic, automated NLP evaluation metrics that compare generated text against reference text. They are appropriate for this scenario because they don't require human intervention and can be computed automatically. The evaluation job in Bedrock supports these programmatic metrics.

Distractor Analysis

  • C is Incorrect: SageMaker Clarify is designed for explainability of traditional ML models and bias detection — not for evaluating the quality of text generated by LLMs. Feature attributions are irrelevant when comparing two different prompt templates on a Bedrock LLM.
  • D is Incorrect: The question explicitly states that the evaluation must be automated without human workers. Human-based evaluation jobs in Bedrock require raters (employees or reviewers) to manually judge responses, violating the automation constraint.
  • E is Incorrect: CloudWatch Synthetics canaries test API availability and latency, not the quality or correctness of generated text. Latency alone is not a valid metric for determining which prompt produces better responses.

Key Takeaway: For automated prompt comparison, use Prompt Management for versioning and programmatic evaluation metrics (ROUGE, BLEU, BERTScore) in Bedrock evaluations to create measurable, reproducible baselines.

Question 5 - Shadow Variant Testing for Model Comparison

An e-commerce company has a product search ranking model in production on Amazon SageMaker. The ML team has trained a new candidate model and wants to compare its performance against the current production model under real traffic conditions, while ensuring end users are not impacted.

The candidate model uses a different framework version and the team must also monitor operational metrics such as latency and error rates during the comparison. The company wants to avoid building custom infrastructure for this comparison.

Which ONE approach should the team use?

  • A) Deploy the candidate model as a shadow variant on the same SageMaker endpoint. Configure the production variant to serve 100% of traffic while routing a copy of live requests to the shadow variant. Monitor CloudWatch metrics for both variants and compare logs after the testing period.

  • B) Deploy the candidate model to a separate SageMaker endpoint with a load balancer in front, routing 50% of traffic to each. Use Amazon CloudWatch to compare latency and error rates between the two endpoints.

  • C) Use Amazon Bedrock evaluations with a judge model to compare the production and candidate model responses on a representative dataset, and evaluate the quality scores without routing live traffic.

  • D) Use AWS Step Functions to build a custom traffic routing workflow that duplicates incoming API Gateway requests, sends one copy to production and one to the candidate model, and stores the responses in DynamoDB for offline comparison.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: SageMaker AI shadow testing (shadow variants) is the native, managed solution for this requirement. It deploys the candidate model alongside the production variant on the same endpoint, asynchronously routes copies of live requests to the shadow variant, and only returns production responses to the caller. CloudWatch metrics are automatically collected for both variants, enabling latency and error rate comparison. This matches the "avoid building custom infrastructure" constraint.

Distractor Analysis

  • B is Incorrect: Routing 50% of traffic to the candidate model means real users receive candidate responses, violating the constraint that end users are not impacted. This is an A/B testing approach, not a shadow comparison.
  • C is Incorrect: Amazon Bedrock evaluations is designed for evaluating generative AI models (LLMs) against a dataset, not for comparing traditional ML models under live traffic conditions. The scenario describes a ranking model, not a GenAI system. Additionally, this approach doesn't evaluate operational metrics like latency under real traffic.
  • D is Incorrect: Building a custom traffic duplication workflow with Step Functions and DynamoDB introduces unnecessary operational overhead. SageMaker shadow tests provide this capability natively without custom infrastructure.

Key Takeaway: Shadow testing on SageMaker AI is the managed approach to compare candidate models against production using live requests without impacting end users, while monitoring both operational and prediction metrics.

Question 6 - Ground Truth Labeling Strategy for Model Quality Monitoring

An insurance company has deployed a claims classification model on Amazon SageMaker. The company needs to set up model quality monitoring with ground truth labels.

The challenge is that ground truth labels are only available 48 hours after predictions are made, after claims have been reviewed by human underwriters. The company wants to minimize storage costs while ensuring the monitoring job can merge predictions with delayed ground truth.

Which ONE strategy should the ML engineer implement?

  • A) Enable data capture on the SageMaker endpoint to save inference inputs and predictions to Amazon S3. Configure a SageMaker Model Monitor model quality schedule with a delayed ground truth ingestion pattern, where the monitoring job merges the S3 inference data with the ground truth labels (also in S3) using an identifier.

  • B) Modify the inference code to synchronously call the underwriter review service and wait for the 48-hour ground truth before returning predictions to the caller. This ensures predictions and ground truth are always paired in real time.

  • C) Store predictions in Amazon DynamoDB with a TTL of 48 hours, and upon label availability, use a DynamoDB stream to trigger a Lambda function to compute the metric. Delete the inference data after computation.

  • D) Use Amazon SageMaker Ground Truth to create a human labeling job that reviews every prediction in real-time, ensuring ground truth is available instantly for the next monitoring cycle.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: SageMaker Model Monitor's model quality monitoring natively supports delayed ground truth ingestion. The endpoint's data capture feature automatically stores inference requests and predictions in S3 at near-zero additional cost (using standard S3 storage classes). The monitoring job, scheduled after the 48-hour labeling window, merges the stored inference data with ground truth labels in S3 using a record identifier. This minimizes storage costs (S3 standard class) and requires no custom merging code.

Distractor Analysis

  • B is Incorrect: Synchronously waiting for human review before returning predictions would introduce catastrophic latency to real-time predictions (48+ hours), completely defeating the purpose of a real-time inference endpoint. This is architecturally invalid.
  • C is Incorrect: Building a custom DynamoDB + TTL + Lambda pipeline introduces unnecessary operational overhead. DynamoDB storage is also more expensive than S3 for this use case, and the TTL-based deletion adds complexity without benefit. Model Monitor's S3-based capture is simpler and more cost-effective.
  • D is Incorrect: Amazon SageMaker Ground Truth is designed for creating training datasets and labeling tasks, not for continuous monitoring of production predictions. Running a human labeling job for every prediction is impractical, extremely expensive, and introduces 48-hour-latency ground truth back into the same problem.

Key Takeaway: SageMaker Model Monitor's model quality monitoring supports delayed ground truth ingestion: capture inferences to S3, ingest labels later, and schedule the monitoring job to merge them by ID.

Question 7 - RAG System Monitoring with Drift Detection

A legal technology company uses Amazon Bedrock Knowledge Bases for a RAG-based contract analysis system. The legal department requires that the system must flag when the retrieved documents contain irrelevant content or miss key clauses, and when the generated answers contain unsupported claims.

The ML team needs to implement a monitoring system using AWS managed services. The company also wants a low-latency real-time check to detect whether the retrieved contexts are relevant to the query before generating an answer.

Which TWO steps should the ML team implement? (Select TWO)

  • A) Set up a SageMaker Model Monitor endpoint to monitor the retrieval layer by treating the vector database's relevance scores as model metrics.

  • B) Use Amazon Bedrock evaluations with a scheduled job to periodically evaluate retrieval accuracy (Context Precision/Recall) and generation quality (Faithfulness, Answer Relevance, Context Relevance) against a curated evaluation dataset.

  • C) In the RAG orchestration code, use Amazon Bedrock Knowledge Bases and the RetrieveAndGenerate API to implement a real-time relevance gating mechanism that rejects or flags responses when retrieval confidence is low.

  • D) Deploy a shadow variant of the RAG pipeline to compare retrieval quality against production under live traffic, logging all retrieved contexts for offline analysis.

  • E) Use Amazon Comprehend to analyze generated responses for sentiment and detect if any answer expresses a negative tone, flagging those as potentially hallucinated.

Click to reveal Answer & Explanation

Correct Answers: B and C

Correct Options Analysis

  • B is Correct: Amazon Bedrock evaluations supports scheduled evaluation jobs that measure both retrieval accuracy and generation quality separately. This is exactly the RAG triad evaluation framework (Context Precision/Recall, Faithfulness, Answer Relevance) recommended for monitoring RAG systems. Periodic evaluation against a curated dataset establishes a baseline and detects drift.
  • C is Correct: Implementing real-time relevance gating in the orchestration layer using the retrieved contexts' relevance scores from Bedrock Knowledge Bases is a valid pattern to flag low-confidence retrievals before generation, providing low-latency real-time checks without waiting for batch evaluation.

Distractor Analysis

  • A is Incorrect: SageMaker Model Monitor is designed for monitoring traditional ML models on SageMaker endpoints, not the retrieval layer of Bedrock Knowledge Bases. Treating vector database relevance scores as "model metrics" in Model Monitor requires the retrieval system to be a SageMaker endpoint, which is not the case when using Bedrock Knowledge Bases.
  • D is Incorrect: Shadow variants are a SageMaker AI endpoint testing feature, not applicable to Bedrock Knowledge Bases RAG systems. Additionally, shadow testing compares two model versions for deployment validation, not for continuous monitoring of retrieval quality in production.
  • E is Incorrect: Amazon Comprehend's sentiment analysis detects emotional tone, not factual accuracy or hallucination. A negative sentiment in a generated answer does not indicate the answer is unsupported or factually incorrect. This is a fundamental mismatch between the metric and the requirement.

Key Takeaway: For RAG monitoring, split retrieval from generation: use Bedrock evaluations to measure both separately, and implement real-time relevance gating in the orchestration code for low-latency flagging.

Question 8 - Training Convergence Debugging

A machine learning engineer is training a deep learning model on Amazon SageMaker for image classification. During training, the engineer observes that the validation loss is improving while the training loss is also improving, but after 20 epochs, the validation loss starts increasing while training loss continues to decrease.

The engineer needs to identify the root cause and fix it. Which TWO actions should the engineer take? (Select TWO)

  • A) Enable early stopping with a patience parameter, monitoring validation loss to terminate training when it stops improving.

  • B) Increase the learning rate significantly to help the model escape the region where training and validation diverge.

  • C) Add the SageMaker debugger to track gradient norms and identify vanishing gradients as the cause of the divergence between training and validation loss.

  • D) Add dropout regularization or L2 weight decay to reduce overfitting, and consider data augmentation to increase training data diversity.

  • E) Switch from stochastic gradient descent (SGD) to a full-batch gradient descent to ensure the model converges to the global minimum.

Click to reveal Answer & Explanation

Correct Answers: A and D

Correct Options Analysis

  • A is Correct: The described pattern (validation loss increasing while training loss continues to decrease) is classic overfitting. Early stopping with a patience parameter monitoring validation loss is an effective technique to halt training before the model memorizes training data and generalizes poorly. This is a standard best practice.
  • D is Correct: Dropout regularization and L2 weight decay explicitly reduce overfitting by preventing the model from relying too heavily on specific neurons or large weights. Data augmentation increases the effective training data size and diversity, further improving generalization.

Distractor Analysis

  • B is Incorrect: Increasing the learning rate would exacerbate the problem. An excessively high learning rate typically causes loss to oscillate or diverge, not fix overfitting. The symptom described here is not a learning rate issue.
  • C is Incorrect: SageMaker Debugger is useful for monitoring training, but vanishing gradients typically cause the loss to plateau (not the validation loss to increase while training loss decreases). The pattern described is overfitting, not vanishing gradients.
  • E is Incorrect: Full-batch gradient descent reduces stochastic noise but does not address overfitting. In fact, full-batch gradient descent can be more prone to overfitting in some cases because it makes deterministic updates that may converge to sharp minima with poor generalization. It's also impractical for large datasets.

Key Takeaway: When training loss improves but validation loss worsens, diagnose it as overfitting and apply regularization (dropout, weight decay, augmentation) or early stopping.

Question 9 - Establishing a Baseline for Traditional ML Model

A manufacturing company has trained a predictive maintenance model using Amazon SageMaker to predict equipment failures from sensor data. The model was trained on historical data from 2023. Before deploying the model to production, the ML team needs to establish a performance baseline to enable future drift detection.

The company has the following constraints:

  • The baseline must be automatically computed and stored for future comparison.
  • The evaluation must be reproducible.
  • The company wants to track multiple model iterations over time with their respective metrics and parameters.

Which TWO actions should the team take? (Select TWO)

  • A) Use Amazon SageMaker AI with MLflow to track the training run, log hyperparameters, metrics (accuracy, precision, recall, F1), and model artifacts, and tag the run as the production baseline.

  • B) Run a SageMaker Model Monitor model quality baseline job on a validation dataset with ground truth labels to compute and store the baseline statistics (mean, standard deviation, and metric thresholds).

  • C) Store the baseline metrics in a text file in the project's Git repository and manually update it after each training run.

  • D) Use AWS Config rules to automatically track model performance metrics over time and generate compliance reports when metrics change.

  • E) Deploy the model to production immediately and use the first week of production data (without ground truth) as the baseline, assuming the model performs correctly.

Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: MLflow in Amazon SageMaker AI provides automatic experiment tracking — logging hyperparameters, metrics, artifacts, and data lineage for every training run. Tagging a run as "baseline" makes it identifiable and reproducible for future comparisons. This satisfies the "reproducible" and "track multiple iterations" requirements.
  • B is Correct: SageMaker Model Monitor's model quality baseline job computes and stores statistical baselines (mean metrics, standard deviations, thresholds) from a labeled validation dataset. This baseline is used by scheduled monitoring jobs for drift detection, satisfying the "automatically computed and stored" requirement.

Distractor Analysis

  • C is Incorrect: Storing baseline metrics manually in a text file is not automatically computed and is prone to human error. It lacks the structured metadata (parameters, data lineage, artifacts) that experiment trackers like MLflow provide. This approach doesn't scale for multiple model iterations.
  • D is Incorrect: AWS Config is designed for infrastructure resource compliance (e.g., tracking configuration changes of S3 buckets, IAM policies, EC2 instances), not for monitoring ML model performance metrics. It cannot track model metrics like accuracy or precision.
  • E is Incorrect: Using unlabeled production data as the baseline violates the requirement for ground truth labels and properly computed model quality baselines. Without ground truth, it's impossible to compute accuracy metrics. This approach assumes the model is correct, which defeats the purpose of drift detection.

Key Takeaway: Establish baselines using MLflow for experiment tracking (reproducible runs with metrics/params) and Model Monitor baseline jobs (statistical thresholds for drift detection).

Question 10 - LLM-as-a-Judge for Content Quality Baseline

A customer service company uses Amazon Bedrock with a large language model (LLM) to generate responses to customer inquiries. The company's quality assurance team wants to create a baseline for response quality and comparability.

The company needs an automated solution that:

  • Evaluates each generated response for quality dimensions (helpfulness, correctness, harmlessness).
  • Provides an explanation for each quality score.
  • Does not require human raters.

Which ONE configuration should the ML engineer implement?

  • A) Use Amazon Bedrock evaluations with a judge model (LLM-as-a-judge). Configure the judge to score responses on scales for helpfulness, correctness, and harmlessness, and instruct it to provide a reasoning statement for each score. Run the job against a curated evaluation dataset.

  • B) Use Amazon SageMaker Clarify to generate SHAP explanations for the LLM's predictions, and compute a single accuracy metric by comparing model outputs to ground truth labels using BLEU score.

  • C) Use Amazon Detective to analyze LLM response patterns and identify anomalies in the generated text, flagging responses that deviate from expected behavior.

  • D) Use Amazon Comprehend to perform sentiment analysis on the generated responses and use the sentiment score as a proxy for response quality, assuming positive sentiment correlates with good quality.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Amazon Bedrock evaluations supports LLM-as-a-judge evaluation mode. A judge model (e.g., Anthropic Claude or another LLM) is instructed via a judge prompt to evaluate generated responses against quality criteria (helpfulness, correctness, harmlessness) on defined rating scales. The judge model also generates natural language explanations for each score, providing interpretability. This is fully automated with no human raters required. The evaluation job runs against a dataset stored in S3 or built-in prompt datasets.

Distractor Analysis

  • B is Incorrect: SageMaker Clarify is designed for explainability of traditional ML models (SHAP values, feature attribution, partial dependencies) and bias detection — not for evaluating generated text quality. BLEU scores measure n-gram overlap with reference texts, which is not appropriate for open-ended customer service responses where many valid wordings exist. Clarify also doesn't generate quality explanations for LLM responses.
  • C is Incorrect: Amazon Detective is a security investigation service that analyzes AWS CloudTrail, VPC Flow Logs, and GuardDuty findings to identify security issues — completely irrelevant to LLM response quality evaluation.
  • D is Incorrect: Sentiment analysis only measures emotional tone (positive vs. negative), which is a poor proxy for response quality metrics like helpfulness, correctness, or harmlessness. A response can be positively worded but factually incorrect, or negatively worded but accurate. This is a fundamental metric mismatch.

Key Takeaway: For automated quality evaluation of LLM outputs with explanations, use Bedrock evaluations with a judge model (LLM-as-a-judge), configured with quality dimension rubrics and reasoning instructions.

Question 11 - Model Baseline with Human Evaluation Integration

A healthcare AI company has developed a medical summarization system using Amazon Bedrock and Amazon Bedrock Knowledge Bases (RAG). Before deploying to production, the company must establish a performance baseline that incorporates both automated metrics and human clinical reviewer feedback.

The constraint is that the evaluation must follow an integrated workflow where automated checks run first, and only responses that pass automated thresholds are sent to human reviewers for subjective quality assessment.

Which TWO services or approaches should the ML engineer combine? (Select TWO)

  • A) Run a programmatic evaluation job in Amazon Bedrock using ROUGE/BERTScore to compute baseline semantic similarity scores against reference summaries.

  • B) Use Amazon SageMaker Ground Truth to create a human review workflow where clinicians label the generated summaries with quality ratings.

  • C) Use AWS Step Functions to orchestrate the workflow: the first step runs the Bedrock programmatic evaluation; conditionally, only summaries that fail the automated quality threshold are routed to a SageMaker Ground Truth human review job for manual assessment.

  • D) Run a Bedrock Human-based evaluation job where internal clinical reviewers rate all generated summaries on a Likert scale for coherence, completeness, and clinical accuracy.

  • E) Store human ratings in a DynamoDB table and use Step Functions to build a custom evaluation workflow that combines programmatic scores and human ratings.

Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: Programmatic evaluation jobs in Bedrock can compute automated metrics (ROUGE, BLEU, BERTScore) against reference summaries. This provides a scalable, objective baseline for semantic similarity and overlap, requiring no human intervention.
  • C is Correct: Step Functions can orchestrate the hybrid evaluation pipeline: automated programmatic evaluation runs first, and conditional routing sends only flagged responses (those failing quality thresholds) to human review via SageMaker Ground Truth or a custom Bedrock human evaluation job. This creates an integrated human-in-the-loop evaluation framework where humans review only edge cases, reducing cost and time.

Distractor Analysis

  • B is Incorrect: Using SageMaker Ground Truth alone for human labeling creates a review workflow but doesn't integrate with automated evaluation. This option only covers the human review portion and doesn't establish the full pipeline with automated metrics first. It also may send all responses to human review, increasing cost.
  • D is Incorrect: Running a human-based evaluation on ALL summaries is expensive and operationally inefficient compared to the hybrid approach required by the scenario. The constraint specifies automated checks first to filter, then human review of flagged outputs.
  • E is Incorrect: Building a custom evaluation workflow with DynamoDB and Step Functions introduces unnecessary complexity compared to using Bedrock's native evaluation features. The combination of Bedrock evaluations for automated scoring + Step Functions for conditional routing to human review is the correct managed approach.

Key Takeaway: For hybrid evaluation with cost optimization, use programmatic evaluation for automated thresholds and route only flagged/low-confidence outputs to human review via Step Functions orchestration.

Question 12 - Explain Model Outputs for Drift Investigation

A financial services company uses Amazon SageMaker to deploy a credit risk scoring model. The model's performance has drifted, and the compliance team needs to understand why specific predictions were made for a set of high-risk accounts.

The company requires:

  • Model-agnostic explanations that show feature contributions for each prediction.
  • Integration with the existing SageMaker endpoint.
  • Explanations for both individual predictions and aggregate feature importance.

Which ONE service should the ML engineer use?

  • A) Amazon SageMaker Clarify with SHAP (SHapley Additive exPlanations) to generate feature attribution explanations for individual predictions and aggregate feature importance reports from the inference endpoint.

  • B) Amazon Bedrock with a judge model to explain why the credit risk model made each prediction in natural language.

  • C) Amazon SageMaker Debugger to capture tensor activations during inference and visualize which neurons contributed most to each prediction.

  • D) AWS X-Ray to trace the inference request path and identify which feature preprocessing steps influenced the final prediction score.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Amazon SageMaker Clarify is the native, model-agnostic explainability service for SageMaker. It supports SHAP and KernelSHAP for feature-level explanations of individual predictions, as well as aggregate feature importance or partial dependence plots. It can be run against SageMaker endpoints and batch transform jobs, making it the correct choice.

Distractor Analysis

  • B is Incorrect: Amazon Bedrock is for generative AI models (LLMs) and doesn't provide feature-level explainability for traditional ML models like credit risk scoring. Using a judge model to "explain" predictions of a traditional model is not a supported pattern.
  • C is Incorrect: SageMaker Debugger captures training time tensor data (weights, gradients, activations) for debugging model convergence and performance. It's not designed for inference-time feature attribution or model-agnostic explainability.
  • D is Incorrect: AWS X-Ray traces distributed application requests and helps identify latency bottlenecks or errors in service calls, not the internal decision-making process of a machine learning model's feature contributions.

Key Takeaway: Use SageMaker Clarify for model-agnostic feature explanations (SHAP) and bias detection — the correct service for understanding why a traditional ML model makes specific predictions.

Question 13 - Monitoring Model Quality Over Time with Ground Truth

A logistics company has deployed a route optimization model on Amazon SageMaker. The model predicts delivery time estimates. Ground truth labels (actual delivery times) are available in an Amazon Redshift data warehouse, but they're populated 24 hours after deliveries complete.

The company needs to monitor model quality daily with minimal implementation effort. The monitoring must alert the operations team when Mean Absolute Error (MAE) exceeds a threshold of 30 minutes.

Which ONE approach should the ML engineer implement?

  • A) Configure a SageMaker Model Monitor model quality monitoring schedule that runs daily. The monitoring job pulls inference data from S3 (captured by endpoint data capture) and ground truth labels from Redshift via a scheduled data export to S3. Set the MAE threshold to 30 minutes in the job configuration, with a CloudWatch alarm routed to the operations team.

  • B) Create an AWS Glue ETL job that runs daily, extracts ground truth from Redshift, joins it with inference data, computes MAE using Spark, and sends a custom CloudWatch metric. Configure an alarm on the metric threshold of 30 minutes.

  • C) Use Amazon QuickSight to create a dashboard from Redshift data and visually monitor the MAE trend. Set up a scheduled email report to be sent daily to the operations team.

  • D) Use Amazon SageMaker Canvas to retrain the model daily with the new ground truth labels, automatically updating the model even if MAE degrades.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: SageMaker Model Monitor's model quality monitoring natively supports scheduled evaluation jobs, merging inference data from S3 with ground truth labels, and computing regression metrics like MAE. The monitoring job automatically emits CloudWatch metrics, and alarms with thresholds (30 minutes) can trigger notifications to the operations team. This requires minimal implementation effort — the service handles the metric computation and comparison against the stored baseline.

Distractor Analysis

  • B is Incorrect: Building a custom Glue ETL job to compute MAE manually requires substantial development effort (writing Spark code, managing IAM, handling failures) compared to Model Monitor's native capabilities. This violates the "minimal implementation effort" constraint.
  • C is Incorrect: QuickSight dashboards are useful for visualization, but they don't provide automated threshold-based alerting. The operations team would need to manually review the dashboard daily, which is not scalable and doesn't meet the "alert" requirement.
  • D is Incorrect: SageMaker Canvas is a no-code ML tool for building models by business analysts, not for monitoring deployed model quality. Retraining the model daily with new ground truth is a potentially valid strategy, but it doesn't provide alerting when MAE exceeds the threshold, and it's not a monitoring solution.

Key Takeaway: SageMaker Model Monitor model quality monitoring is the managed, minimal-effort solution for automated daily model quality tracking with metric thresholds and CloudWatch alarm integration.

Question 14 - Real-Time Drift Detection for GenAI Outputs

A content generation platform uses Amazon Bedrock to power an AI writing assistant. The company has noticed that user feedback scores for response usefulness have declined over the past month. They need to implement real-time drift detection on the quality of generated responses.

The constraint is that the solution must:

  • Have low latency (not batch-based).
  • Work inline with the inference request.
  • Use a second LLM to assess quality.

Which ONE approach should the company implement?

  • A) In the application code, use Amazon Bedrock with a judge model (a second LLM) to score each generated response for usefulness and hallucination risk in real time. If the score drops below a threshold, route the response to a fallback flow or flag it for review. Record the quality scores in CloudWatch custom metrics for drift tracking.

  • B) Schedule a daily evaluation job using Amazon Bedrock evaluations with a judge model to compute usefulness scores on a random sample of responses. If the average score drops, investigate the cause.

  • C) Enable SageMaker Model Monitor on the Bedrock endpoint to monitor the output token distribution and detect drift in real time using the KL divergence metric.

  • D) Use Amazon CloudWatch Synthetics to send canary requests to the Bedrock endpoint every hour with predefined prompts, and manually compare the outputs weekly for quality drift.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: This approach implements real-time LLM-as-a-judge quality gating: a second LLM scores each response for usefulness and hallucination risk at inference time. The inline evaluation adds some latency (acceptable for a writing assistant) and enables immediate flagging or fallback routing. Quality scores can be exported to CloudWatch custom metrics for continuous drift visualization and alerting. This satisfies the low-latency, inline, and second-LLM-judge requirements.

Distractor Analysis

  • B is Incorrect: A daily batch evaluation job introduces detection latency of up to 24 hours, violating the real-time drift detection constraint. While Bedrock evaluations with a judge model is a valid drift detection method, it's not low-latency.
  • C is Incorrect: SageMaker Model Monitor is designed for traditional ML models on SageMaker endpoints, not for Bedrock LLM outputs. Bedrock endpoints don't expose the same data capture and monitoring interface as SageMaker endpoints. Monitoring token distributions alone also doesn't directly measure response quality or usefulness.
  • D is Incorrect: CloudWatch Synthetics with canaries provides availability and latency monitoring, not quality drift detection. Running canary checks every hour still introduces detection latency, and manual comparison of outputs weekly is not scalable.

Key Takeaway: For real-time GenAI quality drift detection, use an inline LLM-as-a-judge pattern that scores each response at inference time and emits custom CloudWatch metrics for monitoring trends.

Question 15 - Experiment Tracking for Reproducible Baselines

A data science team at a marketing analytics company trains multiple customer churn prediction models using different algorithms (XGBoost, LightGBM, logistic regression) and hyperparameter configurations.

The team lead notices that a previous model baseline's performance metrics are no longer reproducible — the team cannot determine which data version, feature engineering steps, or preprocessing pipeline was used.

The team wants to implement reproducible experiment tracking for all future model baselines. The solution must track:

  • Hyperparameters and metrics for every run.
  • Data and code version.
  • Model artifacts.
  • Be queryable and comparable across runs.

Which TWO services or features should the team implement? (Select TWO)

  • A) Use Amazon SageMaker AI with MLflow to track each training run: log hyperparameters, metrics, model artifacts, and tag each run with the dataset version and commit hash. Use the MLflow UI to compare runs.

  • B) Use Amazon SageMaker Experiments (the older version) to create a new experiment for each run, manually recording metrics in a notebook after each run completes.

  • C) Store model metrics in a CSV file in the project's S3 bucket and append a new row after each run. Use Amazon Athena to query the CSV for comparison.

  • D) Integrate Amazon SageMaker Model Registry to version and catalog trained models with their associated metadata (hyperparameters, metrics, data version). Combine this with MLflow tracking for experiment-level metadata.

  • E) Use AWS CloudTrail to automatically track all training job API calls and extract hyperparameters from the training job creation events.

Click to reveal Answer & Explanation

Correct Answers: A and D

Correct Options Analysis

  • A is Correct: MLflow in Amazon SageMaker AI is a fully managed experiment tracking solution that automatically captures hyperparameters, metrics, artifacts, and allows tagging with custom metadata (dataset version, commit hash). The MLflow UI provides built-in comparison across runs, making experiments reproducible and queryable.
  • D is Correct: The SageMaker Model Registry versions trained models with their associated metadata (metrics, hyperparameters, data lineage). Combining the Model Registry (for model deployment and versioning) with MLflow tracking (for experiment-level metadata) provides comprehensive reproducibility: every deployed model is traceable to its training run, data version, and code.

Distractor Analysis

  • B is Incorrect: Manually recording metrics in a notebook after each run is prone to human error and doesn't automatically capture hyperparameters, artifacts, or data lineage. This approach defeats the purpose of automatic reproducible tracking.
  • C is Incorrect: Appending metrics to a CSV file is a manual, unstructured approach that doesn't capture hyperparameters, artifacts, or data/code versioning. It also requires manual discipline to maintain, making it fragile and non-reproducible in practice.
  • E is Incorrect: CloudTrail only records API activity (who called what, when, with what parameters in the API request body). It doesn't provide experiment tracking, metric comparison, or artifact storage. Extracting hyperparameters from CloudTrail logs is impractical and doesn't address experiment reproducibility requirements.

Key Takeaway: For reproducible ML baselines, use MLflow tracking (experiment-level params/metrics/artifacts) combined with SageMaker Model Registry (model versioning with metadata) — both managed services in SageMaker AI.

Question 16 - Data Quality Drift Detection with Baseline Statistics

A streaming analytics company ingests sensor data from IoT devices and uses Amazon SageMaker for anomaly detection. The sensor data has categorical features (device type, firmware version) and numeric features (temperature, vibration).

The ML team wants to detect when the input data distribution starts drifting from the training data, with constraints:

  • Must detect categorical drift (new categories appearing) and numeric drift (statistical distribution shifts).
  • Must run on a scheduled basis with minimal code.
  • Must be integrated with Amazon CloudWatch alarms.

Which TWO actions should the team take? (Select TWO)

  • A) Run a SageMaker Model Monitor data quality baseline job on the training dataset to compute and store baseline statistics (mean, standard deviation, category counts).

  • B) Configure a SageMaker Model Monitor data quality monitoring schedule that runs daily, comparing production data against the stored baseline using KL divergence for numeric features and categorical metrics for new categories.

  • C) Write a custom AWS Lambda function that uses the evidently library to compute drift metrics and sends them to CloudWatch.

  • D) Use Amazon QuickSight ML Insights to automatically detect anomalies in the IoT data stream and alert via CloudWatch.

  • E) Enable AWS Config rules to detect when new IoT device types appear in the data stream, treating them as configuration changes.

Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: SageMaker Model Monitor's data quality baseline job automatically computes statistical summaries (means, standard deviations, quantiles, category counts) from the training dataset and stores them as the reference baseline for future drift detection.
  • B is Correct: The scheduled data quality monitoring job compares incoming production data against the stored baseline, computing metrics like KL divergence (numeric drift), total variation distance, and detecting new categories in categorical features. Violations produce CloudWatch metrics that can trigger alarms. This is a fully managed, minimal-code solution.

Distractor Analysis

  • C is Incorrect: Writing a custom Lambda with the evidently library introduces operational complexity (dependency management, compute constraints, code maintenance) that violates the "minimal code" constraint. Model Monitor provides the same functionality natively.
  • D is Incorrect: QuickSight ML Insights provides basic anomaly detection for dashboards but is not designed for statistical distribution drift detection against a stored training baseline, nor does it provide the same level of metric computation (KL divergence, categorical drift) as Model Monitor.
  • E is Incorrect: AWS Config tracks infrastructure resource configuration changes (e.g., EC2 instance types, S3 bucket policies), not data content changes. It cannot detect new categories appearing in streaming IoT data.

Key Takeaway: For scheduled data quality drift detection with categorical and numeric feature monitoring, use Model Monitor data quality jobs — baseline job for reference stats, monitoring job for scheduled comparisons with CloudWatch alarms.

Question 17 - Shadow Variant Testing for GenAI Model Comparison

A company runs a production chatbot on Amazon SageMaker using a fine-tuned Flan-T5 model. The ML team has fine-tuned a new Mistral model and wants to compare its response quality and latency against the production model before deciding to switch.

The comparison must:

  • Use live production traffic.
  • Not expose users to the new model's responses.
  • Capture both models' outputs for offline human and automated evaluation.
  • Compare operational metrics (latency, error rate) automatically.

Which ONE approach should the team use?

  • A) Configure a SageMaker shadow test with the production Flan-T5 variant and the Mistral shadow variant on the same endpoint. Route 100% of traffic to production, send a copy to the shadow. Log both outputs to S3 for later human/judge evaluation. Monitor CloudWatch metrics for both variants to compare latency and error rates.

  • B) Deploy the Mistral model to a separate endpoint. Use Amazon Route 53 weighted routing to send 95% of traffic to the production Flan-T5 endpoint and 5% to the Mistral endpoint. Monitor CloudWatch metrics for both.

  • C) Use Amazon Bedrock evaluations with a judge model to compare the quality of responses from both models on a curated dataset of customer queries. The evaluation runs offline and doesn't use live traffic.

  • D) Use AWS Step Functions to build a workflow that invokes both endpoints for each production request, stores both responses in DynamoDB, and uses a Lambda function to compute latency comparisons.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: SageMaker shadow testing is the native, managed approach for comparing candidate models against production using live traffic without exposing users to the candidate. The shadow variant receives asynchronous copies of live requests; production responses are returned to users. Both variants' CloudWatch metrics (latency, error rate) are automatically collected. Outputs can be logged to S3 for offline evaluation (human or judge model). This satisfies all constraints.

Distractor Analysis

  • B is Incorrect: Route 53 weighted routing sends 5% of real users to the Mistral model, exposing them to potentially incorrect or lower-quality responses. This violates the "not expose users to the new model" constraint. It's an A/B testing approach, not a shadow comparison.
  • C is Incorrect: Amazon Bedrock evaluations evaluates foundation models using offline datasets, not live SageMaker endpoint traffic. The scenario describes comparing two fine-tuned SageMaker models, not Bedrock FMs. This approach also doesn't test operational metrics under real traffic load.
  • D is Incorrect: Building a custom Step Functions workflow to duplicate requests introduces significant operational overhead. SageMaker shadow testing provides this capability natively without custom code, satisfying the "managed, minimal overhead" expectation.

Key Takeaway: SageMaker shadow tests are the managed solution for comparing model variants with live traffic while ensuring users only receive production responses and operational metrics are automatically collected.

Question 18 - NLP Metrics for Text Generation Evaluation

A language learning platform uses an LLM in Amazon Bedrock to generate short grammar explanations. The team wants to evaluate the quality of generated explanations against reference explanations written by language experts.

The evaluation needs to:

  • Measure lexical overlap (exact wording match).
  • Measure semantic similarity (meaning match).
  • Be fully automated without human raters.

Which TWO metrics should be used in the evaluation job? (Select TWO)

  • A) ROUGE (Recall-Oriented Understudy for Gisting Evaluation) to measure n-gram overlap between generated and reference explanations.

  • B) BERTScore to compute semantic similarity by comparing contextual embeddings of generated and reference explanations.

  • C) Latency (response time in milliseconds) to determine which explanation is more useful.

  • D) Perplexity of the generated text to measure fluency and correctness.

  • E) BLEU (Bilingual Evaluation Understudy) with a high n-gram order (e.g., BLEU-4) to measure phrase-level precision against references.

Click to reveal Answer & Explanation

Correct Answers: A and B (or A and E)

Note: The question asks for TWO metrics — one for lexical overlap and one for semantic similarity. ROUGE and BLEU are both overlap-based; BERTScore is semantic. The intended answers are A (overlap) and B (semantic), though E is also a valid overlap metric. The mark scheme prioritizes A + B as the two distinct metric types.

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: ROUGE measures recall-oriented n-gram overlap between generated text and reference text. It is the standard automated metric for summarization and explanation generation tasks, satisfying the "lexical overlap" requirement.
  • B is Correct: BERTScore computes semantic similarity by comparing contextual BERT embeddings of tokens in the generated and reference texts using cosine similarity. It captures meaning beyond exact word matches, satisfying the "semantic similarity" requirement.

Distractor Analysis

  • C is Incorrect: Latency measures how fast the model responds, not the quality, correctness, or usefulness of the generated explanation. A fast incorrect answer is worse than a slower correct one.
  • D is Incorrect: Perplexity measures how confidently the model generates text (how "surprised" it is by the output), not whether the output is factually correct or semantically similar to reference explanations. Low perplexity can correlate with fluency, but not with correctness or quality compared to references.
  • E is Incorrect: BLEU is a valid overlap-based precision metric, but using it alongside ROUGE would be redundant — both measure lexical overlap. The question explicitly asks for one overlap metric and one semantic metric. In the "Select TWO" context, ROUGE + BERTScore is the optimal pair that covers both metric types.

Key Takeaway: When evaluating generated text, combine overlap-based metrics (ROUGE/BLEU) for lexical matching with embedding-based metrics (BERTScore) for semantic similarity — a single metric rarely settles quality questions.

Question 19 - Debugging Model Convergence Issues on SageMaker

A machine learning engineer is training a neural network for sentiment analysis using Amazon SageMaker. After several epochs, the engineer observes the following in CloudWatch metrics:

  • Training loss: decreasing very slowly (0.85 to 0.82 over 10 epochs).
  • Validation loss: also decreasing very slowly.
  • Accuracy: barely improving (73% to 74% over 10 epochs).

The engineer suspects the model is underfitting the data. Which TWO actions should the engineer take? (Select TWO)

  • A) Increase the model capacity by adding more layers, units per layer, or using a more expressive architecture (e.g., switching from logistic regression to a transformer-based model).

  • B) Increase the learning rate slightly, if the loss curve remains smooth but flattening, suggesting the optimizer is stuck in a plateau.

  • C) Add strong L2 regularization to constrain the model's weights and prevent it from fitting the training data too closely.

  • D) Reduce the training dataset size by 80% to speed up convergence and allow the model to fit the remaining data more precisely.

  • E) Enable SageMaker Debugger to capture gradient norms and confirm that gradients are vanishing to zero, indicating the learning signal is too weak.

Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: Underfitting occurs when the model lacks the capacity to learn the underlying patterns in the data. Increasing model complexity (deeper networks, wider layers, or more expressive architectures) allows the model to capture more intricate decision boundaries and reduce both training and validation loss.
  • B is Correct: If the loss is decreasing smoothly but very slowly, the learning rate may be slightly too low, causing the optimizer to make very small steps toward the minimum. A modest learning rate increase (or warm restart with an increased rate) can help the model escape plateaus and converge faster.

Distractor Analysis

  • C is Incorrect: Strong L2 regularization constrains weight magnitudes, reducing model capacity. This is used to combat overfitting (where validation loss diverges from training loss), but would worsen the underfitting problem described here by further limiting the model's expressiveness.
  • D is Incorrect: Reducing the training dataset size typically increases overfitting (less data for the model to generalize from) and does not address underfitting. In fact, more data, not less, generally helps when the model needs to learn more complex patterns.
  • E is Incorrect: While SageMaker Debugger is a valid tool for monitoring gradients, vanishing gradients typically cause the loss to plateau entirely (no decrease at all) rather than decrease slowly but steadily. The scenario describes a model that is learning but at a very slow rate, which points to insufficient capacity or a slightly low learning rate, not necessarily vanishing gradients.

Key Takeaway: Underfitting is diagnosed by slow, smooth loss decrease across both training and validation — fix it by increasing model capacity or raising the learning rate moderately, not by adding regularization.

Question 20 - Retrieval Accuracy Assessment for RAG Monitoring

An enterprise knowledge management system uses Amazon Bedrock Knowledge Bases for a RAG-based question answering system over internal documents. The quality team has noticed that the system sometimes provides answers that don't cite relevant documents from the knowledge base.

The team wants to implement a continuous retrieval accuracy monitoring solution. The requirements are:

  • Automatically measure how well the retriever finds relevant document chunks.
  • Detect when retrieval quality degrades over time (e.g., due to index staleness or embedding drift).
  • Minimize manual intervention.

Which TWO actions should the team implement? (Select TWO)

  • A) Use Amazon Bedrock evaluations with the retrieval source evaluation type to periodically run jobs that compute Context Precision and Context Recall against a curated evaluation dataset of queries and expected relevant chunks.

  • B) Store the expected relevant chunks for each query in an Amazon S3 bucket, and configure the evaluation job to use this as ground truth for retrieval comparison.

  • C) Implement a manual peer review process where team members rate retrieval quality weekly by reading a random sample of query-document pairs.

  • D) Use SageMaker Model Monitor to monitor the OpenSearch Serverless vector index's CPU utilization and query latency as proxies for retrieval accuracy.

  • E) Enable Amazon CloudWatch Logs Insights to automatically detect when generated answers contain phrases like "I don't know" or "not found in the documents."

Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: Amazon Bedrock evaluations supports the retrieval source evaluation type, which directly measures how well the retriever (Bedrock Knowledge Bases + OpenSearch) finds relevant context passages. Metrics like Context Precision (precision of retrieved chunks at rank K) and Context Recall (completeness of relevant chunks retrieved) are computed automatically by a judge model or programmatically. Scheduled jobs allow continuous monitoring.
  • B is Correct: Storing expected relevant chunks in S3 provides the ground truth necessary for quantitative retrieval accuracy evaluation. The evaluation job compares retrieved chunks against these expected chunks to compute precision/recall metrics. This creates a measurable, reproducible baseline for drift detection.

Distractor Analysis

  • C is Incorrect: Manual peer review is labor-intensive, not scalable, and violates the "minimize manual intervention" requirement. While human review can be valuable for edge cases, the primary monitoring solution should be automated.
  • D is Incorrect: CPU utilization and query latency measure the operational health of the OpenSearch cluster, not retrieval accuracy. A cluster can have excellent performance but still return irrelevant chunks due to embedding drift or stale index data.
  • E is Incorrect: CloudWatch Logs Insights searching for "I don't know" phrases only detects when the model explicitly declines to answer, not when retrieval quality degrades. Irrelevant retrieved contexts can still produce fluent but incorrect answers that don't contain these phrases.

Key Takeaway: Use Bedrock evaluations' retrieval source type with a curated ground truth dataset (stored in S3) to continuously monitor Context Precision/Recall — separating retrieval quality from generation quality enables root cause analysis in RAG systems.

Question 21 - Human-in-the-Loop Evaluation for GenAI Workflow

A pharmaceutical company is developing a medical information chatbot using Amazon Bedrock. The chatbot answers questions about drug interactions and side effects. Due to regulatory requirements, the company must implement a human review gate for high-risk outputs.

The requirements are:

  • Automatically identify which responses need human review before delivery to users.
  • Use a second LLM to pre-screen responses for safety risks and factual accuracy.
  • Only responses flagged as high-risk are routed to human reviewers (pharmacists).
  • The workflow must be orchestrated with managed services.

Which TWO building blocks should the solution architect use? (Select TWO)

  • A) Amazon Bedrock with a guardrails configuration that detects unsafe content and a judge model to score factual accuracy of generated responses. Responses failing thresholds are tagged for review.

  • B) AWS Step Functions workflow that orchestrates: generate response → LLM-judge pre-screening → conditional routing to Amazon A2I (Augmented AI) human review loop if flagged → return approved response to user.

  • C) A single Lambda function that generates the response, calls a second Bedrock model for screening, and directly emails a pharmacist if a response is flagged. The pharmacist manually types the corrected response into a DynamoDB table.

  • D) Amazon SageMaker Ground Truth labeling job that runs as a scheduled batch, where pharmacists review all generated responses from the previous day.

  • E) Amazon EventBridge rules that trigger when guardrails are violated, sending an SNS notification to a health monitoring team.

Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: Amazon Bedrock Guardrails detects unsafe content (e.g., incorrect medical advice patterns) and can block or flag risky responses. An LLM-as-a-judge model scores factual accuracy. These automated pre-screening steps identify which responses need human review, satisfying the "automatically identify high-risk outputs" requirement.
  • B is Correct: AWS Step Functions orchestrates the full human-in-the-loop workflow: generation → automated pre-screening → conditional routing to Amazon A2I (Augmented AI) for human review only when flagged → return of approved/reviewed response. This is a managed, serverless orchestration pattern that meets regulatory compliance with minimal custom code.

Distractor Analysis

  • C is Incorrect: A single Lambda function with manual email to pharmacists lacks scalability, tracking, and auditability. There's no managed human review workflow (no task routing, SLA tracking, review history). The pharmacist manually typing corrections into DynamoDB is a manual, error-prone process that doesn't meet regulatory traceability.
  • D is Incorrect: SageMaker Ground Truth is designed for creating labeled training datasets, not for real-time human review gates in production workflows. A scheduled batch review of all responses (not just high-risk ones) is also inefficient and doesn't meet the "only flagged responses" constraint.
  • E is Incorrect: EventBridge rules with SNS notifications only alert a team about violations; they don't create a human review workflow with task routing, response correction, or integration back into the user-facing flow. This is insufficient for a regulated human-in-the-loop review gate.

Key Takeaway: For regulated GenAI human review gates, combine Guardrails + LLM-as-a-judge pre-screening with Step Functions orchestration and Amazon A2I for managed human review loops of only flagged outputs.

Question 22 - Model Quality Drift Detection for Regression Models

An energy company predicts daily solar power generation using a regression model on Amazon SageMaker. The model's MAPE (Mean Absolute Percentage Error) has increased from 5% to 8% over the past two weeks, indicating potential model drift.

The data science team wants to systematically investigate the root cause by comparing:

  • Input feature distributions between the training period and current production period.
  • Model prediction distributions.
  • Feature importance drift.

Which TWO SageMaker tools should the team use? (Select TWO)

  • A) SageMaker Model Monitor data quality monitoring to compare input feature distributions (KL divergence, Kolmogorov-Smirnov test) between the training baseline and current production data.

  • B) SageMaker Clarify to generate feature attribution comparisons between the training-time baseline and current inference predictions, identifying which features have changed in their impact on the model.

  • C) SageMaker Debugger to capture and visualize gradient updates at inference time, identifying which layers are responsible for the drift.

  • D) Amazon Bedrock evaluations with a judge model to explain why the solar predictions are drifting.

  • E) SageMaker Model Monitor model quality monitoring to compute the percentage of missing values in production data as the sole drift indicator.

Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: SageMaker Model Monitor's data quality monitoring compares current production feature distributions against the training baseline using statistical tests (KL divergence, KS test). This helps identify which input features have shifted, which is a key root cause of model drift.
  • B is Correct: SageMaker Clarify can generate SHAP-based feature attribution reports for both training-time baseline predictions and current production predictions. Comparing these attributions identifies features whose importance to the model's decisions has changed, another key diagnostic for model drift.

Distractor Analysis

  • C is Incorrect: SageMaker Debugger is designed for training-time debugging (monitoring gradients, weights, layer activations during training), not for inference-time drift analysis. It cannot capture "gradient updates at inference time" — inference doesn't compute gradients.
  • D is Incorrect: Amazon Bedrock evaluations is for evaluating GenAI models (LLMs), not traditional ML regression models. Bedrock cannot explain why a solar power prediction model is drifting.
  • E is Incorrect: Monitoring only missing values provides an incomplete drift picture. Model drift can occur without any change in missing value rates (e.g., the distribution of numeric features might shift, or new categories might appear). The question describes investigating multiple aspects (feature distributions, feature importance), not just missing data.

Key Takeaway: For root cause analysis of model drift, combine data quality monitoring (input distribution shifts) with Clarify feature attribution comparisons (model importance shifts) for a complete diagnostic picture.

Question 23 - LLM-as-a-Judge for Bias Detection

A financial advisory company uses Amazon Bedrock to generate investment recommendations in a chat application. The compliance team has identified that the system occasionally produces recommendations that may be biased toward certain demographic groups.

The company needs an automated bias detection solution that:

  • Evaluates generated responses for bias across multiple demographic categories.
  • Uses a judge model to score and explain bias.
  • Can run on a scheduled basis and produce reports.

Which ONE approach should the implementation team use?

  • A) Create an Amazon Bedrock evaluation job in LLM-as-a-judge mode, with a custom prompt instructing the judge model to evaluate responses for demographic bias across categories (race, gender, age) and provide a bias score and explanation. Schedule the job to run weekly against a curated set of prompts.

  • B) Use Amazon SageMaker Clarify bias detection to analyze the LLM's training data and prediction outputs for demographic bias, generating a pre-training bias report.

  • C) Implement a Lambda function that calls the LLM to generate a sample of 100 responses, manually reads them for bias, and stores the findings in a S3 bucket.

  • D) Use Amazon Rekognition to detect demographic information in the generated responses and analyze whether that information correlates with recommendation outcomes.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Amazon Bedrock evaluations with LLM-as-a-judge supports automated bias detection for LLM outputs. The judge model can be instructed via a custom judge prompt to evaluate responses for demographic bias across multiple categories, provide structured bias scores (e.g., on a Likert scale), and explain its reasoning. Scheduled evaluation jobs enable continuous monitoring with reproducible reports. This addresses the LLM-specific bias detection requirement.

Distractor Analysis

  • B is Incorrect: SageMaker Clarify bias detection is designed for traditional ML models (feature-based bias metrics like demographic parity, disparate impact) and requires labeled demographic features in tabular data. It doesn't evaluate the text content of LLM-generated responses for biased language.
  • C is Incorrect: Manual reading of responses for bias is not automated, not scalable, and doesn't produce quantitative bias scores or explanations. The Lambda function only generates the responses; a human must do all the assessment, violating the "automated" requirement.
  • D is Incorrect: Amazon Rekognition is designed for image/video analysis (face detection, object recognition), not for analyzing text generation outputs or detecting language bias in investment recommendations. This is a fundamental service mismatch.

Key Takeaway: For automated bias detection in LLM outputs, use Bedrock evaluations with LLM-as-a-judge configured with custom judge prompts that instruct the model to score and explain bias across demographic dimensions — not Clarify or Rekognition.

Question 24 - Monitoring Retrieval Accuracy with Ground Truth

A customer service company uses a RAG system on Amazon Bedrock Knowledge Bases to answer support questions from a product documentation corpus. The team wants to track the retrieval accuracy of the system over time.

The evaluation team has created a golden dataset of 500 customer questions, each with:

  • The correct answer (human-verified).
  • The expected relevant document chunks from the knowledge base.

The team wants to run this evaluation every week automatically. Which TWO configurations should be set up? (Select TWO)

  • A) Store the golden dataset (questions, expected answers, expected chunks) in Amazon S3 in a structured format (e.g., JSONL).

  • B) Configure an Amazon Bedrock evaluations job in retrieval source evaluation mode, pointing to the Bedrock Knowledge Base and the S3-stored golden dataset. Use a judge model to evaluate the relevance of retrieved chunks against the expected chunks.

  • C) Write a custom Python script that uses the sentence-transformers library to compute cosine similarity between retrieved chunks and expected chunks, executing it inside a SageMaker Processing Job scheduled via EventBridge.

  • D) Use AWS Glue to join the retrieved chunks with expected chunks every week, applying a custom SQL rule that counts how many retrievals were correct.

  • E) Deploy the evaluation script as a SageMaker Batch Transform job that runs weekly and outputs a CSV of similarity scores to CloudWatch Logs.

Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: Storing the golden dataset in S3 provides the ground truth source for the evaluation job. Bedrock evaluations reads the dataset directly from S3, making it the correct data location for automated scheduled evaluations.
  • B is Correct: Amazon Bedrock evaluations with retrieval source evaluation mode is the native way to measure retrieval accuracy for Bedrock Knowledge Bases. The judge model evaluates whether the retrieved chunks are relevant to the queries and the expected chunks, computing metrics like Context Precision and Recall. This is a managed solution requiring minimal code and is ideal for weekly scheduled evaluations.

Distractor Analysis

  • C is Incorrect: Building a custom evaluation script with sentence-transformers introduces avoidable operational overhead (dependency management, scaling, scheduling). Bedrock evaluations already provides retrieval accuracy evaluation natively with a judge model. The question implies "automatically" each week, which points to a managed solution.
  • D is Incorrect: AWS Glue is designed for ETL data transformation and data quality rules (null checks, range checks) on tabular data, not for semantic evaluation of text retrieval relevance. Comparing vector retrieval accuracy using SQL requires embedding similarity computation, which Glue doesn't natively support.
  • E is Incorrect: SageMaker Batch Transform is designed for running inference on large datasets (e.g., scoring thousands of data points through a model), not for evaluating retrieval quality. It doesn't provide the judge model or retrieval evaluation capabilities of Bedrock evaluations.

Key Takeaway: For scheduled RAG retrieval accuracy evaluation, use Bedrock evaluations with retrieval source mode and store the golden dataset in S3 — this is the managed, minimal-overhead approach for computing retrieval metrics weekly.

Question 25 - CloudWatch Metrics for Drift Monitoring

A video streaming company has deployed a recommendation model on Amazon SageMaker. The ML team has configured the model to emit custom metrics to CloudWatch, including prediction confidence scores and the count of unique item IDs in each batch.

The team wants to implement drift detection using only CloudWatch native features, with the constraint of no additional ML-specific services. Which TWO actions should the team take? (Select TWO)

  • A) Create CloudWatch Anomaly Detection on the custom metrics to automatically learn the normal patterns and alert when values deviate from expected bands.

  • B) Configure a CloudWatch Alarm with a fixed threshold on the "unique item IDs" metric to alert when the value drops below a static number, representing catalog coverage change.

  • C) Use CloudWatch Logs Insights to query the raw inference logs and manually detect drift on a daily dashboard.

  • D) Use CloudWatch Synthetics to invoke the model endpoint with a set of canary requests and measure