Skip to content

4.1.1

Question 1 - CloudWatch Generative AI Observability for Agentic Workloads

A financial services company has deployed an AI-powered customer support assistant using Amazon Bedrock Agents with multiple action groups that call internal APIs. The operations team currently monitors only generic Lambda and API Gateway metrics. Recently, users reported that the assistant sometimes stops generating responses midway, and that tool calls occasionally fail silently.

The compliance team requires a unified monitoring solution that provides token usage, latency, error rates, throttling events, and end-to-end prompt traces that capture agent reasoning steps, model invocations, and tool calls, with minimal manual instrumentation effort.

Which service should the ML engineer configure to meet these requirements with the least operational overhead? (Select ONE)

  • A) Amazon CloudWatch generative AI observability with Amazon Bedrock Agents
  • B) AWS X-Ray with the OpenTelemetry Agent sidecar on each Lambda function
  • C) Amazon Managed Service for Prometheus with a custom Grafana dashboard
  • D) AWS CloudTrail with EventBridge rules that trigger alarm notifications
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: Amazon CloudWatch generative AI observability provides pre-built, purpose-built dashboards and telemetry views for Amazon Bedrock workloads, including Agents and AgentCore. It automatically captures model invocation metrics (token usage, latency, errors, throttling) and provides end-to-end prompt tracing that includes agent reasoning steps, model invocations, and tool calls—without requiring the team to write custom instrumentation code.

Distractor Analysis

  • B is Incorrect: While AWS X-Ray does provide distributed tracing, the question specifies "least operational overhead" and a unified view. Using X-Ray with OpenTelemetry sidecars would require custom instrumentation on each Lambda function, adding operational complexity compared to CloudWatch's native generative AI observability.
  • C is Incorrect: Amazon Managed Service for Prometheus with custom Grafana dashboards requires manual setup of metric collectors, dashboards, and instrumentation. It does not automatically provide Bedrock-specific metrics like token usage and agent prompt traces out of the box.
  • D is Incorrect: AWS CloudTrail records API activity for audit purposes but does not capture operational performance metrics such as latency, token usage, or prompt-execution traces. It cannot be used as a real-time monitoring tool for inference workloads.

Key Takeaway: For Bedrock agentic workloads, CloudWatch generative AI observability provides purpose-built, zero-instrumentation monitoring of model invocations and agent execution traces—use it before considering custom tracing stacks.

Question 2 - Bedrock Model Evaluation for Knowledge Base RAG Quality

An e-commerce company uses Amazon Bedrock Knowledge Bases with Amazon OpenSearch Serverless to power a product recommendation chatbot. The ML team wants to systematically assess how well the system answers customer questions using the retrieved documents. They need to run evaluation jobs that measure response quality, correctness, and faithfulness to the retrieved source material, and they must minimize the time spent manually reviewing hundreds of responses.

Which approach should the team use? (Select ONE)

  • A) Use Amazon Bedrock Model Evaluation with automatic metrics and a judge model, configured on the knowledge base responses.
  • B) Write a Lambda function that computes BLEU and ROUGE scores between generated responses and a golden dataset.
  • C) Manually inspect a random sample of 500 chatbot conversations and create a subjective quality scorecard.
  • D) Use Amazon SageMaker Model Monitor with a custom container that computes embedding cosine similarity between queries and responses.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: Amazon Bedrock Model Evaluation supports evaluating knowledge base responses as well as raw model outputs. Automatic evaluation computes metrics such as correctness, completeness, helpfulness, and faithfulness using task-specific algorithms, while judge-model evaluation (an LLM-as-a-judge) scores responses and provides explanations. This directly addresses the requirement to assess response quality and faithfulness to retrieved sources with minimal manual effort.

Distractor Analysis

  • B is Incorrect: BLEU and ROUGE are n-gram overlap metrics originally designed for machine translation and summarization. They do not adequately measure semantic correctness or faithfulness of RAG-generated answers to retrieved context. This approach also requires custom Lambda code and golden dataset maintenance.
  • C is Incorrect: Manual inspection of 500 conversations is labor-intensive, does not scale, and contradicts the constraint of minimizing manual review time. It also introduces subjective bias and does not produce reproducible metrics.
  • D is Incorrect: Amazon SageMaker Model Monitor's embedding-based drift detection evaluates input distribution shifts, not generative response quality or faithfulness to retrieved documents. It would not measure whether the chatbot's answers are correct or grounded in the retrieved sources.

Key Takeaway: Amazon Bedrock Model Evaluation natively supports both automatic metrics and judge-model scoring for knowledge base (RAG) responses, making it the correct choice over custom NLP metrics like BLEU/ROUGE or SageMaker Model Monitor for quality assessment.

Question 3 - Drift Detection Pipeline Architecture

A healthcare platform uses Amazon SageMaker to host a classification model that predicts patient readmission risk from structured clinical features. The ML team needs to implement a fully managed drift detection pipeline that captures a baseline from historical production data, compares current traffic against that baseline, and triggers an alarm when the input feature distribution shifts significantly. The pipeline must minimize custom code and operational overhead.

Which combination of AWS services should the team use? (Select ONE)

  • A) Amazon SageMaker Model Monitor with a data quality schedule, an S3 baseline, and Amazon CloudWatch alarms
  • B) AWS Glue job that computes Kolmogorov-Smirnov tests on weekly batches, with results stored in DynamoDB
  • C) Amazon QuickSight dashboards refreshed hourly from CloudWatch Logs, with manual review by data scientists
  • D) AWS Lambda functions that run statistical tests on event-driven batches and write results to CloudWatch custom metrics
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: Amazon SageMaker Model Monitor natively supports data quality monitoring. The team configures a baseline dataset in S3 (representing stable production traffic), creates a monitoring schedule that captures live inference requests, and Model Monitor automatically computes distribution metrics and compares them against the baseline. Alarms can be configured on the emitted metrics via CloudWatch, fulfilling the fully-managed, minimal-code constraint.

Distractor Analysis

  • B is Incorrect: Building a custom AWS Glue job that manually computes Kolmogorov-Smirnov statistics requires significant custom code, data pipeline maintenance, and scheduling management, violating the "minimize custom code" constraint.
  • C is Incorrect: QuickSight dashboards refreshed hourly from CloudWatch Logs provide passive visualization but not automated drift detection or alerting. The requirement specifies an alarm-triggered pipeline, which manual reviewer dashboards cannot satisfy.
  • D is Incorrect: While Lambda functions with custom metrics could technically work, this approach requires building and maintaining custom statistical test logic, batching data, and handling edge cases—far more operational overhead than the managed SageMaker Model Monitor service.

Key Takeaway: SageMaker Model Monitor with data quality schedules is the managed, native AWS solution for drift detection; it handles baseline capture, distribution comparison, and CloudWatch alarm integration without custom statistical code.

Question 4 - Concept Drift vs. Data Drift Identification

A ride-hailing company operates a surge-pricing model that predicts fare multipliers based on features such as time of day, weather, and number of drivers within a geographic cell. The ML team recently observed that prediction error on the training distribution has remained stable, but the relationship between some features (e.g., number of available drivers) and the correct surge price has changed due to new city regulations.

Which drift detection approach should the team implement to catch this phenomenon? (Select ONE)

  • A) Configure an input feature drift monitor on the model endpoint to detect distributional shifts in incoming requests.
  • B) Set up a monthly model evaluation job that compares predictions against actual outcomes using newly labeled data.
  • C) Use Amazon CloudWatch Anomaly Detection on the endpoint's invocation count metric.
  • D) Deploy an A/B test comparing the old model with a candidate and monitor revenue per 1000 requests.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: This scenario describes concept drift, where the relationship \(P(Y|X)\) between inputs (number of drivers, time of day) and the target (surge price) changes, even though the input distribution \(P(X)\) remains stable. Model evaluation jobs that compare predictions against newly labeled ground-truth data over time can identify concept drift by tracking predictive accuracy on new data. This directly addresses the described phenomenon.

Distractor Analysis

  • A is Incorrect: Input feature drift monitoring (e.g., SageMaker Model Monitor data quality) only detects data drift (a shift in \(P(X)\)). The scenario explicitly states that the input distribution has not changed, so this monitor would not fire an alert and would miss the concept drift.
  • C is Incorrect: CloudWatch Anomaly Detection on invocation counts monitors traffic volume, not the relationship between features and labels. A change in surge-pricing regulations would not alter invocation frequency, so this metric would not capture the drift.
  • D is Incorrect: A/B testing is used to compare model variants on live traffic under a chosen metric—not specifically for detecting that the underlying ground truth has changed. While revenue metrics might indirectly reflect degradation, A/B testing does not directly identify concept drift as a formal monitoring practice.

Key Takeaway: Data drift monitors detect changes in input distributions (\(P(X)\)), while concept drift requires evaluating predictions against fresh ground-truth labels (\(P(Y|X)\))—implement model evaluation jobs or tracking of business metrics to catch concept drift.

Question 5 - Inference Pipeline Anomaly Detection

A logistics company runs a daily batch inference pipeline using AWS Glue for data extraction and transform, then invokes a SageMaker endpoint for route-optimization predictions. The team currently monitors only the Glue job success status and the endpoint error rate. One Monday, the Glue job reports successful completion, but the endpoint's error rate spikes to 35% within an hour. Upon investigation, the team discovers the Glue job silently wrote empty files because the upstream database was unavailable.

Which monitoring improvement would have prevented this failure from going unnoticed until the endpoint errors appeared? (Select ONE)

  • A) Add CloudWatch alarms on the endpoint's high error rate and low invocation count metrics.
  • B) Configure an AWS Glue job to retry database extraction up to 10 times before reporting success.
  • C) Add post-job data quality checks that verify record counts, non-null columns, and schema integrity before allowing downstream inference.
  • D) Use Amazon SageMaker Model Monitor to detect data drift in the endpoint's input features.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Options Analysis

  • C is Correct: The root cause is that the pipeline monitored job completion rather than job output. A data quality check after the Glue job—verifying that the output has the expected number of records, non-null critical fields, and valid schema—would have caught the empty partition and blocked downstream inference. This addresses the monitoring gap directly: a successful exit code is not a guarantee of usable output.

Distractor Analysis

  • A is Incorrect: Adding alarms on the endpoint's error rate only detects the symptom (failed invocations) after the damage has already occurred. It does not prevent the bad data from reaching the endpoint or alert the team before the error spike.
  • B is Incorrect: Retry logic on the Glue job is event-driven, not a monitoring improvement. The upstream database may be unavailable for longer than the retry window, and the job could still successfully finish with empty output after exhausting retries.
  • D is Incorrect: SageMaker Model Monitor detects distributional shifts in endpoint input data, but in this case the endpoint is receiving empty/malformed payloads that cause invocation errors. Model Monitor's drift detection would not necessarily catch a discrete batch of empty records.

Key Takeaway: A successful job exit code does not guarantee usable output—always pair pipeline job monitoring with data quality checks (record counts, schema validation, non-null constraints) to catch upstream failures before they reach inference.

Question 6 - Amazon Bedrock Model Evaluation Types

A media company is building a content summarization application using Amazon Bedrock and a third-party foundation model. The ML team wants to evaluate the model's summarization quality across four dimensions: accuracy, harmfulness, helpfulness, and coherence. They need to identify the specific problems in a generated summary and understand why each score was assigned, so they can tune their prompting strategy.

Which evaluation method should the team use? (Select ONE)

  • A) Automatic model evaluation using task-specific metrics.
  • B) Human-based model evaluation with a curated team of annotators.
  • C) Judge-model evaluation using an Amazon Bedrock model as the evaluator.
  • D) A/B testing using two production variants with weighted traffic distribution.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Options Analysis

  • C is Correct: Judge-model evaluation uses a special "judge" model (an LLM) to score the target model's responses. It provides a numerical score and a natural-language explanation for each dimension, allowing the team to identify specific areas of weakness and tune their prompts. This matches the requirement to identify specific problems and understand the rationale behind each score.

Distractor Analysis

  • A is Incorrect: Automatic model evaluation applies algorithmic metrics (e.g., ROUGE, BERTScore, faithfulness) without providing natural-language explanations of why a score was assigned. It cannot explain the rationale behind a low coherence score.
  • B is Incorrect: Human-based evaluation provides nuanced feedback but is expensive, slow, and requires a dedicated group of human annotators. The question implies a fast, scalable approach that yields per-dimension scores and explanations—judge-model evaluation is the better fit.
  • D is Incorrect: A/B testing compares live production traffic between two model variants on a business metric. It does not evaluate summarization quality across specific dimensions (accuracy, harmfulness, etc.) nor provide per-response explanations for scores.

Key Takeaway: Judge-model evaluation in Amazon Bedrock provides both quantitative scores and natural-language explanations across quality dimensions, making it ideal for iterative prompt engineering; use automatic metrics for scalability and human reviewers for final validation.

Question 7 - A/B Testing with SageMaker Production Variants

An online education platform has deployed a recommendation model using Amazon SageMaker multi-variant endpoints. The current production model ("Variant-A") handles all traffic. The ML team has trained a new candidate model ("Variant-B") and wants to gradually expose it to live users to measure performance on real traffic. The team also wants to ensure that both variants receive the same amount of traffic simultaneously for a fair comparison during the evaluation window.

Which configuration should the team use? (Select ONE)

  • A) Set both variants' weights to 50% and configure Amazon CloudWatch alarms on both variants' metrics.
  • B) Set Variant-A weight to 0 and Variant-B weight to 100, then switch back after a day.
  • C) Deploy Variant-B as a shadow variant, log its predictions, and do not return them to users.
  • D) Use Amazon Bedrock Model Evaluation to compare the two model versions offline.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: SageMaker multi-variant endpoints distribute traffic proportionally to the weights. Setting both variants to 50% (or equal weights) ensures that each variant receives approximately half of the incoming traffic simultaneously, allowing for a fair, concurrent evaluation on live data. CloudWatch metrics for both variants can then be compared over the same evaluation window.

Distractor Analysis

  • B is Incorrect: This is a full cutover rather than a concurrent comparison. It sends no traffic to Variant-A and all traffic to Variant-B, so the team cannot compare both variants on the same traffic pattern during the evaluation window.
  • C is Incorrect: Shadow variants receive a copy of the traffic without returning responses to users, which is useful for testing serving infrastructure but does not evaluate the model against real user outcomes. This does not fulfill the requirement to measure both variants on live traffic.
  • D is Incorrect: Amazon Bedrock Model Evaluation assesses model quality using pre-defined datasets or a judge model—not live production traffic. The requirement specifies A/B testing on live user requests, which is not what Bedrock Model Evaluation provides.

Key Takeaway: SageMaker multi-variant endpoints allow weighted traffic splitting for A/B testing—use equal weights for a fair concurrent comparison, or shadow variants when you need to test infrastructure without exposing candidate responses to users.

Question 8 - Monitoring Foundation Model Invocations

A startup uses Amazon Bedrock to serve a chat application built on a custom fine-tuned model. The ML team wants to monitor model invocation metrics such as token usage per request, latency, error rate, and throttling limits, and also needs to detect when a specific customer's API key is being rate-limited. They want these metrics visible in a centralized dashboard with minimal setup effort.

Which Amazon CloudWatch feature should the team enable? (Select ONE)

  • A) CloudWatch Application Insights with custom rules.
  • B) CloudWatch Container Insights for Amazon EKS.
  • C) CloudWatch generative AI observability for Amazon Bedrock.
  • D) CloudWatch Synthetics canaries targeting the Bedrock API.
Click to reveal Answer & Explanation

Correct Answer: C

Correct Options Analysis

  • C is Correct: CloudWatch generative AI observability for Amazon Bedrock provides a purpose-built dashboard that automatically surfaces model invocation metrics including token usage (prompt and completion tokens), latency, error rates, and throttling events from Bedrock APIs. It requires minimal configuration—essentially enabling the feature and associating the correct IAM permissions.

Distractor Analysis

  • A is Incorrect: CloudWatch Application Insights is designed for monitoring enterprise applications on EC2 and other compute resources, setting up custom rules and dashboards. It does not provide native Bedrock invocation metrics like token usage or prompt-level telemetry.
  • B is Incorrect: CloudWatch Container Insights is specifically for containerized workloads on EKS/ECS. It monitors container-level metrics (CPU, memory, network), not foundation model inference telemetry from Bedrock.
  • D is Incorrect: CloudWatch Synthetics canaries are used to proactively test endpoints by periodically sending simulated requests and checking response availability/latency. They do not capture detailed platform-level metrics like token usage, throttling events, or model-specific inference telemetry.

Key Takeaway: CloudWatch generative AI observability provides native, turnkey dashboards for Bedrock model invocation metrics (tokens, latency, errors, throttling)—use it instead of building custom monitoring with Application Insights or Synthetics.

Question 9 - Drift Detection for Generative AI Embeddings

A global news organization uses Amazon Bedrock with a text-embedding model to power a semantic search and content recommendation system. The ML team wants to monitor whether the distribution of user search queries is drifting over time, which could affect the quality of retrieval results. The query texts are embedded into vectors before being used for search. The team needs a drift detection method that is robust in high-dimensional space and works with the embedded representations.

Which approach should the team implement? (Select ONE)

  • A) Compare the distribution of current query embeddings against a baseline using the maximum mean discrepancy (MMD) statistic, with an alarm triggered when the distance crosses a threshold.
  • B) Compare the distribution of raw query strings using Levenshtein edit distance between each query and a baseline query set.
  • C) Use Amazon CloudWatch Anomaly Detection on the number of unique words per query.
  • D) Run a chi-square test comparing word frequencies between current queries and baseline queries, treating each word as a categorical variable.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: For high-dimensional embeddings (e.g., 384-dim or 1024-dim vectors), traditional distribution tests like chi-square or Kolmogorov-Smirnov become unreliable due to the curse of dimensionality and the continuous nature of embeddings. Maximum Mean Discrepancy (MMD) is a kernel-based statistic specifically designed to compare distributions in high-dimensional or non-Euclidean spaces. This matches the requirement for a drift detection method robust in high dimensions operating on embedded representations.

Distractor Analysis

  • B is Incorrect: Levenshtein edit distance measures string similarity between individual pairs, not distributional differences between groups of queries. It would not capture broader semantic drift in query topics or distribution shifts.
  • C is Incorrect: CloudWatch Anomaly Detection on unique word counts per query is a simple aggregate metric that cannot capture semantic shifts in the types of queries users are asking. It also does not operate on embeddings.
  • D is Incorrect: A chi-square test on word frequencies treats words as independent categorical variables and is unstable in high-dimensional text data. It also ignores semantic relationships (e.g., "car" and "automobile" are different words but semantically similar) and would not detect drift in query meaning or context.

Key Takeaway: For drift detection on generative AI embeddings, use distribution-comparison methods like MMD that are designed for high-dimensional spaces—traditional univariate tests fail as dimensionality increases.

Question 10 - Shadow Variant for Low-Risk Model Evaluation

A critical fraud-detection model serves a financial institution through a SageMaker endpoint. The ML team wants to test a new candidate model against live production traffic, but they cannot risk exposing customers to erroneous fraud predictions. The new model must be evaluated on its predictive accuracy and inference latency under real production load, but its predictions should not influence any customer-facing decisions.

Which deployment strategy should the team use? (Select ONE)

  • A) Deploy the candidate as a shadow variant on the SageMaker endpoint, where its predictions are logged for offline analysis.
  • B) Deploy the candidate with 5% traffic weight, monitor the business impact, and increase if performance looks acceptable.
  • C) Use Amazon Bedrock Model Evaluation with a static validation dataset to score the candidate model before deployment.
  • D) Deploy the candidate on a separate endpoint and run load tests with synthetic traffic generation.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: A shadow variant (also called a ghost variant) receives a copy of live production traffic but its predictions are not returned to users. This allows the team to evaluate the candidate model's accuracy and latency under real production load without any customer-facing risk. The predictions are logged for offline analysis and comparison against the production model's actual outcomes.

Distractor Analysis

  • B is Incorrect: A 5% traffic weight does expose a small number of real customers to the candidate model's predictions. For a critical fraud-detection system, even a small exposure could result in missed fraud or false alerts. The constraint explicitly prohibits exposing customers to erroneous predictions.
  • C is Incorrect: Amazon Bedrock Model Evaluation with a static dataset tests model quality but does not evaluate inference latency or performance under real production load. Static datasets also cannot capture live traffic patterns or seasonal variation.
  • D is Incorrect: Load testing on a separate endpoint with synthetic traffic does not evaluate the candidate model's accuracy on real, live production data. It can measure infrastructure latency, but not how well the model performs on actual transactions.

Key Takeaway: Shadow variants evaluate models on live traffic without user exposure—use them when even small percentages of real traffic pose unacceptable risk; use weighted A/B variants when gradual live exposure provides the best business signal.