Skill 4.1.6: Configure AI-specific performance monitoring for foundation models (FMs), such as Amazon Bedrock evaluations
| Question | Answer & Explanation |
|---|---|
| ### Question 1 - Amazon Bedrock Evaluation Job Types A global e-commerce company is deploying a fine-tuned foundation model (FM) for a customer service chatbot. The AI engineering team needs to evaluate the model's responses for "helpfulness" and "correctness" at scale without relying entirely on human reviewers, but they also require a subset of complex queries to be manually reviewed by domain experts for nuanced quality assessment. The solution must minimize operational overhead by using AWS native features. Which TWO approaches should the engineers use to fulfill these requirements? (Select TWO) - A) Deploy a custom AWS Lambda function that uses the Claude API to automatically grade all responses, bypassing native evaluation features. - B) Create an automatic evaluation job in Amazon Bedrock using a judge model to score the majority of responses. - C) Configure a human-based evaluation job in Amazon Bedrock and assign domain experts to review the subset of complex queries. - D) Use Amazon CloudWatch Logs Insights to search for sentiment in the model's outputs and manually tag them for expert review. - E) Implement a custom A/B testing pipeline using Amazon SageMaker endpoints without using Amazon Bedrock evaluations. |
Correct Answers: B and C Correct Options Analysis: - B is Correct: Amazon Bedrock automatic evaluation jobs use a judge model (LLM-as-a-judge) to automatically score responses on metrics like helpfulness, correctness, and toxicity, minimizing operational overhead for large datasets. - C is Correct: Bedrock human-based evaluation jobs allow domain experts to review and rate model responses, providing the nuanced quality assessment required for the subset of complex queries. Distractor Analysis: - A is Incorrect: Writing custom Lambda code to grade responses violates the "minimize operational overhead" constraint and ignores native Bedrock evaluation capabilities. - D is Incorrect: CloudWatch Logs Insights is for log querying and pattern matching, not for evaluating model response quality or facilitating human review workflows. - E is Incorrect: While SageMaker A/B testing is valid for deployment, it doesn't evaluate response quality or facilitate expert review natively like Bedrock evaluations do. > Key Takeaway: Amazon Bedrock model evaluation supports both automatic (LLM-as-a-judge) and human evaluation workflows, allowing teams to blend scalability with expert human review while minimizing operational overhead. |
| ### Question 2 - CloudWatch Generative AI Observability An AI engineering team is running a generative AI application powered by Amazon Bedrock. They need a centralized dashboard to monitor token consumption, invocation latency, and throttling rates across their deployed models. The solution must require the least operational overhead and use AWS native observability services. Which configuration meets these requirements with the LOWEST operational overhead? (Select ONE) - A) Configure Amazon CloudWatch generative AI observability for the Bedrock models. - B) Enable AWS CloudTrail logging and build custom Amazon QuickSight dashboards. - C) Deploy an Amazon Managed Grafana instance and stream Bedrock CloudWatch metrics to it. - D) Write custom AWS Lambda functions to poll Bedrock metrics and store them in Amazon OpenSearch Serverless. |
Correct Answer: A Correct Option Analysis: - A is Correct: Amazon CloudWatch generative AI observability provides pre-built dashboards specifically for generative AI workloads, natively aggregating token usage, latency, error rates, and throttling metrics without requiring custom dashboard development. Distractor Analysis: - B is Incorrect: AWS CloudTrail logs API calls for audit purposes but does not provide pre-built dashboards or aggregate performance metrics like token consumption and latency. - C is Incorrect: Deploying Amazon Managed Grafana introduces unnecessary operational overhead compared to the pre-built CloudWatch generative AI observability views. - D is Incorrect: Writing custom Lambda functions to poll metrics introduces significant operational overhead and custom code maintenance, violating the lowest overhead constraint. > Key Takeaway: Amazon CloudWatch generative AI observability offers out-of-the-box dashboards tailored for FM workloads, eliminating the need for custom metric collection and visualization infrastructure. |
| ### Question 3 - Bedrock AgentCore Observability A company is deploying autonomous agents using Amazon Bedrock AgentCore. The engineering team needs to trace the agent's internal reasoning steps, including model inference calls, tool invocations, and memory operations, without modifying the agent's codebase to add custom tracing spans. The solution must minimize operational overhead. Which approach will meet these requirements? (Select ONE) - A) Instrument the agent code with custom AWS X-Ray SDK tracing calls. - B) Enable Amazon Bedrock AgentCore Observability for the AgentCore Runtime. - C) Deploy a sidecar container that captures agent logs and forwards them to CloudWatch. - D) Use AWS CloudTrail to monitor API calls made by the agent to external tools. |
Correct Answer: B Correct Option Analysis: - B is Correct: Amazon Bedrock AgentCore Observability provides default, OpenTelemetry-compatible instrumentation for AgentCore Runtime. It automatically captures traces for model inferences, tool invocations, and memory operations without requiring code modifications. Distractor Analysis: - A is Incorrect: Instrumenting the code with the AWS X-Ray SDK requires modifying the codebase, which violates the constraint of not adding custom tracing spans and increases operational overhead. - C is Incorrect: Deploying a sidecar container for log capture introduces unnecessary architectural complexity and operational overhead compared to native AgentCore Observability. - D is Incorrect: AWS CloudTrail only logs control plane API calls, not the internal reasoning, tool invocations, or inference calls of an agent's execution trace. > Key Takeaway: Amazon Bedrock AgentCore Observability provides out-of-the-box OpenTelemetry-compatible tracing for agents without requiring code instrumentation. |
| ### Question 4 - Evaluating Model Safety Metrics A data science team is setting up an automatic evaluation job in Amazon Bedrock to assess a text generation model. The team needs to ensure the model's responses are safe for production use and do not contain harmful, offensive, or discriminatory content. The evaluation must be performed automatically without human intervention. Which TWO built-in metrics should the engineers include in the evaluation job? (Select TWO) - A) Toxicity - B) Factual knowledge - C) Bias - D) Coherence - E) Prompt stitching |
Correct Answers: A and C Correct Options Analysis: - A is Correct: The Toxicity metric in Bedrock automatic evaluations specifically scores responses for harmful, offensive, or inappropriate content. - C is Correct: The Bias metric evaluates the model's responses for discriminatory or prejudiced content, ensuring fairness and safety. Distractor Analysis: - B is Incorrect: Factual knowledge evaluates the model's ability to recall factual information, not its safety or harmfulness. - D is Incorrect: Coherence evaluates the logical consistency and structure of the response, not safety attributes. - E is Incorrect: "Prompt stitching" is not a valid built-in metric in Amazon Bedrock model evaluations. > Key Takeaway: When evaluating FM safety in Amazon Bedrock, use the built-in Toxicity and Bias metrics for automated, objective assessment of harmful or discriminatory content. |
| ### Question 5 - Knowledge Base Performance Evaluation An enterprise has deployed an Amazon Bedrock Knowledge Base for Retrieval-Augmented Generation (RAG). The AI team needs to evaluate how well the knowledge base retrieves relevant information and generates accurate answers based on that information. They want to use AWS native evaluation capabilities to minimize operational overhead. Which action should the team take? (Select ONE) - A) Deploy a custom SageMaker endpoint to compute BERTScore on retrieved documents. - B) Export the Bedrock Knowledge Base logs to S3 and run an AWS Glue ETL job to count retrieved chunks. - C) Create an evaluation job in Amazon Bedrock that targets the Knowledge Base to score retrieval and response quality. - D) Set up a human review process by sending Knowledge Base responses to Amazon Mechanical Turk. |
Correct Answer: C Correct Option Analysis: - C is Correct: Amazon Bedrock model evaluation natively supports evaluating Knowledge Bases. It can score both the retrieval quality (how relevant the fetched chunks are) and the generation quality (how accurate the final response is) automatically. Distractor Analysis: - A is Incorrect: Deploying a custom SageMaker endpoint introduces unnecessary operational overhead and does not natively evaluate the RAG pipeline's end-to-end performance like Bedrock evaluations do. - B is Incorrect: Running a Glue ETL job to count retrieved chunks only measures quantity, not the semantic relevance or response quality required for evaluation. - D is Incorrect: While human review is an option, sending data to Mechanical Turk bypasses AWS native evaluation capabilities and creates data privacy and operational overhead concerns. > Key Takeaway: Amazon Bedrock model evaluation jobs can natively target Knowledge Bases to score both the retrieval relevance and the generated response accuracy in a single automated pipeline. |
| ### Question 6 - Bedrock Evaluation Datasets A machine learning engineer is configuring an automatic evaluation job for a foundation model in Amazon Bedrock. The team wants to assess the model's general performance on reasoning tasks using standard benchmarks to avoid maintaining custom datasets. The solution must minimize operational overhead. Which approach should the engineer take? (Select ONE) - A) Manually copy public benchmark datasets from Hugging Face into an Amazon S3 bucket and point the evaluation job to it. - B) Use the built-in datasets provided by Amazon Bedrock for the automatic evaluation job. - C) Configure an AWS Lambda function to generate synthetic prompts on the fly for the evaluation. - D) Upload a custom CSV of internal enterprise prompts and run a human evaluation job. |
Correct Answer: B Correct Option Analysis: - B is Correct: Amazon Bedrock provides built-in datasets for automatic evaluations. Using these datasets eliminates the need to source, maintain, and configure external datasets, thus minimizing operational overhead. Distractor Analysis: - A is Incorrect: Downloading datasets from external sources like Hugging Face and managing them in S3 introduces unnecessary operational overhead. - C is Incorrect: Writing a Lambda function to generate synthetic prompts adds custom code maintenance and does not use standard benchmarks. - D is Incorrect: The requirement is to assess reasoning tasks automatically; a human evaluation job with custom prompts violates the "minimize operational overhead" constraint and doesn't use standard benchmarks. > Key Takeaway: Amazon Bedrock automatic evaluations offer built-in datasets, allowing engineers to test models on standard benchmarks without the overhead of managing custom datasets. |
| ### Question 7 - End-to-End Request Tracing in GenAI Applications An AI application uses multiple components, including an FM, an embedding model, and a tool-calling agent. When a user reports a delayed response, the engineering team needs to identify exactly which component caused the delay. The team requires an end-to-end view of the request lifecycle with the least operational overhead. Which AWS service or feature provides this consolidated trace view? (Select ONE) - A) Amazon CloudWatch generative AI observability - B) AWS CloudTrail - C) AWS Config - D) Amazon OpenSearch Service (for log aggregation) |
Correct Answer: A Correct Option Analysis: - A is Correct: Amazon CloudWatch generative AI observability provides pre-built views over generative AI workloads, including end-to-end prompt tracing for agents that call tools, capturing model inference steps, tool invocations, and overall request latency. Distractor Analysis: - B is Incorrect: AWS CloudTrail logs API calls but does not provide distributed tracing or end-to-end latency views for a single request's lifecycle. - C is Incorrect: AWS Config tracks resource configurations and compliance, not runtime request traces. - D is Incorrect: OpenSearch can aggregate logs, but building a custom tracing solution requires significant operational overhead and lacks the pre-built generative AI views of CloudWatch. > Key Takeaway: CloudWatch generative AI observability integrates distributed tracing to provide a single, pre-built view of the entire lifecycle of a generative AI request. |
| ### Question 8 - Detecting Data Drift in Foundation Models A production foundation model is receiving user prompts. The AI engineering team wants to monitor if the distribution of the input prompts is shifting over time compared to a stable baseline, as this could degrade model performance. They need to build an automated pipeline to detect this drift. Which pipeline architecture meets these requirements? (Select ONE) - A) Monitor the average token length of prompts and trigger an alarm if it changes by 10%. - B) Generate embeddings of current prompts, compare their distribution to a baseline using a statistical distance metric, and trigger an alarm if the distance crosses a threshold. - C) Use Amazon Bedrock automatic evaluations to score the toxicity of incoming prompts. - D) Set up an A/B test between the current FM and a new FM variant to compare invocation latencies. |
Correct Answer: B Correct Option Analysis: - B is Correct: Data drift in generative AI is detected by capturing embeddings of input prompts, comparing the current distribution to a stable baseline using a statistic suited for high-dimensional data, and alerting when the distance exceeds a predefined threshold. Distractor Analysis: - A is Incorrect: Token length is a superficial metric and does not capture semantic shifts in the distribution of user prompts (data drift). - C is Incorrect: Toxicity evaluation checks the safety of prompts, not statistical distribution shifts (data drift). - D is Incorrect: A/B testing compares model variants on live traffic, it is not a pipeline for detecting input data distribution drift. > Key Takeaway: Detecting data drift in FMs requires an embedding-based pipeline that statistically compares current prompt distributions to a baseline to identify semantic shifts. |
| ### Question 9 - Comparing Models with Bedrock Evaluations An AI team is deciding between two foundation models (Model A and Model B) for a new summarization task. They need to objectively compare the models on specific metrics like coherence and fluency using the exact same evaluation dataset. The process must be fully automated and repeatable to minimize operational overhead. Which approach should the team take? (Select ONE) - A) Set up an Amazon SageMaker A/B test, distribute live traffic equally, and compare CloudWatch latency metrics. - B) Create a single automatic model evaluation job in Amazon Bedrock with both models as targets, using the same built-in or custom dataset. - C) Deploy both models on EC2 instances, run a custom Python evaluation script against them, and store results in RDS. - D) Use Amazon Bedrock Model Distillation to compare the outputs of both models on a training dataset. |
Correct Answer: B Correct Option Analysis: - B is Correct: Amazon Bedrock model evaluation allows engineers to specify multiple models as targets in a single automatic evaluation job. This runs the same dataset against both models and scores them on identical metrics (like coherence and fluency) automatically, providing an objective, repeatable comparison. Distractor Analysis: - A is Incorrect: A/B testing on SageMaker compares operational metrics (like latency/throughput) on live traffic, not objective quality metrics like coherence and fluency on a fixed dataset. - C is Incorrect: Deploying on EC2 and writing custom scripts introduces massive operational overhead compared to using native Bedrock evaluations. - D is Incorrect: Model distillation is used for training a smaller model based on a larger one, not for evaluating and comparing model quality metrics. > Key Takeaway: To objectively compare multiple FMs, configure a single Amazon Bedrock automatic evaluation job with both models as targets to ensure identical datasets and metrics are used. |
| ### Question 10 - Identifying Concept Drift in FM Outputs A customer service foundation model has been in production for 6 months. The input prompts from users have remained statistically similar to the training baseline (verified by an embedding pipeline), but the customer satisfaction scores for the model's responses have dropped significantly over the last month. The team needs to identify the type of drift occurring to address the root cause. Which condition is the team experiencing? (Select ONE) - A) Data drift - B) Concept drift - C) A throttling limit on the model invocation call - D) A failure in one of the tools the assistant calls |
Correct Answer: B Correct Option Analysis: - B is Correct: Concept drift occurs when the input data distribution (\(P(X)\)) remains constant, but the relationship between the input and the desired output ($P(Y |