Skill 2.3.10: Configure RAG system monitoring, including retrieval accuracy assessment
| Question | Answer & Explanation |
|---|---|
| ### Question 1 - Amazon Bedrock Evaluations for Retrieval Accuracy A large enterprise uses Amazon Bedrock Knowledge Bases for a customer support RAG application. The ML team must evaluate the system to determine if the vector database is returning the correct document chunks before the LLM generates the final response. The evaluation must use native AWS tools with the least operational overhead. Which action should the team take to assess the retrieval accuracy natively? - A) Run an Amazon Bedrock evaluation job using built-in retrieval metrics like Context Precision and Context Recall. - B) Deploy a SageMaker shadow variant to compare the RAG endpoint responses with live traffic. - C) Use Amazon CloudWatch Container Insights to monitor the OpenSearch Serverless CPU utilization during queries. - D) Implement an AWS Lambda function to manually compute ROUGE scores between the user query and the retrieved chunks. |
Correct Answer: A Correct Answer Analysis: - A is Correct: Amazon Bedrock Evaluations natively supports evaluating retrieval sources (like Knowledge Bases) by calculating metrics such as Context Precision and Context Recall. This requires no custom infrastructure and directly measures retrieval accuracy. Distractor Analysis: - B is Incorrect: SageMaker shadow variants are used to compare overall endpoint responses against production traffic, not to specifically evaluate the intermediate retrieval steps of a RAG pipeline. - C is Incorrect: Monitoring CPU utilization in CloudWatch tracks infrastructure performance, not the semantic accuracy of the retrieved documents. - D is Incorrect: Writing a custom Lambda function introduces unnecessary operational overhead, and ROUGE measures lexical overlap, not semantic retrieval relevance. > Exam Tip: When asked to evaluate RAG retrieval natively and with least overhead, look for Amazon Bedrock Evaluations using Context Precision/Recall metrics. |
| ### Question 2 - Monitoring RAG Faithfulness A generative AI application uses a RAG architecture with Amazon Bedrock and OpenSearch Serverless. The engineering team notices that the LLM occasionally provides answers that contradict the source documents provided in the prompt context. The team needs to configure a monitoring metric to specifically detect when the generated text is unsupported by the retrieved context, minimizing false positives. Which evaluation metric should they monitor to isolate this issue? - A) Answer Relevance - B) Context Precision - C) Faithfulness - D) Semantic Similarity - E) Word Error Rate (WER) |
Correct Answer: C Correct Answer Analysis: - C is Correct: Faithfulness (also known as Context Utilization or Groundedness) specifically measures whether the facts in the generated answer are supported by the retrieved context. It detects when the LLM hallucinates or contradicts the source documents. Distractor Analysis: - A is Incorrect: Answer Relevance measures how well the generated answer addresses the user's prompt, not whether it is grounded in the retrieved context. - B is Incorrect: Context Precision measures whether the retrieval step fetched the relevant chunks, not whether the LLM adhered to them. - D is Incorrect: Semantic Similarity measures the embedding distance between texts, not factual grounding. - E is Incorrect: Word Error Rate is used for speech-to-text transcription accuracy, not RAG text generation. > Exam Tip: Faithfulness = Is the answer supported by the retrieved context? Answer Relevance = Did it answer the question? Context Precision = Was the right context retrieved? |
| ### Question 3 - Natively Evaluating RAG Components An enterprise uses Amazon Bedrock for their RAG system. They want to run an automated evaluation job to assess retrieval accuracy and generation quality independently. They need to evaluate both the vector search component and the LLM component within the same workflow. Which TWO components can they natively evaluate using Amazon Bedrock Evaluations? (Select TWO) - A) Amazon Bedrock Knowledge Bases (Retrieval source) - B) Amazon Bedrock Foundation Models (Generation source) - C) Amazon OpenSearch Serverless index directly via REST API - D) AWS Lambda functions containing business logic - E) Amazon SageMaker Processing jobs |
Correct Answers: A and B Correct Answer Analysis: - A is Correct: Amazon Bedrock Evaluations natively supports evaluating Knowledge Bases to measure retrieval accuracy (Context Precision, Context Recall). - B is Correct: Amazon Bedrock Evaluations also natively supports evaluating Foundation Models to measure generation quality (Faithfulness, Answer Relevance). Distractor Analysis: - C is Incorrect: While Bedrock Knowledge Bases may use OpenSearch Serverless under the hood, Bedrock Evaluations targets the Knowledge Base abstraction, not the raw OpenSearch REST API directly. - D is Incorrect: Bedrock Evaluations does not evaluate custom Lambda functions; it evaluates AI models and retrieval sources. - E is Incorrect: SageMaker Processing jobs are for data processing and model evaluation on custom datasets, not a native component type for Bedrock Evaluations. > Exam Tip: Amazon Bedrock Evaluations natively supports two main component types: Foundation Models (generation) and Knowledge Bases (retrieval). |
| ### Question 4 - Evaluating Context Recall A financial services company is using a RAG system with a custom embedding model and an OpenSearch vector index. They want to evaluate whether the retrieved passages contain the information necessary to answer the user's query. The team decides to use an automated metric to measure this specific aspect of retrieval accuracy. Which evaluation metric specifically addresses this requirement? - A) Context Recall - B) Context Precision - C) Answer Faithfulness - D) Token Count |
Correct Answer: A Correct Answer Analysis: - A is Correct: Context Recall measures the proportion of relevant information from the ground truth answer that is actually retrieved by the vector database. It checks if the retrieved context contains all the necessary facts to answer the query. Distractor Analysis: - B is Incorrect: Context Precision measures the proportion of retrieved chunks that are actually relevant (i.e., are there irrelevant chunks cluttering the context?). - C is Incorrect: Answer Faithfulness evaluates the generation step, checking if the LLM uses the provided context, not whether the context was retrieved. - D is Incorrect: Token Count measures the length of the text, providing no insight into retrieval accuracy. > Exam Tip: Context Recall = Did we retrieve all the needed facts? Context Precision = Did we avoid retrieving useless facts? |
| ### Question 5 - LLM-as-a-Judge for RAG A healthcare startup is using Amazon Bedrock Knowledge Bases for their medical literature search application. They must monitor the RAG system to ensure the answers are not only factually correct but also relevant to the user's specific medical query. The team decides to use an LLM-as-a-judge approach to evaluate the generation quality natively without requiring human raters for every run. Which AWS feature allows them to configure this natively with the least operational overhead? - A) SageMaker Clarify - B) Amazon Bedrock Evaluations (using a judge model) - C) Amazon Macie - D) AWS CloudTrail |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock Evaluations supports using an automated LLM-as-a-judge (another Foundation Model) to evaluate the outputs of the RAG system. This allows scoring responses for relevance and faithfulness without manual human review overhead. Distractor Analysis: - A is Incorrect: SageMaker Clarify is used for bias detection and explainability in traditional ML models, not for evaluating generative AI output quality. - C is Incorrect: Amazon Macie is a data security service that discovers and protects sensitive data, not an AI evaluation tool. - D is Incorrect: AWS CloudTrail records API calls for auditing, not for evaluating model outputs. > Exam Tip: Amazon Bedrock Evaluations supports three modes: programmatic (built-in metrics), LLM-as-a-judge (automated model scoring), and human evaluation. |
| ### Question 6 - Tracing RAG Inputs and Outputs An e-commerce company has deployed a RAG application using Amazon Bedrock and an OpenSearch vector index. The ML team needs to trace the exact inputs, retrieved contexts, and final outputs for every user interaction to debug a recent spike in irrelevant answers. The solution must capture metadata at each step of the generative AI application with the least operational overhead. Which AWS service should they use to achieve this? - A) AWS CloudTrail - B) Amazon SageMaker AI with MLflow Tracing - C) Amazon VPC Flow Logs - D) Amazon OpenSearch Serverless Audit Logs |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon SageMaker AI with MLflow Tracing allows developers to trace the inputs, outputs, and metadata at each step of a generative AI application, including RAG retrieval and generation steps, facilitating debugging. Distractor Analysis: - A is Incorrect: AWS CloudTrail logs API calls made to AWS services, not the intermediate data payloads within an application's inference logic. - C is Incorrect: VPC Flow Logs capture network traffic metadata, not application-level inputs and outputs. - D is Incorrect: OpenSearch Serverless Audit Logs track access and administrative events, not the semantic content of vector queries or generated responses. > Exam Tip: For tracing the step-by-step inputs/outputs of a GenAI/RAG application, use Amazon SageMaker AI with MLflow Tracing. |
| ### Question 7 - Monitoring Retrieval Latency vs. Accuracy A media company uses a RAG application with Amazon Bedrock Knowledge Bases. They want to monitor the performance of the retrieval system over time to detect both infrastructure degradation and semantic retrieval drift. They need a solution that logs both operational metrics and retrieval performance metrics to a centralized dashboard. Which approach provides the most comprehensive monitoring with the least operational overhead? - A) Manually export OpenSearch logs to an Amazon EC2 instance for custom script analysis. - B) Configure Amazon CloudWatch to collect operational metrics from OpenSearch Serverless, and use Amazon Bedrock Evaluations periodically to assess retrieval accuracy. - C) Write custom Python scripts to query the OpenSearch API and compute cosine similarity, storing results in Amazon RDS. - D) Use AWS X-Ray to trace the vector search queries and automatically compute BLEU scores. |
Correct Answer: B Correct Answer Analysis: - B is Correct: CloudWatch natively monitors OpenSearch Serverless operational metrics (latency, CPU, memory). Bedrock Evaluations provides native retrieval accuracy assessment (semantic drift). Combining them offers comprehensive monitoring without custom infrastructure. Distractor Analysis: - A is Incorrect: Exporting logs to EC2 for custom script analysis introduces massive operational overhead compared to native services. - C is Incorrect: Writing custom Python scripts and using RDS is a high-overhead architectural pattern for monitoring tasks that AWS services can handle natively. - D is Incorrect: AWS X-Ray traces distributed requests but does not automatically compute text generation metrics like BLEU or semantic relevance. > Exam Tip: For RAG monitoring, split concerns: CloudWatch for operational latency, Bedrock Evaluations for semantic accuracy. |
| ### Question 8 - Context Precision for Retrieval An enterprise RAG system uses a custom LLM and a self-managed OpenSearch cluster for retrieval. The team wants to monitor retrieval accuracy by measuring the percentage of retrieved chunks that are actually relevant to the user's query, ensuring the vector search does not return noisy or irrelevant data that could confuse the LLM. Which metric should they implement to evaluate this specific aspect? - A) Faithfulness - B) Context Precision - C) Answer Relevance - D) ROUGE-1 |
Correct Answer: B Correct Answer Analysis: - B is Correct: Context Precision measures the proportion of retrieved chunks that are actually relevant to the query. High context precision means the retriever is not returning useless information that could confuse the generator. Distractor Analysis: - A is Incorrect: Faithfulness measures if the LLM's answer is grounded in the retrieved context, not the quality of the retrieval itself. - C is Incorrect: Answer Relevance measures how well the final answer addresses the prompt, not the purity of the retrieved chunks. - D is Incorrect: ROUGE-1 is an N-gram overlap metric for evaluating text generation against a reference, not for evaluating vector retrieval relevance. > Exam Tip: Context Precision = Signal-to-noise ratio of the retrieval step. High precision means no useless chunks. |
| ### Question 9 - Isolating Retrieval vs. Generation Failures A company is evaluating their RAG system's performance. The LLM generates very fluent and grammatically correct answers, but users complain that the answers frequently miss key facts that are known to be in the source documents. The ML team needs to configure monitoring to isolate whether the issue is with the vector search or the LLM's generation. Which monitoring strategy will effectively isolate the root cause? - A) Measure only the end-to-end response time of the RAG application. - B) Evaluate the Context Recall and Answer Faithfulness independently using Amazon Bedrock Evaluations. - C) Calculate the cosine similarity between the user query and the generated answer. - D) Monitor the total token count of the generated answers over time. |
Correct Answer: B Correct Answer Analysis: - B is Correct: Evaluating Context Recall (retrieval step) and Answer Faithfulness (generation step) independently isolates whether the missing facts are due to the vector search failing to retrieve them, or the LLM ignoring them. Distractor Analysis: - A is Incorrect: Response time measures operational performance, not factual accuracy or missing information. - C is Incorrect: Cosine similarity between the query and the generated answer does not evaluate the retrieval context or factual grounding. - D is Incorrect: Token count tracks output length, not factual completeness. > Exam Tip: To debug missing facts in RAG, evaluate retrieval (Context Recall) and generation (Faithfulness) separately. |
| ### Question 10 - Human Evaluation for RAG A legal tech startup has deployed a RAG application using Amazon Bedrock Knowledge Bases to summarize complex case files. Before releasing a new chunking strategy to production, they need to evaluate its retrieval performance. The team wants to use human reviewers to assess the relevance of retrieved chunks to legal queries, as automated metrics are deemed insufficient for nuanced legal context. The solution must present prompts and retrieved contexts directly to human workers within the AWS console. Which AWS service feature should they use to configure this evaluation? - A) Amazon SageMaker Ground Plus - B) Amazon Bedrock Evaluations (Human evaluation mode) - C) Amazon Mechanical Turk - D) AWS Lambda custom human-in-the-loop workflow |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock Evaluations natively supports a human evaluation mode that presents prompts and responses to human workers directly within the AWS management console, perfect for nuanced subjective tasks like legal text assessment. Distractor Analysis: - A is Incorrect: While SageMaker Ground Plus is used for human labeling, Bedrock Evaluations is the native, lower-overhead choice for evaluating RAG components (Models and Knowledge Bases) specifically. - C is Incorrect: Amazon Mechanical Turk is an external crowdsourcing marketplace, not an integrated AWS native feature for Bedrock RAG evaluation. - D is Incorrect: Building a custom Lambda human-in-the-loop workflow introduces unnecessary operational overhead when Bedrock Evaluations supports this natively. > Exam Tip: Bedrock Evaluations supports human evaluation for subjective tasks, presenting prompts and responses to human workers in the console. |