Skip to content

Skill 2.3.10: Configure RAG system monitoring, including retrieval accuracy assessment

Question 1 - RAG Evaluation Metric Selection

A financial services company has deployed a Retrieval-Augmented Generation (RAG) system using Amazon Bedrock Knowledge Bases to answer customer questions about investment products. The compliance team requires that every generated answer be traceable to specific source documents.

The ML team needs to configure monitoring that can separately identify whether failures stem from the vector retriever not finding relevant passages, or from the LLM not properly using retrieved context.

Which evaluation metric configuration should the team implement to isolate these two failure modes? (Select ONE)

  • A) Monitor only end-to-end answer accuracy using BLEU scores against a golden dataset
  • B) Configure separate metrics: Context Precision and Context Recall for retrieval, and Faithfulness and Answer Relevance for generation
  • C) Track token usage and response latency through Amazon CloudWatch to detect quality degradation
  • D) Use Amazon Bedrock Guardrails to filter outputs and trigger alerts when filters are activated
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: RAG systems have two distinct failure points. Retrieval quality is measured with Context Precision (whether retrieved chunks are relevant) and Context Recall (whether all necessary chunks were retrieved). Generation quality is measured with Faithfulness (whether the answer is grounded in retrieved context) and Answer Relevance (whether the answer addresses the question). Isolating these metrics pinpoints the failing component.

Distractor Analysis

  • A is Incorrect: BLEU is an n-gram overlap metric suited for comparing generated text against reference text. It treats the RAG pipeline as a black box and cannot distinguish between retrieval failures and generation failures.
  • C is Incorrect: Token usage and latency are operational metrics. They may indicate performance problems but cannot identify whether a fluent-but-incorrect answer was caused by a failed retrieval or a hallucinating LLM.
  • D is Incorrect: Guardrails filter content (toxicity, PII, etc.) and block harmful outputs. Filter activations indicate content policy violations, not retrieval versus generation quality issues.

Key Takeaway: In RAG monitoring, always measure the retrieval step and the generation step separately — a fluent answer that omits facts is often a retrieval failure wearing a generation costume.

Question 2 - Bedrock RAG Evaluation Configuration

An e-commerce company has built an intelligent product recommendation chatbot using Amazon Bedrock Knowledge Bases with an OpenSearch Serverless vector store. The team wants to run automated evaluations on the RAG system to measure retrieval quality and response quality on a recurring schedule, without building custom evaluation infrastructure.

The solution must minimize operational overhead and use AWS-native tooling.

Which approach should the ML engineer take? (Select TWO)

  • A) Deploy an AWS Lambda function to compute BERTScore and ROUGE metrics against a curated evaluation dataset on a scheduled CloudWatch Events trigger
  • B) Configure an Amazon Bedrock evaluation job that measures the knowledge base's retrieval accuracy using a ground-truth dataset
  • C) Configure an Amazon Bedrock evaluation job that uses an LLM-as-a-judge to score answer faithfulness and answer relevance
  • D) Build a custom evaluator using Amazon EMR to run batch scoring on the evaluation dataset weekly
  • E) Manually review a random sample of 50 conversations daily through the Bedrock console
Click to reveal Answer & Explanation

Correct Answers: B and C

Correct Options Analysis

  • B is Correct: Amazon Bedrock evaluations natively support evaluating retrieval sources (Knowledge Bases), measuring how accurately the retriever fetches relevant chunks against ground-truth question-context pairs.
  • C is Correct: Bedrock evaluations can run LLM-as-a-judge jobs that automatically score generated responses on dimensions such as faithfulness, relevance, and correctness against the retrieved context.

Distractor Analysis

  • A is Incorrect: Building custom Lambda-based evaluation introduces unnecessary operational overhead. Bedrock evaluation jobs provide native retrieval and generation evaluation without custom code.
  • D is Incorrect: EMR-based batch scoring is a heavyweight approach requiring cluster management. Bedrock evaluations are fully managed and require no infrastructure.
  • E is Incorrect: Manual daily review is operationally burdensome, inconsistent, and not scalable. Human review is valuable as a periodic supplement, not a primary monitoring strategy.

Key Takeaway: Amazon Bedrock evaluations provide managed, scheduled evaluation of both retrieval accuracy and LLM response quality — use them instead of building custom evaluation pipelines.

Question 3 - RAG Drift Detection

A healthcare organization uses a RAG application powered by Amazon Bedrock Knowledge Bases to answer clinical staff questions about drug interactions from updated medical literature weekly. The team must detect when the quality of retrieved context degrades after the knowledge base is synced with new documents.

The solution should proactively notify the team when retrieval quality drops below acceptable thresholds.

Which monitoring architecture meets these requirements with the least operational overhead? (Select ONE)

  • A) Use Amazon CloudWatch Synthetics canaries to query the RAG endpoint every hour and store responses for manual review
  • B) Schedule weekly Amazon Bedrock evaluation jobs against a held-out ground-truth dataset, then export metric scores to CloudWatch for threshold-based alarms
  • C) Configure Amazon SageMaker Model Monitor with a custom baseline to detect data drift in the RAG endpoint's input features
  • D) Use Amazon OpenSearch Serverless monitoring dashboards to track vector index query patterns and set alerts on cache hit ratios
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: Scheduled Bedrock evaluation jobs against a stable ground-truth dataset provide ongoing retrieval quality measurement. Exporting metrics (e.g., Context Recall) to CloudWatch enables setting alarms that trigger when quality falls below thresholds — fully managed, no custom pipeline required.

Distractor Analysis

  • A is Incorrect: CloudWatch Synthetics canaries monitor endpoint availability and latency, but they do not evaluate retrieval accuracy. Manual review of responses is operationally heavy and subjective.
  • C is Incorrect: SageMaker Model Monitor detects drift in model input distributions, but RAG retrieval quality requires comparing retrieved documents against ground truth — not just monitoring input feature shifts.
  • D is Incorrect: OpenSearch monitoring tracks operational metrics like query latency and index throughput, not whether the right documents are being retrieved for specific queries.

Key Takeaway: For RAG drift detection, regularly evaluate against a held-out ground-truth dataset and wire the results into CloudWatch alarms — operational dashboards alone cannot detect retrieval quality degradation.

Question 4 - RAG Monitoring Metrics Architecture

A media company runs a RAG-based content summarization system on Amazon Bedrock with a vector database in Amazon OpenSearch Serverless. The MLOps team needs to build a monitoring dashboard that tracks the RAG system's health across three dimensions:

  1. Whether the retriever is fetching relevant context
  2. Whether the LLM answer is grounded in that context
  3. Whether the endpoint is meeting latency and throughput SLAs

Which monitoring configuration best addresses all three dimensions? (Select THREE)

  • A) Use Amazon Bedrock evaluation jobs to compute Context Precision and Context Recall metrics
  • B) Use Amazon Bedrock evaluation jobs with LLM-as-a-judge to compute Faithfulness metrics
  • C) Use CloudWatch to track InvocationLatency, Invocations, and ModelLatency for endpoint performance
  • D) Use Amazon Guardrails to track content filter activations as a proxy for answer quality
  • E) Use only CloudWatch Logs Insights to inspect individual user queries for quality issues
Click to reveal Answer & Explanation

Correct Answers: A, B, and C

Correct Options Analysis

  • A is Correct: Context Precision and Context Recall are the canonical metrics for evaluating whether the retriever fetched the right context — measuring retrieval quality independently.
  • B is Correct: Faithfulness evaluates whether generated answers are grounded in the retrieved context — measuring generation quality independently.
  • C is Correct: CloudWatch provides native endpoint operational metrics (latency, invocation counts, error rates) to track SLA compliance for the serving layer.

Distractor Analysis

  • D is Incorrect: Guardrails content filter activations indicate toxic or blocked content — they are not proxies for semantic answer quality or retrieval accuracy.
  • E is Incorrect: Log inspections are reactive, ad hoc, and not scalable. They cannot systematically measure retrieval accuracy or answer grounding across the full traffic volume.

Key Takeaway: A complete RAG monitoring stack separates three concerns: retrieval quality (Context Precision/Recall), generation quality (Faithfulness/Relevance), and operational SLAs (CloudWatch metrics).

Question 5 - Live Traffic RAG Monitoring

A telecom company has deployed a RAG chatbot using Amazon Bedrock Knowledge Bases to answer billing questions. The system serves thousands of customers daily. The team wants to continuously monitor live production traffic to detect when the retrieval system starts returning less relevant context for real user queries.

They need a solution that does not require them to maintain a separate evaluation dataset for every new topic.

Which approach meets these requirements? (Select ONE)

  • A) Create an Amazon Bedrock evaluation job with a curated ground-truth dataset covering all known billing topics, and schedule it weekly
  • B) Use Amazon Bedrock Knowledge Bases' built-in relevance scoring per retrieved chunk, log scores to CloudWatch, and alert when average relevance drops below a threshold
  • C) Route 10% of live traffic to a shadow RAG variant with a different embedding model and compare answers offline
  • D) Capture user feedback signals (thumbs up/down) in the chatbot UI and use them as the only quality metric
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: Knowledge Bases returns retrieval scores for each chunk. By logging these scores and tracking their average over time in CloudWatch, the team can detect retrieval degradation on actual live queries — no need for a manually curated evaluation dataset.

Distractor Analysis

  • A is Incorrect: A curated ground-truth dataset is valuable but may not reflect the diversity of live user queries, and it requires ongoing maintenance. It also does not provide continuous live traffic monitoring.
  • C is Incorrect: Shadow testing compares two candidate variants, but does not directly measure whether the retrieval results are relevant for the incoming queries. It also introduces a second model variant to maintain.
  • D is Incorrect: User feedback is a useful signal but is sparse, biased, and often too late to detect gradual retrieval quality degradation. It cannot serve as the only quality metric.

Key Takeaway: For live traffic monitoring of RAG systems, leverage the retrieval confidence scores already emitted by the knowledge base service and track their distributions in CloudWatch for drift.

Question 6 - Debugging RAG Quality Degradation

An engineering team operates a RAG system for a legal research platform. After a document corpus update, users report that answers are fluent but sometimes omit key facts from the updated source documents.

The team needs to determine the root cause using Amazon Bedrock evaluation results.

Which combination of evaluation signals would identify whether the issue is in retrieval or generation? (Select ONE)

  • A) High Context Recall + Low Faithfulness → the retriever found the right chunks, but the LLM failed to use them
  • B) High Answer Relevance + Low BLEU → the LLM is paraphrasing too much and must be retrained
  • C) Low Context Precision + High Faithfulness → the retriever is returning too many chunks, overwhelming the LLM
  • D) Low InvocationLatency + High ErrorRate → the endpoint is rate-limiting the retriever
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: High Context Recall means the correct facts were present in the retrieved context. Low Faithfulness means the LLM failed to generate an answer grounded in that context. This combination isolates the failure to the generation step — the model is ignoring or dropping facts it was given.

Distractor Analysis

  • B is Incorrect: Low BLEU indicates lexical mismatch with reference text, which may reflect valid paraphrasing rather than quality issues. BLEU does not distinguish between retrieval and generation failures.
  • C is Incorrect: Low Context Precision with High Faithfulness suggests the LLM grounded its answer in whatever was retrieved, but the retrieved chunks may be irrelevant. This points to retrieval, not "overwhelming" the LLM.
  • D is Incorrect: Latency and error rates are operational signals that do not relate to fact omission or retrieved context quality.

Key Takeaway: To isolate RAG failures: check Context Recall (did we fetch the right facts?) and Faithfulness (did the LLM use them?). High Recall + Low Faithfulness = generation problem. Low Recall = retrieval problem.

Question 7 - RAG Evaluation Dataset Design

A manufacturing enterprise is configuring an Amazon Bedrock evaluation job for their RAG-based technician assistant. The system retrieves maintenance procedures from technical manuals and generates step-by-step repair guidance.

The evaluation must measure both retrieval and generation performance against a single evaluation dataset.

What should the evaluation dataset contain for meaningful results? (Select THREE)

  • A) Prompt-output pairs where the output is the final generated answer only, with no context references
  • B) Query-context pairs mapping each question to the specific document chunks that contain the correct answer
  • C) Golden responses written by senior technicians that serve as the reference for answer quality scoring
  • D) The raw source PDFs of the technical manuals for batch reprocessing during evaluation
  • E) Labels indicating which retrieved passages are relevant for each query to enable precision and recall calculations
Click to reveal Answer & Explanation

Correct Answers: B, C, and E

Correct Options Analysis

  • B is Correct: Query-context pairs are essential for retrieval evaluation. They allow the evaluation job to verify whether the retriever returns the correct chunks that contain the answer.
  • C is Correct: Golden responses provide ground-truth reference answers for generation quality evaluation, enabling metrics like answer correctness and semantic similarity scoring.
  • E is Correct: Relevance labels on retrieved passages enable computation of Context Precision and Context Recall — the core retrieval accuracy metrics for RAG evaluation.

Distractor Analysis

  • A is Incorrect: Providing only the final generated answer without context references treats the RAG pipeline as a black box. This makes it impossible to determine whether poor answers stem from retrieval failures or generation failures.
  • D is Incorrect: Raw source PDFs are ingestion inputs, not evaluation labels. The evaluation job already has access to the knowledge base — the dataset needs ground-truth annotations, not source documents.

Key Takeaway: A well-designed RAG evaluation dataset requires three components: queries, relevance-labeled context chunks for retrieval scoring, and golden reference answers for generation scoring.

Question 8 - Human Evaluation Integration

A financial advisory firm has deployed a RAG system that generates investment recommendations from market reports. The compliance department requires human review of responses for certain high-risk queries before they are returned to clients.

The system is built on Amazon Bedrock with a multi-step orchestration using AWS Step Functions.

Which configuration integrates human evaluation with the least operational overhead? (Select ONE)

  • A) Configure a Step Functions workflow that detects high-risk queries, routes them to an Amazon A2I (Augmented AI) human review task, and only returns responses after human approval
  • B) Log all RAG responses to S3 and have compliance analysts manually review them weekly through the S3 console
  • C) Use Amazon Bedrock Guardrails to block all high-risk queries without human review, eliminating the need for review workflows
  • D) Deploy a custom React dashboard on EC2 where reviewers can inspect responses and click an "approve" button before responses are served
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: Amazon A2I (Augmented AI) is a managed human review service that integrates natively with Step Functions. The workflow can conditionally route high-risk queries to human reviewers, pause execution until review is complete, and only then return the approved response to the client. No custom infrastructure is required.

Distractor Analysis

  • B is Incorrect: Weekly manual review is reactive and cannot meet the "before returned to clients" requirement for real-time human-in-the-loop validation.
  • C is Incorrect: Blocking high-risk queries does not fulfill the requirement — the business needs these queries answered, but with human oversight, not rejected.
  • D is Incorrect: Building a custom review dashboard on EC2 introduces significant operational overhead (servers, security, availability) compared to using the managed A2I service.

Key Takeaway: For human-in-the-loop RAG workflows with low operational overhead, use Step Functions to orchestrate the flow and Amazon A2I to manage human review tasks.

Question 9 - Bedrock Evaluation Job Types

A retail company has a RAG system for customer service. The ML team is comparing three different embedding models for their knowledge base, which currently uses Amazon OpenSearch Serverless for vector storage.

They want to evaluate which embedding model produces the best retrieval accuracy before selecting one for production use.

Which evaluation approach should they use in Amazon Bedrock? (Select ONE)

  • A) Deploy all three embedding models as SageMaker endpoints and run a custom retrieval evaluation script that computes cosine similarity on a sample dataset
  • B) Run an Amazon Bedrock evaluation job with a retrieval-focused evaluation dataset to measure Context Precision and Context Recall against each candidate embedding model
  • C) Use Amazon Bedrock Prompt Management to store three different prompt templates and compare their outputs side-by-side through the console
  • D) Deploy the RAG system with all three embeddings in parallel and route 33% of production traffic to each, then compare user feedback scores
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: Amazon Bedrock evaluation jobs support evaluating knowledge bases with different configurations (including embedding models). Running a retrieval-focused evaluation against the same ground-truth dataset allows a direct, controlled comparison of context precision and context recall for each candidate embedding model.

Distractor Analysis

  • A is Incorrect: Deploying custom SageMaker endpoints and running a custom script introduces significant operational overhead and does not leverage Bedrock's native evaluation capability for knowledge bases.
  • C is Incorrect: Prompt Management compares prompt text variations — it does not evaluate retrieval accuracy or embedding model quality.
  • D is Incorrect: Live traffic splitting is risky because customers are exposed to potentially inferior embedding models. It also conflates embedding quality with user feedback biases and takes much longer to reach conclusions.

Key Takeaway: When comparing candidate retrieval components (embedding models, chunking strategies), use offline Bedrock evaluation jobs against the same ground-truth dataset for controlled, repeatable comparison — not live traffic experiments.

Question 10 - RAG Monitoring Metrics Priority

A logistics company has a RAG system that ships with Amazon Bedrock answering warehouse inventory queries. The system recently started generating confident-sounding but factually wrong answers about stock levels.

The ML team reviews their monitoring dashboards and finds that operational metrics (latency, invocations, error rate) are all within normal ranges.

What monitoring gap allowed this quality regression to go undetected? (Select ONE)

  • A) They did not configure CloudWatch alarms on InvocationLatency and ErrorRate
  • B) They did not monitor retrieval and generation quality metrics separately, only operational metrics
  • C) They did not implement AWS Shield to protect the RAG endpoint from malicious queries
  • D) They did not set up server access logs on the OpenSearch Serverless collection
Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: Operational metrics like latency and error rate measure infrastructure health, not answer quality. A RAG system can operate perfectly from an operational standpoint while producing factually incorrect answers due to retrieval degradation, poor embedding alignment, or LLM hallucination. Monitoring Context Precision/Recall and Faithfulness/Relevance separately would have detected the quality regression.

Distractor Analysis

  • A is Incorrect: CloudWatch alarms on operational metrics would have remained in a healthy state — the problem was answer quality, not endpoint performance.
  • C is Incorrect: AWS Shield provides DDoS protection for internet-facing endpoints. It does not evaluate content accuracy or retrieval quality.
  • D is Incorrect: Server access logs record who accessed the vector store and from where — they do not reveal whether the vector store returned the correct chunks for a given query.

Key Takeaway: Operational health ≠ quality health. RAG systems require dedicated quality monitoring (retrieval accuracy + answer grounding) to catch factual regression that operational dashboards cannot see.