Skip to content

Skill 3.1.8: Apply Retrieval Augmented Generation (RAG) System Configurations

Question 1 - RAG Retrieval Strategy Selection

A financial services company is building a generative AI assistant for its wealth management advisors using Amazon Bedrock Knowledge Bases. The assistant must answer questions from a corpus of SEC filings, investment research reports, and internal compliance documents stored in Amazon S3.

Advisors frequently search for exact fund ticker symbols (e.g., "VTI", "BND"), exact legal clause references (e.g., "Section 12(d)(1)"), and specific numerical figures (e.g., "expense ratio of 0.03%"). The current vector-only retrieval frequently misses documents containing these exact identifiers because embeddings blur precise alphanumeric strings. The team needs to improve recall for exact terms without replacing the existing vector store.

Which retrieval strategy should the ML engineer configure in the Bedrock Knowledge Base to address this requirement with minimal operational overhead? (Select ONE)

  • A) Enable hybrid search on the knowledge base and verify the vector store maintains a filterable text field.
  • B) Increase the number of returned passages (top_k) from the default to the maximum allowed.
  • C) Add a reranking model to reorder retrieved candidates based on exact token matching.
  • D) Deploy a custom Lambda function that preprocesses queries and injects exact-match filters into the vector search.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Answer Analysis

  • A is Correct: Hybrid search combines semantic vector similarity with lexical/keyword matching (e.g., BM25). Amazon Bedrock Knowledge Bases supports hybrid search natively, allowing exact tokens like ticker symbols and clause references to be matched precisely while still leveraging semantic embeddings for broader conceptual queries. This directly addresses the recall problem for exact alphanumeric strings with zero custom code.

Distractor Analysis

  • B is Incorrect: Increasing top_k widens the candidate set and may coincidentally include more correct passages, but it does not specifically address exact-term recall. It also increases prompt token costs and introduces more irrelevant context, degrading answer quality without fixing the root cause.
  • C is Incorrect: Reranking reorders candidates that were already retrieved by the first-pass vector search. If the exact-token passages were never retrieved because embeddings failed to match them, reranking cannot rescue them—it only improves ordering, not recall.
  • D is Incorrect: Writing a custom Lambda preprocessing layer introduces code to write, test, and maintain. Bedrock Knowledge Bases provides native hybrid search capability, so building a custom solution creates unnecessary operational overhead.

Key Takeaway: Hybrid search is the native Bedrock Knowledge Bases solution for exact-term recall problems—it adds keyword matching to vector search without custom infrastructure.

Question 2 - Reranking Configuration

A legal technology company uses Amazon Bedrock Knowledge Bases with an Amazon OpenSearch Serverless vector collection to power a contract analysis assistant. The knowledge base contains thousands of lengthy contracts, amendments, and legal opinions.

In testing, the correct answer passage is consistently retrieved but appears at position 8–12 out of 20 returned passages. The LLM consumes the top 5 passages as context, so the correct passage is frequently excluded, leading to incomplete answers. The company needs to ensure the most semantically relevant passages are ranked first without increasing the total number of returned passages.

Which configuration change directly addresses this problem? (Select ONE)

  • A) Enable hybrid search to better match keyword-heavy legal terminology.
  • B) Configure a reranking model in the knowledge base to reorder retrieved passages by relevance.
  • C) Reduce the number of returned passages to 5 to force the embedding model to be more precise.
  • D) Re-embed all documents using a higher-dimensional embedding model.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: Reranking is a second-stage pass that re-scores candidates retrieved by vector search. A cross-encoder reranker evaluates the query and passage jointly, producing deeper semantic relevance scores that reorder results. Since the correct passage is already retrieved (position 8–12), reranking will move it to the top positions, ensuring it is included in the LLM's context window.

Distractor Analysis

  • A is Incorrect: Hybrid search improves recall for exact keyword matches, but the problem here is not recall—the correct passage is already retrieved. Adding keyword search addresses a different failure mode and may not fix ordering.
  • C is Incorrect: Reducing top_k to 5 when the correct passage sits at position 8–12 would completely exclude it from the context window, making the answer worse. This shows a fundamental misunderstanding of the relationship between retrieval candidates and ranking quality.
  • D is Incorrect: Re-embedding with higher dimensions may marginally improve semantic distinctions but requires a costly full re-indexing of the knowledge base. It also does not guarantee the correct passage will rank higher—the problem is best addressed with reranking rather than changing embedding dimensions.

Key Takeaway: Reranking fixes ordering problems only; if a passage was never retrieved (recall problem), reranking cannot help—use hybrid search or increase top_k instead.

Question 3 - Metadata Filtering

A healthcare organization uses Amazon Bedrock Knowledge Bases to build a clinical support assistant for physicians. The knowledge base contains medical records, drug formularies, insurance policies, and research papers across multiple hospital departments.

Physicians consistently complain that the assistant returns information about outdated treatment protocols, pediatric medications for adult patients, and policies from other hospital departments. The vector similarity search retrieves these out-of-scope passages because they share topical similarity with valid results.

The compliance team requires that results be restricted to the current year, the patient's department, and approved treatment protocols only. Which retrieval configuration should the ML engineer apply with minimal operational overhead? (Select ONE)

  • A) Apply metadata filters on the knowledge base to narrow the candidate set before vector similarity is computed.
  • B) Add a reranking model trained specifically on compliance violations to push invalid passages down the ranking.
  • C) Increase the number of returned passages so valid results dominate the context window.
  • D) Use query decomposition to break the question into sub-questions for each metadata category.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Answer Analysis

  • A is Correct: Metadata filters are applied before similarity search, restricting the candidate set to documents matching specific attributes (e.g., year=2024, department=cardiology, status=approved). This directly eliminates stale, out-of-department, or unapproved documents from consideration, ensuring retrieved passages comply with the stated constraints. Bedrock Knowledge Bases supports metadata filtering natively.

Distractor Analysis

  • B is Incorrect: Reranking can only reorder passages already retrieved; it cannot guarantee exclusion of non-compliant passages. Training a custom compliance reranker also introduces significant engineering effort, violating the minimal operational overhead requirement.
  • C is Incorrect: Returning more passages increases the odds of including valid results but also adds more invalid content, raising token costs and potentially confusing the LLM. It does not enforce compliance constraints.
  • D is Incorrect: Query decomposition breaks compound questions into sub-questions to improve topical retrieval, but it does not enforce metadata constraints like department or year. It adds complexity without addressing the need for hard filtering.

Key Takeaway: Metadata filters narrow the candidate set before similarity search runs, making them the correct tool for compliance-based restrictions on scope, staleness, or department.

Question 4 - Query Decomposition

A management consulting firm deploys an AI assistant using Amazon Bedrock Knowledge Bases to help consultants navigate a corpus containing market entry strategies, financial models, operational benchmarks, and regulatory frameworks across multiple industries.

Consultants often ask compound questions like "Compare the market entry strategies and regulatory challenges for electric vehicle manufacturers in the EU and Southeast Asia" or "What were the revenue impacts of our pricing recommendations and what operational risks did clients report in 2023?"

The assistant currently retrieves passages that cover only one sub-topic (e.g., only market entry, or only regulatory challenges), missing the second component entirely. Which retrieval configuration best addresses this compound-query recall problem? (Select ONE)

  • A) Enable hybrid search to capture both components of the query.
  • B) Configure query decomposition to break the compound question into sub-questions for targeted retrieval.
  • C) Add a reranking model to ensure passages covering both topics are ranked highest.
  • D) Increase top_k to retrieve more passages that might cover all sub-topics.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: Query decomposition splits a compound question into separate sub-questions (e.g., "market entry strategies for EU EVs", "regulatory challenges for EVs in Southeast Asia"). Each sub-query retrieves against its own topic, and the results are merged. This ensures coverage of multiple distinct components of the original query, which a single embedding often blurs.

Distractor Analysis

  • A is Incorrect: Hybrid search improves exact-term recall but does not address the problem of a compound query where a single embedding vector represents multiple unrelated topics. Keyword matching can supplement, but it does not decompose the query.
  • C is Incorrect: Reranking reorders what was already retrieved; if passages about the second sub-topic were never retrieved because the query embedding skewed toward the first topic, reranking cannot recover them.
  • D is Incorrect: Increasing top_k may incidentally retrieve passages about the second sub-topic, but it also floods the context window with partially relevant content, raising costs and diluting answer quality. It does not systematically ensure each sub-topic is covered.

Key Takeaway: Query decomposition is the correct retrieval lever when a compound question's multiple topics dilute a single embedding query—each sub-question retrieves independently.

Question 5 - Vector Store Compatibility

A startup is building a RAG application using Amazon Bedrock Knowledge Bases. The team initially chose Amazon OpenSearch Serverless as the vector store, but the operations team wants to evaluate whether Amazon Neptune Analytics or a custom Amazon RDS for PostgreSQL with pgvector would reduce overall cost while supporting the same retrieval capabilities.

The application requires hybrid search (semantic + lexical) because users frequently search for product SKUs and part numbers. The ML engineer must determine which vector store options support hybrid search in Bedrock Knowledge Bases without significant custom integration work.

Which statement accurately guides the engineer's decision? (Select ONE)

  • A) All Bedrock Knowledge Bases vector store options support hybrid search out of the box.
  • B) Hybrid search is supported only when the vector store maintains a filterable text field, such as OpenSearch Serverless or Amazon Aurora PostgreSQL with pgvector.
  • C) Hybrid search requires Amazon OpenSearch Serverless exclusively; all other vector stores only support semantic search.
  • D) Hybrid search can be enabled on any vector store by adding a custom Lambda function that writes keyword annotations.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: Amazon Bedrock Knowledge Bases hybrid search depends on the underlying vector store supporting both vector and full-text/lexical queries. Stores like OpenSearch Serverless and Amazon Aurora PostgreSQL (pgvector) maintain a filterable text field enabling BM25-style keyword matching alongside vector similarity. The requirement highlights exact SKU/part-number matching, which hybrid search addresses.

Distractor Analysis

  • A is Incorrect: Not all supported vector stores provide the lexical search capability required for hybrid search. The feature's availability varies by engine, so a blanket statement that all options support it is inaccurate.
  • C is Incorrect: While OpenSearch Serverless supports hybrid search, it is not the exclusive option. Aurora PostgreSQL with pgvector also supports hybrid search configurations within Bedrock Knowledge Bases.
  • D is Incorrect: Writing custom Lambda functions to enable hybrid search defeats the purpose of using a managed service and introduces unnecessary operational overhead. Native support in the vector store is the correct lever.

Key Takeaway: Hybrid search support in Bedrock Knowledge Bases depends on the vector store maintaining a filterable text field—verify engine capabilities before selecting a store.

Question 6 - Reranking vs. Increasing Top-K

An e-commerce company runs a product recommendation assistant using Amazon Bedrock Knowledge Bases with an Amazon OpenSearch Serverless vector store. The knowledge base contains product descriptions, customer reviews, and sizing guides.

The assistant's answers currently include too much irrelevant content (low precision), with the correct passages ranked inconsistently among returned results. Users report that the assistant sometimes cites the wrong product or misses critical compatibility warnings. The team wants to improve answer precision without significantly increasing prompt token costs.

Which configuration change should the ML engineer apply? (Select ONE)

  • A) Increase top_k from 10 to 50 and enable hybrid search for better coverage.
  • B) Add a reranking model and reduce the number of passages passed to the LLM after reranking.
  • C) Replace the vector store with Amazon Kendra to eliminate the need for custom retrieval tuning.
  • D) Disable semantic search and rely exclusively on keyword matching.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: Reranking improves precision by re-scoring candidates and ensuring the most relevant passages are at the top. After reranking, the team can reduce the number of passages passed to the LLM (e.g., top 5 instead of top 20), reducing token costs while improving answer quality. This directly addresses both precision and cost concerns.

Distractor Analysis

  • A is Incorrect: Increasing top_k widens recall but lowers precision by passing more irrelevant passages to the LLM. This also increases token usage, directly contradicting the cost constraint.
  • C is Incorrect: Amazon Kendra is a managed intelligent search service, but replacing the existing OpenSearch vector store introduces migration effort and does not inherently solve the precision problem. Bedrock Knowledge Bases integrates with Kendra, but switching stores is a larger architectural change than adding reranking.
  • D is Incorrect: Keyword-only search addresses some precision issues but sacrifices the semantic understanding crucial for natural language queries, potentially missing contextually relevant passages that don't share exact terms.

Key Takeaway: Reranking + reduced top_k is the standard pattern for improving precision while controlling token costs—retrieve broad, rerank, then pass only the top subset to the LLM.

Question 7 - Metadata Filtering for Regulatory Compliance

A pharmaceutical company builds a regulatory submission assistant using Amazon Bedrock Knowledge Bases. The knowledge base contains clinical trial data, FDA guidance documents, internal research memos, and manufacturing specifications.

Internal policies require that the assistant never retrieve information from documents marked confidential for certain user roles, and that clinical trial data must be restricted to studies in phase III or later. The security team demands that restricted documents be excluded at retrieval time, not just down-ranked.

Which retrieval configuration enforces these constraints? (Select ONE)

  • A) Use metadata filters to exclude documents that fail the role or phase criteria before similarity search runs.
  • B) Add a reranking model that heavily penalizes restricted documents during scoring.
  • C) Enable hybrid search and configure keyword patterns to block restricted terms.
  • D) Use query decomposition to separate restricted queries into a different retrieval path.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Answer Analysis

  • A is Correct: Metadata filters operate before the similarity search and hard-exclude documents that fail the specified criteria (e.g., classification != "confidential" OR role_allowed = true, phase >= "III"). This ensures restricted documents are never part of the retrieval candidate set, satisfying the security requirement for exclusion at retrieval time.

Distractor Analysis

  • B is Incorrect: Reranking only reorders that which was already retrieved. Penalizing restricted documents reduces their likelihood of being passed to the LLM but cannot guarantee they are excluded—a probabilistic approach that fails a hard-security requirement.
  • C is Incorrect: Keyword-based blocking cannot reliably enforce document-level access controls. Hybrid search returns keyword matches, but it doesn't provide deterministic filtering by metadata attributes.
  • D is Incorrect: Query decomposition splits queries for topical coverage; it does not enforce access-controlled metadata filtering. Restricted documents could still surface in the sub-query results.

Key Takeaway: When compliance requires never retrieving certain documents, metadata filters are mandatory—reranking or keyword heuristics only reduce likelihood, they don't enforce policy.

Question 8 - Cross-Region RAG Latency Optimization

A multinational media company uses Amazon Bedrock Knowledge Bases across multiple regions. In the primary region (us-east-1), the knowledge base uses Amazon OpenSearch Serverless for vector storage and an embedding model from Amazon Bedrock.

The company is expanding to Europe (eu-west-1) to serve EU users with lower latency. Compliance requires that user data and document embeddings remain within the EU region. The company wants to replicate the solution with minimal operational overhead and maintain the same retrieval capabilities.

Which approach best meets these requirements? (Select ONE)

  • A) Deploy a second Bedrock Knowledge Base and OpenSearch Serverless collection in eu-west-1, replicating source documents to an S3 bucket in that region.
  • B) Use the existing OpenSearch Serverless collection and enable cross-region replication to eu-west-1.
  • C) Configure Amazon CloudFront to cache the vector embeddings and serve European requests from edge locations.
  • D) Keep the knowledge base in us-east-1 and use Amazon Global Accelerator to route EU traffic with reduced latency.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Answer Analysis

  • A is Correct: Compliance requires EU data residency. Deploying a dedicated Bedrock Knowledge Base and OpenSearch Serverless collection in eu-west-1 keeps embeddings and document processing within the EU region. Source documents can be replicated via S3 Cross-Region Replication (CRR) to an eu-west-1 bucket, from which the new knowledge base ingests. This meets compliance and latency goals with managed AWS services.

Distractor Analysis

  • B is Incorrect: OpenSearch Serverless cross-region replication is not a native managed feature for Bedrock Knowledge Bases vector collections. Even if attempted, it would not address the need for region-local embedding generation and compliance with EU data residency without custom implementation.
  • C is Incorrect: CloudFront caches HTTP responses; it cannot cache vector embeddings for direct retrieval and re-ranking. Vector similarity searches require access to the full vector database.
  • D is Incorrect: Global Accelerator improves network path latency but still routes requests to us-east-1, violating EU data residency compliance because user data and embeddings would be processed and stored outside the EU.

Key Takeaway: Data residency requirements typically dictate full regional deployment of the RAG stack—replicate source data and deploy a separate knowledge base and vector store.

Question 9 - RAG Evaluation and Optimization

An engineering team at a telecommunications company runs a customer support assistant powered by Amazon Bedrock Knowledge Bases. The assistant retrieves passages from a knowledge base containing product manuals, troubleshooting guides, and billing policies.

The team plans to evaluate whether changing the retrieval configuration (hybrid search, reranking, or metadata filtering) improves answer quality. They need to measure the impact using Amazon Bedrock Evaluations.

Which evaluation approach best measures retrieval quality improvements? (Select ONE)

  • A) Configure a knowledge base evaluation using retrieved response metrics such as precision-at-K and recall to compare retrieval configurations.
  • B) Use a custom SageMaker training job to train a classifier that predicts retrieval relevance.
  • C) Manually review a random sample of LLM responses and score them on a Likert scale.
  • D) Deploy an A/B testing framework with both versions and rely on user satisfaction metrics.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Answer Analysis

  • A is Correct: Amazon Bedrock Evaluations (including RAG evaluation) provides automated metrics such as precision@K, recall@K, and MRR to measure retrieval quality against ground-truth queries. It enables systematic comparison of retrieval configurations (e.g., hybrid vs. semantic, reranking on/off) without manual effort.

Distractor Analysis

  • B is Incorrect: Training a custom classifier in SageMaker is a heavy lift involving data labeling, model training, and deployment—far exceeding the minimal overhead expected for evaluating retrieval configuration changes.
  • C is Incorrect: Manual review is error-prone, slow, and does not scale. It also introduces human bias and is not a systematic method for comparing multiple retrieval configurations.
  • D is Incorrect: A/B testing with live users is valuable for overall satisfaction but it is heavily influenced by LLM generation quality in addition to retrieval quality. It does not isolate the impact of retrieval configuration changes with ground-truth metrics.

Key Takeaway: Use Amazon Bedrock Evaluations' retrieval metrics (precision@K, recall@K) to systematically compare retrieval configurations against ground-truth query-answer pairs.

Question 10 - SageMaker RAG Configuration

A data science team builds a RAG pipeline using Amazon SageMaker AI rather than Bedrock Knowledge Bases. The architecture includes a SageMaker endpoint for embeddings, a SageMaker endpoint for an open-weight LLM, and an Amazon OpenSearch Serverless collection for vector storage.

The team wants to implement two-stage retrieval: fast vector search followed by reranking to improve answer quality. They also need to handle exact part numbers and model names that users frequently query.

Which TWO configuration steps should the team implement? (Select TWO)

  • A) Enable hybrid search on the OpenSearch Serverless collection by creating a keyword index on the text field and combining BM25 and vector scores in the query.
  • B) Deploy a reranker model (e.g., Cohere Rerank or a cross-encoder) on a separate SageMaker endpoint and integrate it as a second-stage pass after initial vector retrieval.
  • C) Convert the RAG pipeline to Bedrock Knowledge Bases because OpenSearch Serverless does not support reranking.
  • D) Use Amazon Kendra instead of OpenSearch Serverless because it provides built-in reranking without any custom code.
  • E) Configure a multi-model endpoint in SageMaker to host both the embedding model and the reranker in a single container.
Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Answer Analysis

  • A is Correct: Hybrid search in OpenSearch can be implemented by combining BM25 (keyword) and k-NN (vector) scores in the query. Since users query exact part numbers, hybrid search improves recall for exact matches.
  • B is Correct: Two-stage retrieval with a dedicated reranker is a standard RAG optimization pattern. The reranker can be deployed as a SageMaker endpoint (real-time or serverless) and integrated between the initial retrieval (top 20–50 candidates) and the LLM context construction (top 3–5 after reranking).

Distractor Analysis

  • C is Incorrect: OpenSearch Serverless supports custom retrieval architectures; the fact that Bedrock Knowledge Bases has native reranking does not make OpenSearch incapable of supporting a custom reranking workflow.
  • D is Incorrect: While Kendra is a viable managed search alternative, migrating the entire vector store to Kendra introduces significant architectural changes and potential data migration effort. It is not necessary when OpenSearch plus a SageMaker-hosted reranker works.
  • E is Incorrect: A multi-model endpoint hosts multiple models that share the same framework and serving container. Embedding models (e.g., sentence transformers) and rerankers (cross-encoder architectures) typically use different model frameworks or require separate serving logic, making a multi-model endpoint inappropriate.

Key Takeaway: When building custom RAG on SageMaker + OpenSearch, hybrid search is implemented via OpenSearch query configuration, and rerankers are deployed as separate SageMaker endpoints.