Skill 2.3.10: Configure RAG system monitoring, including retrieval accuracy assessment.
Task 2.3: ML Model and Foundation Model (FM) Development - Analyze and evaluate the performance of ML and AI systems.
Objective: Instrument RAG to isolate retrieval vs generation failures, measure retrieval accuracy against ground truth, monitor live quality/safety/cost, and alert on drift.
RAG Monitoring Layers
| Layer | Key metrics | AWS services | Watch for |
|---|---|---|---|
| Ingestion | Sync success, doc count, embedding failures, freshness | Bedrock KB, CloudWatch, EventBridge | Stale index, missing docs, failed sync |
| Retrieval | Context Relevance/Recall/Precision, Recall@k, Precision@k, MRR, NDCG, empty results, latency | Bedrock Evaluations, CloudWatch, vector store metrics | Wrong/missing chunks, poor ranking, latency spikes |
| Generation | Faithfulness, Answer Relevance, Correctness, Completeness, Helpfulness, citation precision/recall, hallucination, tokens | Bedrock Evaluations, Model Invocation Logging, CloudWatch, Guardrails | Hallucination, omission, unsafe content, cost |
| End-to-end | User feedback, task success, guardrail interventions | CloudWatch, QuickSight, Athena | Real-world quality drift |
| Safety | Toxicity, bias, PII, prompt attacks | Bedrock Guardrails, CloudWatch | Compliance violations |
Retrieval Accuracy Assessment
- Golden dataset: queries + relevant doc IDs + reference contexts/answers. Representative, held out.
- Core metrics: Recall@k (all relevant retrieved?), Precision@k (retrieved relevant?), MRR (first relevant rank), NDCG (ranking quality), Hit Rate (any relevant in top-k).
- Bedrock RAG metrics: Context Relevance, Context Recall, Context Precision (retrieval); Faithfulness, Answer Relevance, Correctness, Completeness, Helpfulness (generation).
- RAG Triad: Context Relevance + Faithfulness + Answer Relevance.
- Tuning levers: chunk size/overlap, embedding model, vector store, top-k, hybrid search, reranking, metadata filters.
- Monitor over time: empty result rate, retrieval latency, score distribution, index staleness, embedding drift.
Split retrieval from generation. A fluent answer missing source facts is often a retrieval failure. Measure both layers separately to find root cause.
AWS Service Mapping
| Need | Service | Notes |
|---|---|---|
| Offline RAG evaluation | Amazon Bedrock Evaluations | Programmatic, LLM-as-judge, or human. Evaluate retrieval, generation, or end-to-end. |
| Live LLM call logging | Bedrock Model Invocation Logging | Must be enabled. Logs to CloudWatch Logs or S3. Captures request/response, tokens, latency. |
| Metrics/alarms | Amazon CloudWatch | Bedrock runtime metrics, custom metrics, logs, alarms. |
| KB ingestion monitoring | Bedrock Knowledge Bases + EventBridge + CloudWatch | Sync job status, document counts, freshness. |
| Hallucination/safety | Bedrock Guardrails | Contextual grounding check, PII, harmful content, prompt attacks. Monitor interventions. |
| Prompt versioning | Bedrock Prompt Management | Version prompts, compare variants. |
| RAG tracing | SageMaker AI with MLflow | Trace inputs/outputs at each RAG step. |
| Live variant comparison | SageMaker AI shadow tests | Copy live traffic to shadow variant; production still answers callers. |
| Log analysis | CloudWatch Logs Insights, Athena, QuickSight | Query logs, build dashboards. |
Key Configurations
- Bedrock Evaluations: choose RAG job; select metrics; provide prompt dataset or use KB. Automated jobs run via API; human jobs in console require display for raters. Use LLM-as-judge for scale; calibrate with human labels.
- Model Invocation Logging: enable for Bedrock; configure IAM role, S3/CloudWatch destination, KMS encryption. Not on by default. Logs may contain PII.
- CloudWatch: create custom metrics for retrieval accuracy, empty results, user feedback. Alarms on latency, error rate, throttle, guardrail interventions, faithfulness drop.
- Knowledge Bases: monitor sync jobs, document counts, last sync time. Use EventBridge for job state changes; alert on failures.
- Guardrails: enable contextual grounding check for RAG; monitor interventions. Does not replace evaluation.
- Vector store: monitor OpenSearch/Aurora/pgvector health: CPU, JVM, search latency, indexing rate, OCU.
- Evaluation cadence: offline regression before release; online monitoring continuously; human review for high-stakes outputs.
Security & Compliance
- IAM: least privilege for KB, evaluation jobs, logging, Guardrails. Separate roles.
- Encryption at rest: KMS for S3, OpenSearch, Aurora, CloudWatch Logs, Bedrock.
- Encryption in transit: TLS.
- Private access: VPC endpoints (PrivateLink) for Bedrock, S3, CloudWatch, OpenSearch.
- Data privacy: mask/redact PII in logs; retention policies; Guardrails sensitive info filters.
Trade-offs & Anti-Patterns
- Offline eval vs online monitoring: offline = release gate; online = drift/real traffic. Use both.
- LLM-as-judge vs human: LLM = scalable/cheap; human = authoritative/expensive. Calibrate judge with human labels.
- Retrieval vs generation metrics: only final answer quality hides root cause.
- Top-k trade-off: higher k improves recall but adds latency, cost, noise.
- Chunk size trade-off: small = precise but may lose context; large = context but noise.
- Anti-patterns: single BLEU/ROUGE for RAG; ignoring retrieval latency; no golden dataset; not versioning prompts/embeddings/chunking; logging PII unencrypted; assuming offline eval predicts production; not monitoring KB sync failures; using SageMaker Model Monitor for RAG semantic metrics.
Exam Gotchas - RAG evaluation must separate retrieval (context relevance/recall/precision) from generation (faithfulness/answer relevance/correctness). - Bedrock Evaluations supports programmatic, LLM-as-judge, and human modes. Human jobs in console require browser display; automated do not. - Model Invocation Logging is not enabled by default. Configure IAM, S3/CloudWatch, KMS. - Guardrails contextual grounding detects hallucinations but does not replace evaluation. - Shadow testing compares candidate vs production on live traffic without affecting callers; A/B testing affects users. - SageMaker Model Monitor is for traditional ML drift/quality, not RAG semantic retrieval/generation metrics. - Stale KB index is a common retrieval failure; monitor sync jobs. - KMS + TLS + VPC endpoints are the security baseline for RAG monitoring data.