Skill 3.3.11: Configure AI-specific pipeline orchestration for RAG system updates and knowledge base refresh cycles
Task 3.3: Deployment and Orchestration of ML and AI Workflows - Implement automated orchestration and continuous integration and continuous delivery (CI/CD) pipelines for MLOps and AI workloads
Core Concept & Objective
- RAG refresh orchestration: Automate updates to retrieval index and knowledge base so RAG answers stay current without full app rebuild.
- Primary AWS objective: Minimize embedding/compute cost, maintain retrieval quality, enable audit/rollback, and keep CI/CD for RAG artifacts.
- Pipeline stages: Source docs → ingestion → parsing/chunking → embedding → vector index → retrieval → generation. Orchestration triggers only stages affected by change.
AWS Services & Responsibilities
| Service | Role in RAG Orchestration | Key Config/Note |
|---|---|---|
| Amazon Bedrock Knowledge Bases | Managed RAG ingestion, chunking, embedding, vector store sync. | Data source, ingestion jobs, incremental resync. Asynchronous. |
| Amazon S3 | Source document storage. | EventBridge notifications for ObjectCreated/Removed. |
| Amazon EventBridge | Event routing and schedules. | Triggers Step Functions/Lambda on doc changes or cron. |
| AWS Step Functions | Orchestrate multi-step refresh. | Retry, error handling, wait for ingestion completion. |
| AWS Lambda | Custom chunking, embedding, index promotion, alias switch. | Idempotency, DLQ for failed events. |
| Vector Store (OpenSearch Serverless, Aurora pgvector, Neptune Analytics) | Stores embeddings for retrieval. | Use index aliases for atomic blue/green promotion. |
| Amazon Bedrock Prompt Management | Version prompts used by RAG generation. | Release prompt versions; avoid embedding in app code. |
| SageMaker Model Registry / MLflow | Version embedding models, rerankers, fine-tuned FMs. | Lineage for audit and rollback. |
| AWS CodePipeline / CodeBuild | CI/CD for RAG app code, prompt changes, evaluation gates. | Gate deployment on retrieval/answer quality metrics. |
| CloudWatch / CloudTrail | Monitoring, alarms, audit. | Ingestion job status, failed jobs, API calls. |
Managed vs Custom RAG Pipeline
| Aspect | Amazon Bedrock Knowledge Bases | Custom (SageMaker Pipelines + Lambda + OpenSearch) |
|---|---|---|
| Ingestion | Managed, incremental resync | Build delta detection yourself |
| Chunking/Embedding | Configurable, but limited | Full control |
| Orchestration | EventBridge + StartIngestionJob | Step Functions, SageMaker Pipelines |
| Evaluation | Bedrock Model Evaluation | Custom metrics |
| Use Case | Rapid deployment, standard RAG | Advanced chunking, hybrid search, strict control |
Refresh Triggers: Scheduled vs Event-Driven
| Trigger | When to Use | Trade-off | Exam Note |
|---|---|---|---|
| Scheduled (EventBridge cron) | Predictable cadence, simple reconciliation. | Wastes work if no changes; delayed freshness. | Good for periodic full reconciliation. |
| Event-Driven (S3 → EventBridge → Step Functions/Lambda → Bedrock ingestion) | Immediate updates on doc changes. | Requires idempotency, DLQ, event exists. | Efficient; only processes deltas. |
| Manual | During pipeline trust-building. | Not for production automation. | Violates CI/CD principles. |
| Hybrid | Event for hot docs + schedule for reconciliation. | More complex but robust. | Common production pattern. |
Gotcha: Incremental resync processes only added, changed, or removed documents since last run. It does NOT regenerate all vectors.
Incremental Resync vs Full Reindex
| Change Type | Required Action | Cost | Notes |
|---|---|---|---|
| New/changed/deleted docs | Incremental resync | Low | Skips unchanged; re-embeds changed; removes deleted. |
| Embedding model change | Full reindex | High | All vectors must be regenerated. |
| Chunking strategy change | Full reindex | High | Chunk boundaries change → new embeddings. |
| Index dimensions/schema change | Full reindex | High | Existing vectors incompatible. |
| Index copy | Not a refresh | Low but stale | Copy does not regenerate vectors unless routed through embedding pipeline. |
Hard Rule: Changing embedding model, chunking, or dimensions invalidates all existing vectors. Copying an index is not a refresh.
Orchestration Patterns
- Event-Driven Incremental Refresh: S3 event → EventBridge → Step Functions → Bedrock StartIngestionJob → wait for completion → evaluate retrieval → switch alias if pass.
- Scheduled Reconciliation: EventBridge schedule → Step Functions → compare source vs index → ingest deltas → evaluate.
- Blue/Green Index Promotion: Build new index/version → evaluate retrieval & answer quality → switch alias atomically. Rollback by switching back.
- CI/CD for RAG App: CodePipeline stages: source → build/test → deploy. Gate on RAG evaluation (faithfulness, relevance, context recall). Version prompts with Bedrock Prompt Management.
- Agent Orchestration: If using Bedrock Agents, update action groups, knowledge bases, agent versions; test before alias switch.
Security & Compliance
- IAM: Bedrock KB service role with S3 read, vector store write, KMS decrypt. Least privilege.
- Encryption: KMS at rest for S3, vector store, Bedrock KB. TLS in transit.
- Network: VPC endpoints for S3, Bedrock, OpenSearch Serverless. PrivateLink for private access.
- Audit: CloudTrail for ingestion job API calls, CloudWatch Logs for pipeline steps. Version all artifacts (prompts, models, chunking config).
- Data Residency: Keep source, embeddings, and vector store in same Region; check service availability.
Monitoring & Evaluation
- Ingestion Job Status: CloudWatch metrics, EventBridge events for completion/failure. Alarm on failed jobs.
- Retrieval Quality: context recall, precision, MRR. Answer quality: faithfulness, relevance, toxicity.
- Operational: latency, 5XX errors, token usage, cost.
- Rollback: Keep previous index alias, prompt version, model version. Automatic rollback on alarm or evaluation failure.
Anti-Patterns & Distractor Traps
- Treating index copy as refresh: Leaves stale vectors; looks like progress.
- Full reindex for every doc edit: Unnecessary cost/latency.
- No versioning for prompts/embedding model/chunking: Cannot explain regressions or rollback.
- Ignoring deleted documents: Stale vectors return deleted content.
- Deploying prompt changes without evaluation: Prompt change alters behavior like weight change.
- Assuming resync regenerates all vectors: It does not.
- Manual refresh in production: Violates automation and audit.
- No idempotency in event-driven pipeline: Duplicate events cause duplicate ingestion.
Exam Gotchas: Bedrock KB ingestion jobs are asynchronous. Wait for completion before querying. EventBridge routes S3 events and schedules. Step Functions orchestrates with retries. Use aliases for atomic index switch. Changing embedding model = full reindex. Incremental resync = only deltas.