Skip to content

Skill 3.8: Implement automated agent deployment pipelines and agent version management.

Task 3.3: Deployment and Orchestration of ML and AI Workflows - Implement automated orchestration and continuous integration and continuous delivery (CI/CD) pipelines for MLOps and AI workloads.

Core objective: Treat Amazon Bedrock Agents, prompts, guardrails, and knowledge bases as versioned artifacts; automate promotion from dev to prod via CI/CD; enable safe rollout and rollback using aliases.

Key services: Amazon Bedrock Agents, Bedrock Prompt Management, Bedrock Guardrails, Bedrock Knowledge Bases, AWS CodePipeline, CodeBuild, CodeDeploy, CloudFormation/CDK, EventBridge, Step Functions, CloudWatch, CloudTrail.

Agent Versioning Model

Artifact Mutability Production Use Key Notes
DRAFT Mutable Never Working copy; changes require PrepareAgent
Version Immutable Yes (via alias) Snapshot of agent config, action groups, KBs, prompts, guardrails, model
Alias Mutable Yes Stable endpoint; points to version; can have routing config for canary
Prompt version Immutable Yes Versioned prompt with variables; variants for A/B
Guardrail version Immutable Yes Versioned safety policy
Fine-tuned model Immutable Yes Pin agent to specific model ARN/version

Gotcha: Production applications must call an alias ARN, never DRAFT or a specific version. Alias enables rollback by repointing.

CI/CD Pipeline Stages for Agents

Stage Service Actions
Source CodeCommit / CodeConnections Git repo with IaC (CloudFormation/CDK), Lambda code, OpenAPI schemas, prompts, guardrails
Build CodeBuild Unit tests, lint prompts, validate OpenAPI, package Lambda, build artifacts
Deploy Dev CloudFormation / Step Functions Create/update agent DRAFT, PrepareAgent, create version, create/update dev alias
Test CodeBuild / Lambda Invoke dev alias with golden prompts; test action groups, KB retrieval, guardrails; LLM-as-a-judge evaluation
Approve CodePipeline Manual approval gate
Deploy Prod CloudFormation / Lambda Create new agent version; update prod alias to new version (optionally canary routing); deploy Lambda action groups
Monitor CloudWatch Alarms on errors, latency, guardrail interventions, action group failures
Rollback Lambda / Step Functions Update alias to previous version; revert Lambda

Automated Agent Deployment Pipeline

  • IaC: Define agents, aliases, prompts, guardrails, knowledge bases in CloudFormation/CDK. Use change sets for preview.
  • Orchestration: CodePipeline coordinates source, build, test, approval, deploy. Use Step Functions for complex rollback logic.
  • Action groups: Deploy Lambda functions via CodeDeploy or CloudFormation. Ensure resource policy allows Bedrock to invoke.
  • Knowledge bases: Sync data sources via ingestion jobs. Event-driven on S3 changes or scheduled. Incremental sync only processes added/changed/deleted docs.
  • Prompts: Use Bedrock Prompt Management. Version prompts; reference by ARN in agent. Do not embed prompt text in application code.
  • Guardrails: Version and attach to agent. Test blocked content in pipeline.

Agent Version Management & Rollback

  • Create version: After testing DRAFT, create immutable version. Version includes all agent config.
  • Alias update: Point alias to new version. For canary, use alias routing configuration to split traffic between old and new versions.
  • Rollback: Repoint alias to previous version. No redeploy needed.
  • Audit: CloudTrail logs alias updates and version creations. Use tags for traceability.
  • Fine-tuned models: Pin agent to specific model version ARN. Update agent version when model changes.

Testing & Evaluation

  • Code tests: Unit tests for Lambda, OpenAPI schema validation.
  • Agent tests: Invoke alias with fixed prompt set. Verify action group calls, knowledge base citations, guardrail blocks.
  • Generative evaluation: LLM-as-a-judge, metrics (faithfulness, relevance, toxicity). Compare candidate vs baseline. Gate deployment on score threshold.
  • Prompt testing: A/B variants, regression tests. Version prompts.
  • Trace analysis: Inspect orchestration steps, model invocations, action group inputs/outputs.

Security & Compliance

  • IAM: Agent execution role with least privilege to models, KBs, Lambdas. CodePipeline service role. Lambda resource policy.
  • KMS: Encrypt agent data, prompts, KBs at rest. TLS in transit.
  • VPC endpoints: PrivateLink for Bedrock, S3, KMS, Lambda.
  • Guardrails: Content filters, PII redaction, denied topics.
  • CloudTrail: Log all Bedrock API calls. Monitor alias changes.
  • Least privilege: Scope agent execution role to specific model ARNs, KB IDs, Lambda ARNs.

Trade-offs & Anti-Patterns

Decision Use Avoid
DRAFT vs Version DRAFT for dev; Version for release Pointing prod to DRAFT
Alias vs direct version Alias for prod; enables rollback Hardcoding version ARN in app
CloudFormation vs custom Lambda IaC for reproducibility; custom for complex rollback Manual console changes
Event-driven vs scheduled KB sync Event-driven for irregular changes; scheduled for periodic Full re-index on every change
Prompt versioning Version prompts like code Editing prompts in production
Canary vs all-at-once Canary with alias routing + alarms All-at-once without rollback

Exam Gotchas: - Bedrock Agents have native versioning; do not confuse with SageMaker Model Registry. - PrepareAgent must be called after DRAFT changes before creating a version. - Alias routing configuration enables canary/blue-green, but automatic rollback is not built-in; implement with CloudWatch alarms + Lambda/Step Functions. - Knowledge base sync is incremental; changing embedding model or chunking requires full re-index. - Prompt changes are behavior changes; version them alongside agent. - CodeDeploy deploys compute (Lambda/ECS), not agent configuration; use CloudFormation/Step Functions for agent versions/aliases. - Fine-tuned model versions are separate from agent versions; pin both for reproducibility.

Key AWS Service Roles

Service Role in Agent CI/CD
Amazon Bedrock Agents Versioned agent orchestration
Bedrock Prompt Management Versioned prompts, variants
Bedrock Guardrails Versioned safety policies
Bedrock Knowledge Bases RAG, incremental sync
AWS CodePipeline CI/CD orchestration, approval gates
AWS CodeBuild Build, test, evaluation
AWS CodeDeploy Deploy action group Lambdas/containers
AWS CloudFormation/CDK IaC for agents, prompts, guardrails
Amazon EventBridge Triggers (S3, schedule, approval)
AWS Step Functions Complex rollback, orchestration
Amazon CloudWatch Alarms, metrics, logs
AWS CloudTrail Audit API calls