Skill 3.3.7: Manage Prompts (Amazon Bedrock Prompt Management)
Task 3.3: Deployment and Orchestration of ML and AI Workflows — CI/CD for MLOps and AI Workloads
Exam focus: Prompts are versioned, testable, deployable artifacts. A prompt change is a production change.
Core Objective & Mental Model
- Business objective: Release prompt changes repeatably, traceably, reversibly.
- Behavior unit: Prompt + model + inference config + guardrail + tools + knowledge base = behavior. Version all together.
- Lifecycle: Draft → Variant → Evaluate → Version → Approve → Deploy → Monitor → Rollback.
- Bedrock Prompt Management: Central store for prompts, variants, variables, versions, model/inference settings, guardrails.
- Prompt Flows: For multi-step orchestration; version the flow as well as embedded prompts.
Bedrock Prompt Management Constructs
| Construct | Purpose | Exam Notes |
|---|---|---|
| Prompt | Logical use-case container | Has variants, variables, metadata |
| Variant | Candidate config | Compare models, params, system/user messages, guardrails |
| Variable | Runtime input | Validate required values; never store secrets |
| Version | Immutable release snapshot | Prod uses version ARN; never edit released version |
| Default variant | Fallback | Pin version + variant for reproducibility |
| Guardrail | Safety/policy filter | Attach and test before release |
| Model/inference config | FM ID, temp, topP, max tokens | Store with prompt; avoid app hardcoding |
Key Bedrock operations (conceptual): CreatePrompt, CreatePromptVersion, GetPrompt, ListPrompts, UpdatePrompt, DeletePrompt, InvokeModel with prompt ARN.
Gotcha: A prompt version is immutable. Production must reference a version ARN, not draft.
CI/CD Pipeline Pattern
| Stage | Action | AWS Service |
|---|---|---|
| Source | Prompt templates, tests, eval datasets in Git | CodeCommit / CodeConnections |
| Build | Syntax, variable render, prompt linting | CodeBuild |
| Test | Unit, injection, guardrail, regression tests | CodeBuild |
| Evaluate | Candidate vs baseline on golden set | Bedrock Evaluations / custom |
| Approve | Metric threshold or manual gate | CodePipeline approval / Lambda |
| Deploy | Create prompt version; update app config | Bedrock Prompt Management + IaC/Lambda |
| Monitor | Quality, latency, toxicity, drift, cost | CloudWatch, EventBridge |
| Rollback | Repoint to previous version ARN | App config / canary rollback |
Hard rule: Pipeline should end at versioning + approval, not auto-deploy to 100% without a gate. Separate prompt release from app deployment where possible.
Orchestration split: SageMaker Pipelines orchestrates ML steps; CodePipeline orchestrates release. They hand off at Model Registry / Prompt Management.
CodeDeploy: Rolls out app/compute changes; prompt versioning is handled by Bedrock Prompt Management. Use canary/linear for app routing to new prompt.
Testing & Evaluation Gates
- Deterministic tests: variables render, no unresolved placeholders, required vars present, length/schema validity.
- Safety tests: prompt injection, jailbreak, PII, toxicity, guardrail coverage.
- Quality tests: golden prompt set; candidate vs baseline using LLM-as-a-judge or reference metrics.
- Regression tests: new model + same prompt; new prompt + same model. Both gated.
- A/B/canary: small traffic slice; measure task success, latency, cost, safety.
- Human review: required for high-risk, subjective, brand-sensitive prompts.
Exam trap: Exact string matching fails for generative AI. Use aggregate scores/rubrics/reference metrics.
Versioning, Lineage & Audit
- Prompt version: Immutable; ARN includes prompt ID + version.
- Lineage: prompt version → eval dataset → model version → guardrail → deployment.
- Model Registry: For ML model packages; Prompt Management for prompt versions.
- MLflow on SageMaker AI: Track prompt experiments; register winning run for lineage.
- Fine-tuned models: Register model package; deploy by version; prompt must reference correct model version.
- Agents: Version agent config, tools, prompts together; rollback agent version/alias.
- RAG: Version generation prompt; ingestion resync processes only added/changed/removed docs. Reindex when embedding model/chunking/dimensions change.
Security & Compliance
- IAM: Least privilege for prompt CRUD/version/invoke operations.
- Encryption: Prompts at rest encrypted by Bedrock; use KMS CMK where required. TLS in transit.
- Private access: VPC endpoints (PrivateLink) for Bedrock where required.
- Secrets: Never store API keys, credentials, PII in prompt text/variables.
- Audit: CloudTrail for CRUD/version ops; CloudWatch Logs for invocation metadata (avoid sensitive prompt logging).
- Guardrails: Attach to prompt/variant; verify blocked content before release.
Gotcha: Prompt variables are not a secret store. Use Secrets Manager / Parameter Store, inject at runtime.
When to Use X vs Y
| Decision | Use | Avoid / Trade-off |
|---|---|---|
| Prompt Management vs app code | Central versioning, reuse, eval, guardrails | App strings: no audit, no rollback, drift |
| Variant vs Version | Variant = experiment/candidate | Version = immutable release snapshot |
| Prompt Management vs Prompt Flows | Single prompt lifecycle | Flows for multi-step orchestration; version whole flow |
| Scheduled vs event-driven deploy | Scheduled for batch/eval cycles | Event-driven for approval or dataset update |
| Auto-deploy vs approval gate | Low-risk internal prompts | High-risk/customer-facing: manual + canary |
| Canary vs all-at-once | Canary/linear with alarms for prod | All-at-once only for dev/test or emergency rollback ready |
Anti-Patterns & Distractor Traps
- Editing prompts directly in production → no version, no rollback, no audit.
- Hardcoding model ID / inference params with prompt → can't reproduce behavior.
- No evaluation gate → prompt regression ships silently.
- Exact-match testing on generative output → flaky pipelines.
- Ignoring prompt injection → security/safety failure.
- Versioning model but not prompt → cannot explain behavior change.
- Using draft ARN in prod → unintended changes.
- No rollback plan → bad prompt version stays live.
- Storing PII/secrets in prompts → compliance violation.
- Assuming prompt version = model version → independent; both must be pinned.
Exam Gotchas
Prompt version ARN is the deployment unit. Pin it in production; never reference draft.
A prompt change is a code change. It belongs in Git, CodePipeline, CodeBuild tests, and approval gates.
Generative tests are evaluations, not assertions. Score against criteria; compare aggregate to baseline.
Alarms drive rollback. For canary prompt deployments, configure CloudWatch alarms on error/latency/quality; no alarm = no automatic rollback.
Bedrock Prompt Management ≠ SageMaker Model Registry. Use Model Registry for ML model packages; use Prompt Management for prompt versions.