Skip to content

3.3.9

Skill 3.3: Implement automated orchestration and continuous integration and continuous delivery (CI/CD) pipelines

Question 1 – Automated Rollback Configuration

A financial services company deploys a fraud detection model to a SageMaker endpoint using blue/green deployment. The MLOps team needs automatic rollback if the new model version's 5XX error rate exceeds 2% or its average latency exceeds 500ms during a 30-minute validation window after a canary shift of 25% traffic. The team wants minimal operational overhead.

Which configuration achieves this requirement? (Select ONE)

  • A) Use SageMaker Deployment Guardrails with a CloudWatch alarm on Invocation5XXErrors > 2% and ModelLatency > 500ms, set the baking period to 30 minutes, and use an AutoRollbackConfiguration tied to these alarms.
  • B) Configure the endpoint with Application Auto Scaling policies triggered on Invocation5XXErrors and ModelLatency, which automatically reverts to the previous model version when thresholds breach.
  • C) Create a Lambda function that polls CloudWatch metrics every minute, manually shifts traffic back to the old model version using UpdateEndpointWeightsAndCapacities if the thresholds breach.
  • D) Use CodeDeploy to deploy the new model and configure its CloudWatch alarms for automatic rollback based on the specified error and latency thresholds.
  • E) Enable SageMaker Model Monitor and set its action to revert the endpoint to the previous model version when it detects data drift or high error rates.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: SageMaker Deployment Guardrails (blue/green deployment with managed traffic shifting) supports AutoRollbackConfiguration that watches one or more CloudWatch alarms during a baking period. If any alarm trips, SageMaker automatically reroutes 100% of traffic back to the original blue fleet, requiring zero manual intervention.

Distractor Analysis

  • B is Incorrect: Application Auto Scaling manages compute capacity based on utilization metrics—it cannot trigger rollbacks or change the model version serving production traffic.
  • C is Incorrect: Writing custom Lambda code to poll metrics and programmatically shift traffic introduces unnecessary operational complexity when SageMaker natively supports automated rollback.
  • D is Incorrect: CodeDeploy is designed for application code deployments to compute targets (EC2, ECS, Lambda), not for managing SageMaker model endpoint blue/green deployments.
  • E is Incorrect: SageMaker Model Monitor detects data drift and quality issues but does not have built-in traffic shifting or rollback capabilities for deployment validation.

Key Takeaway: SageMaker Deployment Guardrails requires at least one CloudWatch alarm linked via AutoRollbackConfiguration — without an alarm, the baking period will complete and a bad model will be marked as successfully deployed.

Question 2 – External Git Repository Integration

A healthcare ML team uses a self-hosted GitLab instance as their source repository, with strict security compliance policies that prohibit long-lived credentials stored in the pipeline definition. They need a fully managed CI/CD pipeline in AWS that automatically triggers on pushes to the main branch and executes SageMaker training jobs.

Which combination should the team use? (Select TWO)

  • A) Create an AWS CodeConnections resource linked to the external GitLab repository using OAuth, avoiding embedded credentials.
  • B) Store a GitLab Personal Access Token in AWS Secrets Manager and reference it in the CodePipeline source stage configuration.
  • C) Configure CodePipeline to use the CodeConnections source action with the external GitLab repository, detecting changes on the main branch.
  • D) Use AWS CodeCommit as a mirror repository with a scheduled git pull from GitLab every hour to replicate changes.
  • E) Create an IAM role with a GitLab trust policy so CodePipeline can assume it and access the external repository directly.
Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: AWS CodeConnections (formerly CodeStar Connections) provides a managed OAuth/GitHub App-based connection to external Git providers like GitHub, GitLab, and Bitbucket, eliminating the need to store static credentials.
  • C is Correct: CodePipeline's source stage with a CodeConnections action automatically detects commits on the configured branch (e.g., main) and triggers the pipeline, with the repository metadata stored in the connection resource rather than the pipeline definition.

Distractor Analysis

  • B is Incorrect: Storing a PAT in Secrets Manager still requires pipeline configuration to reference it, and violates compliance "no credentials stored" policy—this approach places credentials inside the pipeline configuration even if encrypted.
  • D is Incorrect: Maintaining a mirror repository adds significant operational overhead, introduces deployment lag of up to an hour, and requires custom orchestration that negates the purpose of a managed pipeline source integration.
  • E is Incorrect: External GitLab does not support IAM role trust policies; this option describes a non-existent integration. CodeConnections exists specifically because external providers cannot use IAM roles for authentication.

Key Takeaway: AWS CodeConnections is the only managed AWS service that links CodePipeline to third-party Git repositories without requiring static credentials stored in pipeline configuration.

Question 3 – Event-Driven Retraining Pipeline

A retail company has a demand forecasting model that uses daily sales data uploaded to S3 at irregular intervals by different store systems. The MLOps team wants the model retrained whenever new sales data lands, with the retrained model registered to SageMaker Model Registry for approval before any deployment. They want minimal infrastructure management overhead.

Which architecture achieves this? (Select ONE)

  • A) Configure an S3 Event Notification to invoke an AWS Lambda function that calls StartPipelineExecution on a SageMaker Pipeline, which runs training, evaluation, and registers the model with PendingManualApproval status in the Model Registry.
  • B) Set an EventBridge scheduled rule to run the SageMaker Pipeline every 15 minutes, which checks for new data at the start of the processing step and exits early if no new data is found.
  • C) Deploy a lightweight EC2 instance running a cron job that polls the S3 bucket for new object keys every 5 minutes and triggers the pipeline via the AWS CLI.
  • D) Use S3 Event Notifications to directly trigger a SageMaker training job, bypassing pipeline orchestration, and manually register the model artifact after training completes.
  • E) Configure the SageMaker Pipeline to deploy directly to a production endpoint after successful training, avoiding the overhead of registry approval.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: S3 Event Notifications fire immediately when new objects land, triggering a minimal Lambda function that starts the SageMaker Pipeline. The pipeline ends at Model Registry registration with PendingManualApproval status, maintaining separation between training and production deployment. This is fully serverless with no infrastructure to manage.

Distractor Analysis

  • B is Incorrect: A fixed 15-minute polling interval introduces unnecessary latency (up to 15 minutes) and wastes compute every time the check finds no new data. Event-driven triggers are inherently better for irregular workloads.
  • C is Incorrect: Managing an EC2 instance with cron requires patching, scaling, and uptime management—unnecessary operational overhead when S3 → Lambda → Pipeline provides the same functionality serverlessly.
  • D is Incorrect: Directly triggering a training job bypasses the value of SageMaker Pipelines (evaluation, condition steps, automated registration), and manually registering artifacts breaks automation and lineage tracking.
  • E is Incorrect: Deploying directly from training to production removes the critical human-in-the-loop approval gate that the Model Registry provides, increasing risk of bad models reaching production.

Key Takeaway: Event-driven retraining uses S3 Event Notifications or EventBridge rules to trigger SageMaker Pipelines, but the pipeline should end at Model Registry registration with approval required for deployment.

Question 4 – AI Pipeline Model Evaluation Gate

A social media platform is upgrading its content moderation LLM. The new model must pass automated evaluation before deployment: toxicity scores must not increase by more than 5% on a curated prompt set, and response latency must stay under 800ms. The team uses SageMaker Pipelines with a Model Registry for governance.

Which TWO implementations complete this requirement? (Select TWO)

  • A) Add a SageMaker Pipeline ProcessingStep that runs the curated prompt set against both the baseline and candidate model, computes toxicity metrics, and a ConditionStep comparing scores to the threshold that fails the pipeline if exceeded.
  • B) Add a ConditionStep that compares the candidate model's evaluation metrics from the training step against static thresholds using CloudWatch metrics for toxicity and latency.
  • C) Add a pipeline step that runs EvaluateStep (SageMaker Clarify) to generate model explainability artifacts, then blocks registration if the SHAP values indicate bias above 0.5.
  • D) Register the model to the Model Registry with PendingManualApproval status only after the evaluation step passes, preventing poor models from receiving production approval.
  • E) Use CodePipeline with a CodeBuild project running pytest unit tests against the model's serving code to assert that response latency does not exceed 800ms across the golden prompt set.
Click to reveal Answer & Explanation

Correct Answers: A and D

Correct Options Analysis

  • A is Correct: SageMaker Pipelines supports ProcessingStep for custom evaluation scripts that run prompt sets through both baseline and candidate models. A ConditionStep evaluates numeric metrics (e.g., toxicity increase of 5%) and causes pipeline failure if the condition fails, ensuring no model that violates quality gates proceeds.
  • D is Correct: Gating registration on evaluation success means the model only enters the Model Registry after passing quality checks. Setting PendingManualApproval adds an additional human review layer before deployment.

Distractor Analysis

  • B is Incorrect: Pipeline condition steps read explicit pipeline step outputs (e.g., evaluation metrics stored in S3), not runtime CloudWatch metrics. This option confuses deployment guardrails monitoring with pipeline evaluation gating.
  • C is Incorrect: SageMaker Clarify provides bias and explainability analysis for traditional ML models, but SHAP-based bias scores are not a replacement for toxicity evaluation on LLM outputs. The team needs content moderation-specific metrics.
  • E is Incorrect: Unit tests via pytest verify deterministic code behavior—they cannot evaluate generative AI output quality or reliably measure response latency across a non-deterministic prompt set within a build environment.

Key Takeaway: Pipeline evaluation for generative AI requires running a fixed prompt set through candidate and baseline models, scoring the outputs, and using a ConditionStep to fail the pipeline if metrics deviate from acceptable thresholds.

Question 5 – Incremental RAG Ingestion

An enterprise knowledge base serves a customer support chatbot using Amazon Bedrock Knowledge Bases with an Amazon OpenSearch Serverless vector collection. The source S3 bucket contains over 100,000 documents, and the support team edits approximately 20 documents daily. Compliance requires that document changes be reflected in the chatbot within 15 minutes, and the solution must be cost-effective with minimal operational overhead.

Which approach should the team implement? (Select ONE)

  • A) Trigger an Amazon Bedrock StartIngestionJob API call via S3 Event Notification and Lambda whenever an object is created, modified, or deleted, leveraging Bedrock KB's native incremental resync capability.
  • B) Set an EventBridge Scheduled Rule to run the full ingestion job every 15 minutes, letting Bedrock KB handle delta reconciliation automatically.
  • C) Delete and recreate the OpenSearch Serverless collection daily during off-peak hours to ensure all document changes are captured.
  • D) Use AWS Glue ETL to incrementally process changed documents, manually compute new embeddings with SageMaker, and update the OpenSearch Serverless index via bulk operations.
  • E) Configure the Bedrock Knowledge Base with IncrementalSyncEnabled=true in its data source settings, and Amazon Bedrock automatically detects and applies all S3 changes as they occur without any external trigger.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: Bedrock Knowledge Bases supports incremental resync via the StartIngestionJob API. S3 Event Notifications detect changes immediately, Lambda triggers the API call, and Bedrock processes only added, modified, or deleted documents. This is event-driven, meets the 15-minute window, and minimizes embedding API costs by avoiding re-processing unchanged documents.

Distractor Analysis

  • B is Incorrect: A 15-minute scheduled full ingestion run will work but wastes compute on re-checking all 100,000 documents and may not meet the latency requirement consistently. Event-driven triggering is more precise and cost-effective.
  • C is Incorrect: Deleting and recreating the vector collection daily requires full reindexing, re-embedding all 100,000+ documents, and incurs significant OpenAI/embedding model costs. This violates the "cost-effective" constraint.
  • D is Incorrect: Building a custom Glue ETL + manual embedding pipeline introduces significant operational overhead compared to Bedrock KB's native incremental ingestion.
  • E is Incorrect: Bedrock Knowledge Bases does not have an IncrementalSyncEnabled auto-detect feature. An IngestionJob must be explicitly started via API call or console action for each sync.

Key Takeaway: Bedrock KB ingestion resync is naturally incremental—unchanged documents are skipped—but the job must be triggered externally (e.g., S3 events + Lambda) since Bedrock doesn't automatically detect S3 changes.

Question 6 – Model Versioning with Lineage Audit

A financial services company must comply with regulations requiring complete audit trails for any model that serves customer-facing decisions. Auditors need to know: (1) which model version served traffic on a specific date, (2) which training run produced that model, and (3) which dataset version was used in that training run. The team uses SageMaker Pipelines and MLflow on SageMaker AI.

Which approach establishes this audit trail? (Select ONE)

  • A) Configure the SageMaker Pipeline to run training jobs, register successful models to a SageMaker Model Registry group, and tag each registry entry with the MLflow experiment run ID that generated the model, allowing full lineage tracking.
  • B) Store training artifacts in S3 with a date-prefix naming convention (e.g., s3://bucket/2024-07-15/model.tar.gz) and rely on the model file's timestamp for auditing purposes.
  • C) Use CodeCommit to version control the training code, enabling auditors to reconstruct which model was deployed by reviewing the code repository history.
  • D) Enable CloudTrail on the endpoint and assume that any inference request that occurred on a specific date can be traced back to the exact model configuration in use because CloudTrail logs include model registry references.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: The Model Registry provides immutable model versions with lineage metadata. Linking registry entries to MLflow run IDs chains the complete path: Model Registry version → MLflow experiment run → training dataset used, allowing auditors to answer all three questions.

Distractor Analysis

  • B is Incorrect: S3 file timestamps indicate when the artifact was stored, not which model was actively serving traffic on a specific date, nor which dataset was used in training. This approach provides no deployment history.
  • C is Incorrect: Code versioning alone cannot tell auditors which specific model artifact was deployed, because the same code can be trained with different data or hyperparameters, producing different model versions.
  • D is Incorrect: CloudTrail records API calls and management events but does not capture model deployment lineage or dataset lineage for individual inference requests. This option describes functionality that doesn't exist.

Key Takeaway: A proper audit trail chains Model Registry → training pipeline execution → experiment run → dataset version, and SageMaker Model Registry + MLflow integration provides this end-to-end lineage automatically.

Question 7 – Prompt Versioning and Testing

A marketing agency is building a content generation application using Amazon Bedrock with Claude 3.5 Sonnet. They have 10 different prompt templates (prompt engineering variants) and need to improve response quality over time. They require: (1) version control for prompts to enable A/B testing, (2) ability to roll back prompt changes quickly, and (3) automated regression testing comparing prompt versions against a standard evaluation dataset before deployment.

Which TWO AWS services/features achieve this? (Select TWO)

  • A) Store prompts as versioned resources in Amazon Bedrock Prompt Management, using variant comparison to evaluate performance differences between prompt versions.
  • B) Commit prompt templates into a CodeCommit repository and use CodePipeline to deploy updated prompts via Bedrock's UpdatePrompt API, with a CodeBuild step running automated prompt evaluation.
  • C) Embed prompts directly in the Lambda function code and use Lambda versioning to manage prompt deployments.
  • D) Use MLflow on SageMaker AI to store prompts as logged parameters (tracked experiments) and register the best-performing prompt as a model registry artifact.
  • E) Store prompts in an S3 bucket with CloudFront for low-latency prompt retrieval by the application at runtime.
Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: Amazon Bedrock Prompt Management natively supports prompts as versioned resources with variables. It provides variant comparison, allowing prompt engineers to test prompt versions side-by-side against the same metrics before releasing a specific version to production.
  • B is Correct: Version-controlling prompts in CodeCommit creates an audit trail, enables rollback through git revert, and CodePipeline can automate the deployment of prompt updates via Bedrock APIs. A CodeBuild step running automated evaluation against a golden prompt dataset serves as a regression gate.

Distractor Analysis

  • C is Incorrect: Embedding prompts in application code creates a tangled monolithic approach with no prompt-specific versioning, no A/B testing capability, and requires redeploying the application for any prompt change—violating separation of concerns.
  • D is Incorrect: MLflow tracks prompts as experimental parameters for ML experiments, not as production-deployable prompt artifacts with variant comparison or rollback capabilities. This option misuses MLflow for a purpose it doesn't serve.
  • E is Incorrect: Storing prompts in S3 without versioning orchestration or management lacks version comparison, rollback capability, or automated testing. This is essentially a static file storage approach with no prompt lifecycle management.

Key Takeaway: Amazon Bedrock Prompt Management is the native AWS service for versioning, testing, and releasing prompts—and combining it with CI/CD (CodeCommit/CodePipeline) provides complete prompt lifecycle automation.

Question 8 – FM Fine-Tuned Model Deployment Automation

An e-commerce company fine-tuned Amazon Bedrock Titan for product categorization. The team has completed training and has the fine-tuned model ready for deployment. They need automated deployment with model versioning, ability to roll back to a previous fine-tuned model quickly, and traceability of which fine-tuned model version served when. They want minimal operational overhead.

Which approach fulfills this requirement? (Select ONE)

  • A) Import the fine-tuned model into Amazon Bedrock, create a provisioned throughput with the specific model version ID, and manage deployment state via CodePipeline with version tracking in a Model Registry-like implementation.
  • B) Use SageMaker Model Registry to register the fine-tuned model artifact, then deploy it via SageMaker endpoint with blue/green guardrails, managing versions through standard model versioning.
  • C) Store the fine-tuned model's weights in an S3 bucket with a naming convention (e.g., model-v2.tar.gz) and update the application's API configuration to point to the new S3 path.
  • D) Keep the fine-tuned model in Bedrock's model inventory and maintain a configuration table in DynamoDB that manually maps dates to model version IDs.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: Amazon Bedrock stores fine-tuned models with unique model version identifiers. Deploying via Provisioned Throughput references a specific model version ID, enabling version-controlled deployment. CodePipeline can automate the deployment, and Bedrock's API infrastructure maintains the model version inventory for audit and rollback.

Distractor Analysis

  • B is Incorrect: SageMaker Model Registry cannot register Bedrock fine-tuned models. Bedrock Titan fine-tuned models are hosted within Amazon Bedrock's managed model infrastructure and are not exportable as SageMaker model artifacts. This option pairs incompatible AWS model management systems.
  • C is Incorrect: Storing model weights in S3 doesn't apply to Amazon Bedrock fine-tuned models, which reside in Bedrock's managed model inventory. This approach describes a custom deployment pattern inconsistent with the managed fine-tuning process.
  • D is Incorrect: Maintaining a manual configuration table is operational overhead—exactly what the team wants to minimize. It also lacks automated deployment capabilities, and a manual table is prone to human error.

Key Takeaway: Amazon Bedrock fine-tuned models live in Bedrock's managed inventory and are deployed by model version ID through Provisioned Throughput—not through standard SageMaker model deployment tooling.

Question 9 – Rollback Capability in Multimodel Pipelines

A fintech company has a multi-model AI system: a traditional fraud detection model (SageMaker endpoint), a LLM-powered chatbot for customer service (Bedrock), and a prompt template library. The system undergoes monthly coordinated updates. Senior leadership demands that in the event of degraded performance, the entire system can revert to the previous known-good state within 5 minutes without manual provisioning.

Which architecture pattern provides this capability? (Select ONE)

  • A) Maintain a complete version manifest (fraud model version in Model Registry, Bedrock agent alias with target version IDs, prompt version in Prompt Management) in CodeCommit, and use CodePipeline with a rollback stage that redeploys the previous manifest versions when triggered.
  • B) Enable Blue/Green deployment guardrails on all three components simultaneously (SageMaker endpoint, Bedrock alias, prompt store), with CloudWatch alarms triggering full system rollback.
  • C) Create nightly snapshots of all production configurations (endpoint weights, Bedrock alias mappings, Lambda environment variables) to S3, with a manual restore process documented in a runbook.
  • D) Use AWS Backup to create recovery points for all model endpoints and restore from backup in the event of performance degradation.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

  • A is Correct: A versioned manifest approach supports full-system state rollback. The Model Registry retains previous fraud model versions, Bedrock aliases can point to different model versions, and Prompt Management maintains prompt versions. CodePipeline with a rollback stage (triggered manually or by automation) can redeploy the previous manifest versions within the 5-minute window without manual provisioning.

Distractor Analysis

  • B is Incorrect: Deployment Guardrails (blue/green) apply only to SageMaker endpoint model deployments—they do not manage Bedrock aliases or prompt versioning. There is no "full system rollback" guardrail that spans all three components.
  • C is Incorrect: Manual restore processes are prone to human error and lack the 5-minute recovery guarantee. Snapshotting configurations alone doesn't ensure the snapshots include all necessary model artifacts (e.g., model containers, weights, prompt state).
  • D is Incorrect: AWS Backup primarily addresses data protection and EC2/EFS/EBS/RDS snapshots—it does not provide model version rollback for SageMaker endpoints, Bedrock managed services, or prompt management.

Key Takeaway: Full-system rollback for multi-model AI architectures requires a versioned manifest approach (Model Registry + Bedrock aliases + Prompt Management) coordinated through CI/CD, not a single deployment mechanism.

Question 10 – Agent Version Management

A company is building an Amazon Bedrock agent that uses a Lambda function for order processing. The agent has two components under development: (1) the agent's instruction prompts and tool configuration, and (2) the Lambda function's code. The team wants to release agent changes safely with version control for both components, and they need to roll back the agent if the new version introduces conversational errors.

Which combination achieves this? (Select TWO)

  • A) Use Bedrock's agent versioning to create a new agent version whenever instruction prompts or tool configurations change, with a production alias pointing to the currently active version.
  • B) Manage the Lambda function code in CodeCommit, deploy updates via CodePipeline, and maintain Lambda aliases for version tracking and rollback.
  • C) Edit the agent's instructions directly in the Bedrock console and immediately update the production alias to reflect the new configuration without creating a version.
  • D) Store the Lambda function code in an S3 bucket with different object keys per version, and configure the agent to use the correct key at runtime.
  • E) Configure Bedrock Agent Guardrails to detect conversational errors and automatically revert the agent to the previous version when errors are detected.
Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: Amazon Bedrock agents support versioning—each version captures the agent's instructions, tool configurations, and model selection at a point in time. Production aliases point to specific agent versions, enabling management of which version serves traffic and enabling quick rollback by updating the alias.
  • B is Correct: Lambda code should follow standard CI/CD practices: version control in CodeCommit, automated deployment through CodePipeline, and Lambda aliases for tracking and rolling back code changes. This ensures the agent's execution layer is also independently versioned and auditable.

Distractor Analysis

  • C is Incorrect: Editing instructions directly in the console without creating a version breaks auditability and prevents rollback. There's no version to revert to, and the "production alias" would still point to an older captured state—not the new instructions.
  • D is Incorrect: Storing Lambda code in S3 with manual key management is not a proper deployment strategy—this bypasses CI/CD, lacks automated deployment, and cannot be invoked by Lambda without external orchestration.
  • E is Incorrect: Bedrock Guardrails provides content filtering and safety checks but does not include automatic agent version rollback capabilities or conversational error detection triggers.

Key Takeaway: Agent versioning in Bedrock (for prompts/config) combined with Lambda aliases (for custom code) provides complete version control and rollback capability for agent-based AI systems.