Skip to content

Skill 1.1.1: Extract data from data sources

Question 1 - SageMaker Deployment Guardrails Rollback Configuration

A financial services company is deploying a fraud detection model to a SageMaker endpoint using deployment guardrails. The model must be released gradually with an automatic rollback if model latency exceeds acceptable thresholds during the initial deployment phase. The operations team insists on minimizing manual intervention during the deployment.

Which configuration should the ML engineer implement to meet these requirements? (Select ONE)

  • A) Configure a blue/green deployment with a linear traffic shifting strategy and attach an Amazon CloudWatch alarm on ModelLatency to the deployment guardrails' AutoRollbackConfig with a defined baking period.
  • B) Configure a blue/green deployment with an all-at-once traffic shifting strategy and enable SageMaker Model Monitor to detect data drift after deployment.
  • C) Deploy using SageMaker multi-model endpoints and manually monitor CloudWatch dashboards after the shift completes.
  • D) Configure a canary traffic shifting strategy without CloudWatch alarms, but set a longer baking period to allow manual inspection.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: SageMaker Deployment Guardrails support linear traffic shifting, which moves traffic in even increments (e.g., 10% at a time) to the new fleet. Attaching a CloudWatch alarm on ModelLatency to AutoRollbackConfig ensures that if latency thresholds are breached during the baking period, SageMaker automatically reverts traffic to the blue fleet—satisfying the requirement for minimal manual intervention.

Distractor Analysis

  • B is Incorrect: All-at-once traffic shifting sends 100% of traffic to the new fleet immediately, providing no gradual rollout. Model Monitor detects drift post-deployment but does not trigger automatic rollback during the baking period.
  • C is Incorrect: Multi-model endpoints are designed for hosting multiple models on shared infrastructure, not for blue/green deployment guardrails. Manual monitoring adds operational overhead and violates the "minimize manual intervention" constraint.
  • D is Incorrect: Canary shifting without an attached CloudWatch alarm means SageMaker has no signal to trigger a rollback. A longer baking period merely delays the inevitable bad deployment completion without protection.

Key Takeaway: SageMaker Deployment Guardrails require a CloudWatch alarm linked to AutoRollbackConfig for automated rollback; the traffic shifting strategy (canary/linear/all-at-once) controls rollout shape, but only alarms provide the rollback signal.

Question 2 - Event-Driven Retraining Pipeline

A retail company has an Amazon SageMaker Pipeline for training a product recommendation model. The model must be re-trained only when new labeled clickstream data arrives in an S3 bucket, which occurs irregularly throughout the day. The ML team wants the solution to integrate with their existing CI/CD workflow and avoid unnecessary training costs.

Which architecture satisfies these requirements? (Select ONE)

  • A) Configure an Amazon EventBridge rule that listens for S3 ObjectCreated events and invokes StartPipelineExecution on the SageMaker Pipeline. The pipeline ends by registering the model in the SageMaker Model Registry with PendingManualApproval status.
  • B) Schedule the SageMaker Pipeline to run hourly using an EventBridge cron rule and check for new data at the start of each run.
  • C) Create an AWS Lambda function that runs every 10 minutes to list the S3 bucket, compare file timestamps, and manually start the training job if new data is found.
  • D) Configure SageMaker Model Monitor on the existing endpoint and trigger a training job when data drift is detected.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: An event-driven architecture using EventBridge rules is the most efficient way to trigger retraining on irregular data arrivals. S3 ObjectCreated events are delivered to EventBridge, which invokes StartPipelineExecution directly. The pipeline ending at the Model Registry with approval status separates training from deployment, allowing a human gate before production traffic—a core MLOps best practice.

Distractor Analysis

  • B is Incorrect: Hourly scheduling wastes compute when no data has arrived and introduces latency when data arrives immediately after a scheduled run. The requirement specifies retraining only when new data arrives.
  • C is Incorrect: A Lambda function polling S3 every 10 minutes adds custom code and operational overhead. It also violates the "automated event-driven" principle by introducing latency and unnecessary polling costs.
  • D is Incorrect: Model Monitor detects data drift on live endpoints, not the arrival of new labeled data in S3. Moreover, data drift indicates distribution changes, not necessarily the availability of new training labels.

Key Takeaway: Event-driven triggers (EventBridge + S3 events) are ideal for irregular retraining patterns, and pipelines should end at registry registration—not automatic deployment—to maintain a human or test gate.

Question 3 - Generative AI Testing in CI/CD

A startup is building a customer support chatbot using Amazon Bedrock and a foundation model. Before any prompt or model configuration change reaches production, the team must ensure response quality does not degrade. Responses vary between runs for identical prompts due to model non-determinism. The ML team wants automated gates in their CodePipeline to compare new versions against the baseline.

Which testing approach should the team implement? (Select ONE)

  • A) Run a curated golden prompt dataset through both the baseline and candidate versions, score responses using LLM-as-a-judge metrics (e.g., faithfulness, relevance), and block deployment if the candidate score falls below baseline thresholds.
  • B) Write unit tests that assert the exact string response for each golden prompt matches the expected output stored in CodeCommit.
  • C) Perform A/B testing in production by deploying both versions to separate endpoints and manually reviewing customer feedback.
  • D) Monitor endpoint latency and invocation counts during a canary deployment, and rollback if latency exceeds 500ms.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Generative AI responses are non-deterministic, so exact-match assertions are inappropriate. Running a golden prompt dataset through both model versions and scoring responses with LLM-as-a-judge or model evaluation metrics (faithfulness, answer relevance, toxicity) provides a quantitative quality gate. This can be implemented in a CodeBuild stage that blocks pipeline progression if scores fall below a baseline—exactly fitting automated CI/CD requirements.

Distractor Analysis

  • B is Incorrect: Exact string assertions for generative AI will fail almost every time due to natural response variation, causing false negatives and flaky pipelines.
  • C is Incorrect: A/B testing in production still exposes real customers to potentially degraded responses before quality is confirmed. Automated pre-production evaluation gates are required here.
  • D is Incorrect: Monitoring latency during canary deployment checks infrastructure health, not semantic response quality. High latency does not indicate whether responses are accurate or helpful.

Key Takeaway: Generative AI testing requires evaluation-based gates (golden prompt datasets + LLM-as-a-judge scoring) rather than deterministic assertions, because model output varies between runs.

Question 4 - Model Registry Lineage and Audit

A healthcare AI company must comply with strict audit requirements. Regulators need to know exactly which model version served production traffic during a past incident and which training run produced that model, including hyperparameters, dataset location, and evaluation metrics.

Which solution provides the most complete audit trail with the least operational overhead? (Select ONE)

  • A) Use Amazon SageMaker Model Registry to register each model version with automatic lineage tracking from the SageMaker Pipeline run, linking the model package to the training job, dataset URI, and hyperparameters.
  • B) Store model artifacts in S3 and maintain a spreadsheet mapping each endpoint deployment to artifacts and training parameters.
  • C) Tag the container image used for inference with the training job ID and model hash, then query Amazon ECR during an audit.
  • D) Enable CloudTrail logging alone to capture endpoint invocations and training job executions.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: The SageMaker Model Registry provides native lineage tracking that automatically links a registered model version to the SageMaker Pipeline execution, training job, input dataset S3 URIs, hyperparameters, and evaluation metrics. This creates an immutable audit trail that directly answers "what served when" and "what produced it"—with zero custom infrastructure.

Distractor Analysis

  • B is Incorrect: A manual spreadsheet is error-prone, does not scale, and violates the "least operational overhead" principle for production ML systems. Manual tracking cannot guarantee consistency or completeness.
  • C is Incorrect: ECR image tags only capture serving environment metadata, not the specific training run, dataset, or hyperparameters that produced model weights.
  • D is Incorrect: CloudTrail captures API activity but does not link model artifacts to training lineage, hyperparameters, or evaluation metrics. It provides "who called what API" but not "which model version served traffic."

Key Takeaway: SageMaker Model Registry + SageMaker Pipelines provides automatic, immutable lineage tracking for model auditability without manual bookkeeping.

Question 5 - Prompt Version Management

A product team is building a content generation application on Amazon Bedrock. They frequently iterate on prompt wording (e.g., tone, structure) and need to test variations side-by-side, release a specific prompt version to production, and rollback if quality degrades. The application must reference prompts by immutable identifiers, not hardcoded strings in the codebase.

Which solution meets these requirements? (Select ONE)

  • A) Use Amazon Bedrock Prompt Management to create versioned prompts with variables and comparable variants. The application references the prompt by its ARN/version ID, and CI/CD gates deployment based on evaluation scores of the new prompt variant.
  • B) Store prompts in a DynamoDB table keyed by a UUID and have the application query the table at runtime using the latest timestamp.
  • C) Embed prompt strings directly in the application code and tag releases in Git with PROMPT_VERSION.
  • D) Use AWS Secrets Manager to store prompt text as encrypted secrets and rotate them when prompts change.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Amazon Bedrock Prompt Management provides native prompt versioning with immutable ARNs/version IDs, supports variables and comparable variants, and integrates with CI/CD workflows so teams can test variants and promote winners to production. This satisfies auditability, rollback capability, and "reference by identifier" requirements.

Distractor Analysis

  • B is Incorrect: DynamoDB keyed by UUID provides storage but not native versioning, variant comparison, or immutable release identifiers. The application would need custom version logic, increasing operational complexity.
  • C is Incorrect: Embedding prompts in code requires code changes and redeployments for every prompt iteration. Git tags do not provide runtime prompt version management or side-by-side variant testing.
  • D is Incorrect: Secrets Manager is designed for storing credentials and API keys, not prompt text versioning. It lacks version comparison, variant testing, and prompt-specific features.

Key Takeaway: Amazon Bedrock Prompt Management provides versioned, testable, and comparable prompt artifacts that applications reference by ARN—essential MLOps practice for generative AI workloads.

Question 6 - Bedrock Knowledge Base Incremental Refresh

A legal document search application uses Amazon Bedrock Knowledge Bases with an S3 data source and OpenSearch Serverless vector store. The S3 bucket contains 10,000 legal documents. Legal teams frequently edit 20-30 documents daily. The ML team wants the knowledge base to stay current without incurring full reindexing costs every day.

Which orchestration approach is the most cost-effective and ensures the vector index reflects all document changes? (Select ONE)

  • A) Configure an EventBridge rule that triggers StartIngestionJob on the Bedrock Knowledge Base when S3 PutObject or DeleteObject events occur. Bedrock performs an incremental sync, processing only added, modified, or deleted documents.
  • B) Schedule a nightly full reindexing job that reprocesses all 10,000 documents to ensure consistency.
  • C) Create a Lambda function that copies the entire OpenSearch index to a new collection nightly and swaps the endpoint.
  • D) Use AWS Glue to detect changed files and load them into a new S3 bucket, then manually trigger full ingestion each week.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Bedrock Knowledge Bases' StartIngestionJob performs incremental synchronization by default, processing only new, modified, or deleted documents since the last sync. Triggering on S3 events ensures immediate reflection of legal document edits without recomputing embeddings for unchanged files—minimizing embedding costs and latency.

Distractor Analysis

  • B is Incorrect: Full nightly reindexing reprocesses all 10,000 documents, wasting substantial embedding compute and time when only 20-30 files change daily. This directly violates the "cost-effective" constraint.
  • C is Incorrect: Copying the OpenSearch index does not regenerate embeddings for changed documents—it merely duplicates stale vectors. This provides no actual refresh of content.
  • D is Incorrect: Using Glue for change detection adds unnecessary ETL infrastructure. Weekly manual triggers introduce latency for legal updates and add operational overhead.

Key Takeaway: Bedrock Knowledge Bases' incremental ingestion (StartIngestionJob) processes only documents added, changed, or removed since the last run—making event-driven refresh both cost-effective and current.

Question 7 - CodeBuild and Model Evaluation Integration

An ML team is building a CodePipeline that trains a classification model using SageMaker Pipelines, evaluates it against a precision threshold, and deploys it to a SageMaker endpoint only if the threshold is met. If evaluation fails, the pipeline must stop and notify the team.

Which configuration correctly implements these deployment gates? (Select ONE)

  • A) Add a CodeBuild stage after the SageMaker Pipeline step. This stage runs sagemaker:RegisterModel only if the captured evaluation metric meets the threshold; otherwise, the CodeBuild step exits with a non-zero status, causing CodePipeline to fail and trigger an SNS notification.
  • B) In the SageMaker Pipeline, train the model and automatically call CreateEndpoint in the same step, then rely on Model Monitor to catch quality issues post-deployment.
  • C) Deploy all models to a staging endpoint and have a Data Scientist manually approve based on a notebook review.
  • D) Configure SageMaker Deployment Guardrails to rollback if the training accuracy (not evaluation metric) falls below threshold.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: CodePipeline stages can include CodeBuild actions that execute evaluation scripts. If the model fails the precision threshold on the held-out set, the CodeBuild action returns a non-zero exit code, halting the pipeline. SNS notification on pipeline failure meets the notification requirement. This correctly implements an automated quality gate before deployment.

Distractor Analysis

  • B is Incorrect: Deploying and then monitoring post-deployment means bad models reach production traffic before being caught. The requirement specifies evaluation must happen before deployment.
  • C is Incorrect: Manual approval for every model iteration introduces human latency and operational overhead. The pipeline must be fully automated except where the requirement mandates human intervention (which it does not here).
  • D is Incorrect: Deployment Guardrails use CloudWatch alarms on production endpoint metrics to trigger rollback, not evaluation metrics from a training pipeline. They also operate after deployment, not as a pre-deployment gate.

Key Takeaway: CodeBuild stages with exit-code-based gates are the standard mechanism for stopping CI/CD pipelines when model evaluation fails quality thresholds.

Question 8 - Training Job Configuration

A data science team needs to configure a SageMaker training job for a deep learning model that requires GPU acceleration. The training dataset is stored in S3 as TFRecord files. The team wants to minimize cost by using spot instances for training while ensuring the job can resume from checkpoints if interrupted.

Which TWO configurations should the team implement? (Select TWO)

  • A) Specify a GPU instance type (e.g., ml.p3.2xlarge) for the training job and enable Managed Spot Training with a checkpoint S3 path configured via the training script.
  • B) Set MaxRuntimeInSeconds to 1 hour to force the job to use Spot instances.
  • C) Enable automatic model tuning with Bayesian optimization to reduce training time.
  • D) Pass the S3 URI of the TFRecord dataset as the input data channel with TrainingInputMode set to File or Pipe.
  • E) Configure the training job to use AWS Inferentia instances to accelerate training.
Click to reveal Answer & Explanation

Correct Answers: A and D

Correct Options Analysis

  • A is Correct: GPU instances like ml.p3.2xlarge are designed for deep learning workloads. Managed Spot Training lets SageMaker use spare capacity at up to 70-90% discount. Checkpoints saved to S3 allow the job to resume from the last checkpoint when Spot capacity is reclaimed.
  • D is Correct: TFRecord files in S3 are accessed through SageMaker input data channels with TrainingInputMode set to File (downloads to local storage) or Pipe (streams data during training), which is the standard SageMaker pattern.

Distractor Analysis

  • B is Incorrect: MaxRuntimeInSeconds is a stopping condition that terminates training after the specified duration. It does not enable Spot instances—Spot is configured via EnableManagedSpotTraining in the training config.
  • C is Incorrect: Automatic model tuning (hyperparameter optimization) launches multiple training jobs to find optimal hyperparameters. It does not reduce a single training job's duration or enable Spot capacity.
  • E is Incorrect: AWS Inferentia instances (e.g., ml.inf1, ml.inf2) are designed for low-latency inference, not training. Training requires GPU (P/G series) or Trainium (ml.trn1) instances.

Key Takeaway: Managed Spot Training with S3 checkpoints provides cost savings and interruption resilience; SageMaker training jobs read data through input channels from S3.

Question 9 - CI/CD Pipeline for Fine-Tuned Foundation Model

A company fine-tunes foundation models using Amazon SageMaker and deploys them to Bedrock for inference. They want an automated pipeline that triggers when a model artifact is registered in the Model Registry, tests the fine-tuned model against evaluation datasets, and deploys only if quality passes. The deployment must be traceable to the specific fine-tuned version.

Which pipeline design achieves this with the least operational overhead? (Select ONE)

  • A) Use EventBridge to detect registry ModelApproved events, trigger a CodePipeline with stages: evaluate the model using CodeBuild (custom evaluation container), then invoke Bedrock CreateProvisionedModelThroughput or import the fine-tuned model with its version ID for deployment.
  • B) Manually copy the model artifact to an S3 bucket, then have the Data Science team run a Jupyter notebook to evaluate and deploy.
  • C) Use AWS Step Functions to chain Lambda functions that call InvokeEndpoint on a temporary endpoint for evaluation, then manually tag the model with version metadata.
  • D) Configure SageMaker Model Monitor to compare the fine-tuned model's performance against the base model on the existing production endpoint, then automatically route traffic if performance is better.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: EventBridge rules can trigger pipelines when models are approved in the Model Registry. CodePipeline with CodeBuild stages supports automated evaluation. Deploying the fine-tuned model to Bedrock with its version ID from the registry ensures the deployment is traceable to a specific version. This uses fully managed pipeline services with no custom orchestration code.

Distractor Analysis

  • B is Incorrect: Manual notebook evaluation and deployment introduces human latency, lacks automation, and creates no repeatable pipeline. This violates the "automated pipeline" and "least operational overhead" requirements.
  • C is Incorrect: While Step Functions can orchestrate some tasks, requiring a temporary endpoint for every evaluation is wasteful. Custom Lambda functions add operational overhead compared to CodePipeline's managed build/test stages.
  • D is Incorrect: Model Monitor is designed for post-deployment monitoring of live endpoints, not pre-deployment evaluation. Comparing performance on the production endpoint exposes customers to the untested model.

Key Takeaway: EventBridge + CodePipeline + Model Registry automation provides a fully managed, traceable pipeline for fine-tuned foundation model deployment to Bedrock.

Question 10 - Agent Version Management and Deployment

A company is building AI agents using Amazon Bedrock Agents. Each agent has configurations (system prompt, model selection, tools, knowledge bases). The team needs to test agent versions in staging before production, track which agent version is serving in production, and perform automated rollbacks if a new version performs poorly.

Which approach provides the most comprehensive agent version management? (Select ONE)

  • A) Use Amazon Bedrock Agent versioning to create immutable agent versions. Run automated evaluation tests in staging on new versions, then alias production to the tested version. Rollback involves re-pointing the production alias to the previous agent version.
  • B) Store agent configuration files in Git and manually apply changes to the production agent using the Bedrock console.
  • C) Create duplicate agents named agent-prod-v1, agent-prod-v2, and have the application select the agent based on a hardcoded ID.
  • D) Use AWS AppConfig to store agent configuration as feature flags and toggle at runtime based on user feedback.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Amazon Bedrock Agents support native agent versions (immutable snapshots) and aliases for routing traffic. New versions can be tested in staging environments, and production aliases can be pointed to tested versions. Rollback is a simple alias re-pointing—automated, traceable, and meets audit requirements with low operational overhead.

Distractor Analysis

  • B is Incorrect: Manual configuration changes in the Bedrock console create no immutable version trail and risk unintended drift. Git-only tracking does not integrate with Bedrock's native versioning for serving traffic.
  • C is Incorrect: Creating duplicate agents with naming conventions requires application code changes for every version switch. This fragments agent configuration management, lacks automated rollback, and creates configuration drift between versions.
  • D is Incorrect: AppConfig is designed for application configuration and feature flags, not agent lifecycle management. It does not provide agent-specific testing, deployment, or rollback capabilities natively.

Key Takeaway: Amazon Bedrock Agents' native versioning + aliases provide immutable, testable, and reversible agent deployments—the foundational pattern for MLOps with AI agents.