Skip to content

Skill 1.1.1: Extract data from data sources

Question 1 - Automated Model Quality Gates in SageMaker Pipelines

A financial services company has built a credit risk model training pipeline using Amazon SageMaker Pipelines. The pipeline consists of preprocessing, training, model evaluation, and model registration steps. The ML engineering team wants to ensure that a newly trained model only gets registered if its area under the ROC curve (AUC) score exceeds 0.85 on a held-out validation dataset. Additionally, if the evaluation fails, the pipeline should automatically send a notification to the ML team via Amazon SNS and halt execution.

Which configuration should the engineer implement to meet these requirements with the least operational overhead?

  • A) Add a ConditionStep in SageMaker Pipelines that compares the evaluation metric against 0.85, register the model if it passes, and configure a PipelineExecutionFailure event rule in Amazon EventBridge to trigger the SNS notification on any pipeline failure.
  • B) Create a Lambda step after evaluation that manually checks the AUC value from the evaluation report, raises an exception if below 0.85, and includes a try-except block to publish to SNS before exiting.
  • C) Use AWS Step Functions to orchestrate the pipeline stages, adding a Choice state to evaluate the AUC metric and an SNS Publish state for failure notifications.
  • D) Configure the Model Registry to automatically reject models with AUC below 0.85 using a ModelPackageGroup validation policy, and set up an SNS topic as the failure target.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: SageMaker Pipelines provides a native ConditionStep designed specifically for gating model registration based on evaluation metrics. EventBridge integrates natively with SageMaker Pipelines and can route PipelineExecutionFailed events directly to an SNS topic without custom code.

Distractor Analysis

  • B is Incorrect: While possible, writing a custom Lambda function with manual exception handling and SNS publishing introduces unnecessary operational overhead. SageMaker Pipelines already provides native condition steps and EventBridge integration.
  • C is Incorrect: Migrating the entire pipeline to Step Functions is unnecessary—SageMaker Pipelines is purpose-built for ML workflows and already includes condition evaluation capabilities.
  • D is Incorrect: The Model Registry does not have an automated "reject on metric threshold" validation policy. The registry catalogs and tracks approval status, but the conditional logic belongs in the pipeline itself.

Key Takeaway: Use SageMaker Pipelines' native ConditionStep for metric-based gating, and EventBridge rules for automated failure notifications—avoid custom Lambda glue code when native integrations exist.

Question 2 - Testing Prompt Variants in a Bedrock CI/CD Pipeline

An e-commerce company manages a customer support chatbot powered by Amazon Bedrock using Claude 3 models. The ML team has stored multiple prompt templates in Amazon Bedrock Prompt Management and wants to implement a CI/CD pipeline that tests two competing prompt variants against a curated set of 50 customer query examples. The evaluation must assess semantic similarity, factual accuracy, and tone appropriateness without writing custom evaluation algorithms.

Which AWS service should the team integrate into their CodePipeline stage to automate this prompt evaluation with minimal custom code?

  • A) Amazon Bedrock Model Evaluation with automatic prompt comparison mode
  • B) A custom AWS Lambda function that calls the Bedrock InvokeModel API for each prompt variant and computes BLEU, ROUGE, and BERTScore using a bundled Python library
  • C) Amazon SageMaker Clarify with auto-generated evaluation prompts for LLM outputs
  • D) AWS CodeBuild running a shell script that scores responses using pre-installed npm packages for text analysis
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Amazon Bedrock Model Evaluation supports automatic evaluation of model responses using built-in metrics, including semantic similarity, factual accuracy, and tone. It also allows comparing prompt variants against a curated dataset without writing custom evaluation code.

Distractor Analysis

  • B is Incorrect: Bundling evaluation libraries (BLEU, ROUGE, BERTScore) into a Lambda function requires significant custom code, dependency management, and operational overhead—violating the "minimal custom code" constraint.
  • C is Incorrect: SageMaker Clarify is designed for model explainability, bias detection, and LLM evaluation on SageMaker-hosted endpoints, not for native prompt variant testing on Bedrock models with automatic semantic metrics.
  • D is Incorrect: Using npm text analysis packages in CodeBuild does not provide semantic similarity, factual accuracy, or tone evaluation—these require LLM-as-judge capabilities or specialized language models, not generic text libraries.

Key Takeaway: For evaluating generative AI responses in Bedrock pipelines, use Amazon Bedrock Model Evaluation—it natively handles non-deterministic outputs with semantic scoring, avoiding brittle exact-match assertions or custom NLP libraries.

Question 3 - Rollback Configuration for SageMaker Endpoint Deployment

A healthcare analytics company is deploying a new version of a patient readmission prediction model to a SageMaker real-time endpoint. The current endpoint has been serving production traffic reliably for six months. The team wants to use SageMaker Deployment Guardrails to shift 25% of traffic to the new model version first, monitor it for errors during a 30-minute baking period, and automatically route all traffic back to the original model if the Invocation5XXErrors metric exceeds 2%.

Which configuration achieves the automatic rollback requirement?

  • A) Configure the endpoint with blue/green deployment, set TrafficRoutingConfig with canary 25% and baking period 30 minutes, and attach a CloudWatch Alarm on Invocation5XXErrors > 2% to the AutoRollbackConfig.
  • B) Configure the endpoint with rolling deployment, set the instance update batch size to 25%, and create a CloudWatch Alarm that triggers an AWS Lambda function to revert the endpoint configuration.
  • C) Use AWS CodeDeploy with a canary deployment strategy, setting the canary percentage to 25% and configuring automatic rollback on CloudWatch alarm breach.
  • D) Deploy the new model to a separate endpoint with a small instance count, manually monitor the error rate for 30 minutes, and update the application routing if the error rate stays below 2%.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: SageMaker Deployment Guardrails natively supports canary traffic routing with automatic rollback. The AutoRollbackConfig must reference a CloudWatch Alarm that watches the specified metric (Invocation5XXErrors) during the baking period. If the alarm trips, SageMaker automatically reverts traffic to the blue fleet.

Distractor Analysis

  • B is Incorrect: Rolling deployment is designed for updating instances within a single fleet (e.g., changing instance types), not for blue/green model version rollouts. Additionally, a custom Lambda to revert the endpoint adds unnecessary complexity when Deployment Guardrails handles rollback natively.
  • C is Incorrect: AWS CodeDeploy does not support SageMaker real-time inference endpoints as deployment targets. It is designed for EC2, Lambda, ECS, and on-premises deployments.
  • D is Incorrect: Manual monitoring and application-level routing introduces significant operational overhead, removes automatic rollback capability, and requires custom application routing logic.

Key Takeaway: For SageMaker model version deployments, use Deployment Guardrails with AutoRollbackConfig tied to CloudWatch Alarms—never build custom rollback scripts or use CodeDeploy for ML endpoint traffic shifting.

Question 4 - Event-Driven Retraining Pipeline for Irregular Data Uploads

An IoT manufacturing company streams sensor data from factory equipment into Amazon S3 in batches that arrive irregularly—sometimes daily, sometimes weekly. The ML team has built a SageMaker Pipeline that preprocesses new labeled data, retrains a predictive maintenance model, evaluates it, and registers the model in the SageMaker Model Registry. The team wants retraining to start automatically when new labeled data arrives, and the pipeline should not execute when no new data is available.

Which TWO services should the team configure to implement this event-driven retraining? (Select TWO)

  • A) Configure an S3 Event Notification for s3:ObjectCreated:* on the labeled data bucket/prefix.
  • B) Create an Amazon EventBridge rule that matches the S3 event and invokes StartPipelineExecution on the SageMaker Pipeline.
  • C) Schedule the pipeline using an EventBridge cron rule that runs every 15 minutes to check for new data in S3.
  • D) Create an AWS Lambda function that polls the S3 bucket every few minutes and starts the pipeline when new data is detected.
  • E) Configure SageMaker Model Monitor to detect new data in S3 and trigger the training pipeline automatically.
Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: S3 Event Notifications emit events whenever objects are created in the specified bucket/prefix. This triggers immediate detection of new labeled data without polling overhead.
  • B is Correct: EventBridge natively integrates with S3 Event Notifications and can directly invoke SageMaker Pipeline executions via the StartPipelineExecution API, creating a fully serverless, event-driven trigger without custom Lambda code.

Distractor Analysis

  • C is Incorrect: A 15-minute cron schedule wastes compute by checking even when no data has arrived, and may introduce unnecessary delays (up to 15 minutes) after data lands. It also requires additional logic to check for new data.
  • D is Incorrect: Polling S3 with a Lambda function introduces custom code, additional operational overhead, and unnecessary Lambda invocation costs, violating the "automated with least operational overhead" principle.
  • E is Incorrect: SageMaker Model Monitor is designed for detecting data drift and model quality degradation in running endpoints, not for detecting new training data in S3 buckets or triggering pipeline executions.

Key Takeaway: For event-driven retraining, use S3 Event Notifications → EventBridge → SageMaker Pipelines—this is a fully serverless, zero-polling pattern that executes only when data actually arrives.

Question 5 - Pipeline Testing Configuration for Traditional ML Models

A retail company uses AWS CodePipeline with AWS CodeBuild to manage their machine learning deployment pipeline. The pipeline currently does the following:

  1. Builds and validates the application container
  2. Runs unit tests on application code
  3. Trains a recommendation model via SageMaker Training Job
  4. Packages the model artifact

The ML team wants to add a model quality gate that runs after training and before model registration, evaluating the model's performance on a held-out dataset with threshold checks (RMSE < 0.5, Precision@10 > 0.7). If thresholds fail, the pipeline must stop and prevent registration.

Where should the team add this evaluation step?

  • A) Add a SageMaker Processing Job step in the pipeline that runs the evaluation script, computes metrics, and integrate a ConditionStep to gate the Model Registry step based on metric thresholds.
  • B) Add a CodeBuild unit test that asserts the evaluation metrics are within thresholds before the training job starts.
  • C) Add a pre-build validation script in CodeBuild that estimates model quality based on training data statistics, rather than running evaluation on held-out data.
  • D) Perform evaluation manually in the SageMaker Model Registry after registration and update the approval status based on metrics.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: A SageMaker Processing Job is the correct place to run model evaluation on held-out data after training. The ConditionStep natively compares evaluation metrics against thresholds and gates the subsequent Model Registry step, ensuring only quality models are registered.

Distractor Analysis

  • B is Incorrect: Running unit tests before training only tests code logic, not model quality. Model evaluation requires actual trained model outputs on held-out data, which only exists after the training job completes.
  • C is Incorrect: Estimating quality from training data statistics without running actual evaluation on held-out data does not provide real model performance metrics (RMSE, Precision@10) and cannot accurately gate model quality.
  • D is Incorrect: Manual evaluation after registration means the model is already registered in the registry, violating the requirement to prevent registration when thresholds fail. Approval status alone does not prevent the model from being registered.

Key Takeaway: Model evaluation belongs in a dedicated processing step after training, gated by a ConditionStep that compares held-out metrics against thresholds—never rely on code unit tests or manual post-registration reviews for model quality.

Question 6 - Experiment Tracking and Model Versioning in CI/CD Workflow

A data science team at a logistics company regularly experiments with different hyperparameters for a delivery time prediction model using SageMaker Training Jobs. They want every successful training run to be automatically logged with its hyperparameters, metrics, and input data location. When a training run achieves the best validation loss among all experiments, the model should be promoted to SageMaker Model Registry with an approval status of "PendingManualApproval" for the ML lead to review.

Which AWS service combination should the team use to achieve this with minimal manual tracking overhead?

  • A) Use MLflow on Amazon SageMaker to track experiment runs, configure auto-registration of the best performing run into the Model Registry, and set the initial approval status to "PendingManualApproval" in the registry.
  • B) Create a DynamoDB table to manually log hyperparameters and metrics after each training job, and use a Lambda function to identify the best run and register it in the Model Registry.
  • C) Store experiment metadata as JSON files in S3, have team members update a shared spreadsheet with results, and manually create a model package in the registry when a good model is found.
  • D) Use Amazon CloudWatch Logs to capture training job logs, set up a log metric filter to identify the best run, and configure SageMaker Model Monitor to automatically promote the best model.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: MLflow on Amazon SageMaker provides managed experiment tracking that automatically logs parameters, metrics, and artifacts from training runs. It can auto-register the best performing model (based on a specified metric like validation loss) directly into the SageMaker Model Registry with configurable approval status. This creates a seamless lineage from experiment to registered model with minimal manual effort.

Distractor Analysis

  • B is Incorrect: Building a custom DynamoDB logging system and Lambda-based best-run identification introduces significant operational overhead and custom code. MLflow on SageMaker already provides this functionality natively.
  • C is Incorrect: Manual JSON file storage and spreadsheet tracking defeats the purpose of automated MLOps, creates versioning challenges, and does not provide any lineage trail from experiment to registry.
  • D is Incorrect: CloudWatch Logs capture execution logs but do not provide structured experiment tracking, metric comparison, or best-run selection. SageMaker Model Monitor detects runtime drift and quality issues on deployed models, not training-run promotion.

Key Takeaway: Use MLflow on SageMaker for experiment tracking and automatic model registration—it provides native lineage from training runs to the Model Registry without custom tracking systems.

Question 7 - CI/CD Pipeline Source Configuration for External Git Repository

A startup's ML engineering team maintains their code in a GitHub repository (hosted outside AWS). They are setting up AWS CodePipeline to automate their ML model deployment process. The security team has mandated that no credentials or personal access tokens be stored directly in the pipeline definition, CloudFormation templates, or environment variables.

Which AWS service should the team use to connect CodePipeline to the GitHub repository?

  • A) AWS CodeConnections (formerly CodeStar Connections)
  • B) AWS CodeCommit as a mirror of the GitHub repository
  • C) AWS Secrets Manager to store the GitHub PAT and reference it from CodePipeline
  • D) AWS CodeBuild with a webhook listener for GitHub events
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: AWS CodeConnections provides a managed, secure integration between AWS services (CodePipeline/CodeBuild) and external Git providers (GitHub, GitLab, Bitbucket). It handles OAuth/App-based authentication during the connection setup, storing the credential securely in AWS rather than requiring PATs in pipeline definitions or templates.

Distractor Analysis

  • B is Incorrect: Mirroring to CodeCommit requires a separate synchronization mechanism (e.g., Lambda function or scheduled push) to keep the mirror updated with GitHub, introducing operational complexity and potential sync lag between code changes and pipeline execution.
  • C is Incorrect: While Secrets Manager securely stores credentials, the requirement specifically prohibits credentials stored in pipeline configurations. This approach requires the pipeline to retrieve and manage the secret, adding unnecessary steps and still involving static credential management.
  • D is Incorrect: Setting up a custom webhook listener in CodeBuild still requires configuring the GitHub webhook with authentication credentials, and CodeBuild itself is a build execution service, not a repository connection management service.

Key Takeaway: Use AWS CodeConnections for integrating external Git repositories with CodePipeline—it provides secure OAuth/App-based authentication without ever placing credentials in pipeline code or templates.

Question 8 - Automated Testing Strategy for SageMaker Pipelines

A financial technology company is building a SageMaker Pipeline for automated loan default prediction. The pipeline:

  1. Processes raw data from S3 using SageMaker Processing
  2. Trains an XGBoost model
  3. Evaluates the model
  4. Registers the model in Model Registry

The ML team wants to ensure that the input data quality is validated before training begins. Specifically, they need to check for missing values exceeding 10% in any column, class imbalance of 80/20 or worse, and dataset size below 10,000 rows. If any of these conditions are met, the pipeline should fail immediately without starting the training step.

Where should these data validation steps be implemented?

  • A) Add a data validation step using SageMaker Processing at the beginning of the pipeline, with a script that checks all three conditions and raises an exception on failure, causing the pipeline to halt before training.
  • B) Configure SageMaker Clarify to analyze the dataset before training and automatically block the training job if data quality issues are detected.
  • C) Add data quality checks as unit tests in the CodeBuild stage that runs before the SageMaker Pipeline is triggered.
  • D) Configure Amazon DataZone to automatically validate data quality rules before the pipeline consumes the dataset.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: A SageMaker Processing step at the start of the pipeline can run custom validation logic on the raw data, checking all specified conditions. When the processing script encounters data quality violations, it raises an exception, causing the step to fail and the pipeline to halt before proceeding to training.

Distractor Analysis

  • B is Incorrect: SageMaker Clarify is designed for model explainability, bias detection, and fairness analysis—not for raw data validation. It analyzes trained models' predictions, not input data quality properties like missing values or dataset size.
  • C is Incorrect: Unit tests in CodeBuild run before the pipeline starts and validate code correctness, not the actual data content in S3. Data properties like missing value percentage and class imbalance can only be measured on the actual dataset, which may change between pipeline runs.
  • D is Incorrect: Amazon DataZone is a data cataloging and governance service, not a data quality validation service. It helps discover and catalog data but does not automatically enforce custom quality rules before pipeline execution.

Key Takeaway: Data validation belongs as a dedicated SageMaker Processing step within the pipeline, not as unit tests or catalog solutions—this ensures every pipeline run validates the actual input data before training begins.

Question 9 - Retrieval-Augmented Generation (RAG) Pipeline Refresh Strategy

A legal research platform uses Amazon Bedrock Knowledge Bases with Amazon OpenSearch Serverless to store vector embeddings of legal documents. The document corpus contains approximately 5 million documents. Recently, 50 documents were updated with new case rulings, and 3 documents were deleted. The platform wants to keep their vector index current without incurring unnecessary embedding costs.

Which strategy should the team implement?

  • A) Trigger a Bedrock Knowledge Base ingestion resync (StartIngestionJob) that incrementally processes only the 50 modified and 3 deleted documents, updating or removing their respective chunks and embeddings.
  • B) Delete the entire vector collection in OpenSearch Serverless and run a full ingestion job to rebuild all 5 million embeddings from scratch.
  • C) Create a new OpenSearch Serverless collection, copy over unchanged vectors, and manually update the 50 changed document vectors using a custom embedding pipeline.
  • D) Schedule a full ingestion job that processes every document in the corpus every night to ensure the index remains up to date.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Bedrock Knowledge Bases ingestion resync performs incremental synchronization—it identifies documents that have been added, modified, or deleted since the last ingestion and processes only those changes. This minimizes embedding API calls (processing only 53 documents instead of 5 million) and keeps the vector index current at minimal cost.

Distractor Analysis

  • B is Incorrect: Deleting and rebuilding the entire index requires re-embedding all 5 million documents, incurring massive embedding API costs, significant time, and unnecessary operational risk—violating the "unnecessary embedding costs" constraint.
  • C is Incorrect: Manually copying vectors and updating 50 documents with a custom embedding pipeline introduces significant operational complexity. The existing OpenSearch Serverless collection does not need replacement, and custom code is unnecessary since Bedrock Knowledge Bases handles incremental updates natively.
  • D is Incorrect: Full nightly ingestion processes all 5 million documents regardless of changes. While the ingestion engine may skip unchanged documents in some cases, relying on full ingestion jobs still evaluates the entire corpus and can incur unnecessary processing overhead compared to an incremental resync.

Key Takeaway: Bedrock Knowledge Bases ingestion resync is designed for incremental updates—it processes only changed documents (additions, modifications, deletions), making frequent refreshes cost-effective even for large corpora.

Question 10 - Model Registry Approval Status Gating

A healthcare AI company has built a SageMaker Pipeline that automatically retrains a medical imaging model monthly. The pipeline trains the model, evaluates it, and registers it in the SageMaker Model Registry with an approval status of "PendingManualApproval." The deployment team wants to ensure that only manually approved model versions are deployed to production endpoints by a separate deployment pipeline in AWS CodePipeline.

What is the correct deployment step configuration for the deployment pipeline?

  • A) Use a Lambda function in the deployment pipeline that queries the Model Registry for the latest approved model version (status = "Approved"), retrieves the model artifact S3 URI, and configures the SageMaker endpoint with that model version.
  • B) Configure the deployment pipeline to automatically deploy the most recently registered model regardless of approval status, with a manual check after deployment.
  • C) Configure the deployment pipeline to use the arn of the latest model package directly, ignoring the registry approval status since the training pipeline already ensures quality.
  • D) Use AWS CodeDeploy with an approval gate that requires manual intervention before proceeding with the deployment.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: A Lambda function in the deployment pipeline that queries the Model Registry for the latest "Approved" status model package ensures that only manually vetted model versions are deployed. This creates a proper human-in-the-loop approval gate between training and deployment, using the registry as the source of truth for what is deployable.

Distractor Analysis

  • B is Incorrect: Deploying the latest registered model regardless of approval status bypasses the manual review requirement entirely. A model may be registered but pending approval, and this configuration would push it to production without human sign-off.
  • C is Incorrect: Using the latest model package ARN directly ignores the registry approval status altogether, violating the requirement for manual approval before deployment.
  • D is Incorrect: AWS CodeDeploy does not support SageMaker real-time inference endpoints as deployment targets. Manual approval in CodePipeline is also not the same as Model Registry approval—the requirement is to check the registry's approval status, not add a separate pipeline approval gate.

Key Takeaway: The SageMaker Model Registry uses approval status as the gating mechanism for production deployments—deployment pipelines should query for approved versions only, ensuring human oversight without embedding approval logic elsewhere.