Skip to content

Skill 3.3.5: Build and integrate mechanisms to re-train models

Question 1 - Event-Driven Retraining Pipeline

A financial services company has deployed a fraud detection model on Amazon SageMaker that must be retrained when new labeled transaction data arrives in an Amazon S3 bucket. The data arrives irregularly—sometimes multiple times per day, sometimes once per week. The ML engineering team wants the retraining pipeline to trigger automatically without manual intervention, minimize idle compute costs, and integrate with Amazon SageMaker Model Registry so a human can approve the model before deployment.

Which solution meets these requirements with the LEAST operational overhead? (Select ONE)

  • A) Configure an S3 Event Notification that sends an ObjectCreated event to Amazon EventBridge, which triggers an AWS Lambda function that starts a SageMaker Pipelines execution. The pipeline ends by registering the model in SageMaker Model Registry with PendingManualApproval status.
  • B) Schedule an Amazon EventBridge rule with a cron expression to run the SageMaker Pipeline every 6 hours. The pipeline checks for new data and retrains only if new files exist.
  • C) Deploy an AWS Glue job that polls the S3 bucket for new data and manually triggers the training job using the SageMaker API.
  • D) Configure Amazon CloudWatch to monitor the S3 bucket size metric and trigger a SageMaker training job when the bucket grows by more than 1 GB.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Answer Analysis

  • A is Correct: This is the ideal event-driven architecture for irregular data arrival. S3 Event Notifications emit ObjectCreated events when new data lands, EventBridge routes the event, and Lambda starts the SageMaker Pipeline with minimal latency. The pipeline ends at the Model Registry with manual approval status, ensuring a human gate before production deployment. This approach has zero idle compute cost because the pipeline only runs when data arrives.

Distractor Analysis

  • B is Incorrect: A fixed schedule wastes compute resources when no new data has arrived and introduces latency when data arrives shortly after a scheduled run. For irregular data patterns, event-driven triggers are superior.
  • C is Incorrect: Using AWS Glue to poll S3 introduces unnecessary operational overhead—the Glue job must run continuously or on a schedule, and polling is an anti-pattern when native event notifications exist.
  • D is Incorrect: Monitoring bucket size is a coarse, unreliable trigger. It does not detect when new labeled data specifically arrives (the bucket may grow due to unrelated objects), and a size threshold adds unnecessary delay.

Key Takeaway: For irregular data arrival, always prefer event-driven triggers (S3 Event Notifications → EventBridge → Lambda → SageMaker Pipelines) over scheduled or polling-based approaches to minimize both latency and idle compute costs.

Question 2 - Retraining Trigger Selection

A retail company operates a product recommendation model on Amazon SageMaker. The data science team has identified that model performance degrades when customer shopping patterns shift during seasonal changes. They need a retraining mechanism that responds to monitored metric degradation in production rather than waiting for a fixed schedule. The solution must be fully automated and must route through an Amazon SageMaker Pipeline for preprocessing, training, and evaluation.

Which mechanism should the team implement? (Select ONE)

  • A) Configure an Amazon CloudWatch Alarm on the endpoint's Invocation4XXErrors metric. When the alarm enters the ALARM state, an Amazon EventBridge rule triggers StartPipelineExecution on the SageMaker retraining pipeline.
  • B) Configure a SageMaker Model Monitor job that emits a CloudWatch metric for feature drift. Create a CloudWatch Alarm with an EventBridge rule that triggers the SageMaker Pipeline when the alarm fires.
  • C) Schedule a weekly SageMaker Pipeline execution using an EventBridge rule with a cron expression to retrain the model regardless of drift metrics.
  • D) Deploy an AWS Step Functions state machine that polls Amazon CloudWatch Logs every hour for drift indicators and triggers retraining when detected.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: Amazon SageMaker Model Monitor continuously evaluates production endpoint data for data quality, model quality, and feature attribution drift. When drift exceeds a specified threshold, Model Monitor emits a CloudWatch metric. A CloudWatch Alarm on that drift metric—routed through Amazon EventBridge—automatically triggers the SageMaker Pipeline for retraining. This is the correct event-driven approach tied to actual model degradation signals.

Distractor Analysis

  • A is Incorrect: Invocation4XXErrors measures client-side errors (e.g., malformed requests), not model performance degradation. A high 4XX rate indicates API usage problems, not the need for model retraining.
  • C is Incorrect: A fixed weekly schedule retrains regardless of whether the model has actually degraded, wasting compute resources and potentially missing rapid degradation between scheduled runs.
  • D is Incorrect: Polling CloudWatch Logs with a Step Functions state machine introduces unnecessary operational complexity and latency. Native CloudWatch Alarms and EventBridge integration provide a more efficient event-driven path.

Key Takeaway: SageMaker Model Monitor → CloudWatch Alarm → EventBridge → SageMaker Pipelines is the correct chain for metric-driven automated retraining when model quality degrades in production.

Question 3 - Retraining Pipeline Integration with Model Registry

An e-commerce company has built a SageMaker Pipeline for retraining a product categorization model. The pipeline includes preprocessing, training, evaluation, and model registration steps. The compliance team requires that no model be deployed to production without human approval after retraining. Additionally, the deployment pipeline must be able to automatically pick up the approved model version once approval is granted.

Which configuration should the ML engineer implement? (Select TWO)

  • A) Configure the SageMaker Pipeline's model registration step to set ModelApprovalStatus to PendingManualApproval in SageMaker Model Registry.
  • B) Configure the SageMaker Pipeline to deploy the model directly to a SageMaker endpoint in the final step to avoid deployment delays.
  • C) Create an Amazon EventBridge rule that listens for SageMakerModelRegistryModelApproved events and triggers a CodePipeline deployment stage using the approved model package ARN.
  • D) Store the model artifact in an S3 bucket and manually copy the model.tar.gz file to the deployment pipeline at approval time.
  • E) Configure the model registration step to automatically set ModelApprovalStatus to Approved so that the deployment pipeline can consume it immediately.
Click to reveal Answer & Explanation

Correct Answer: A and C

Correct Answer Analysis

  • A is Correct: Setting ModelApprovalStatus to PendingManualApproval ensures the trained model lands in the registry but cannot be deployed until a human reviews and approves it in the SageMaker Model Registry console or via API. This satisfies the compliance requirement for human gating.
  • C is Correct: Amazon EventBridge natively emits SageMakerModelRegistryModelApproved events when a registered model version is approved. An EventBridge rule can capture this event and automatically trigger the downstream deployment pipeline (e.g., AWS CodePipeline) with the approved model package ARN, eliminating manual handoff.

Distractor Analysis

  • B is Incorrect: Deploying directly from the retraining pipeline bypasses the human approval gate required by compliance, and couples retraining directly to production deployment—an anti-pattern in MLOps.
  • D is Incorrect: Manually copying artifacts between systems is error-prone and breaks lineage tracking. The Model Registry maintains immutable metadata linking the model package to its training run.
  • E is Incorrect: Setting auto-approval immediately would violate the compliance requirement of human approval before deployment.

Key Takeaway: The separation of concerns—retraining pipeline ends at the Model Registry (PendingManualApproval), and the deployment pipeline starts from the approval event—is the core MLOps pattern for human-gated model releases.

Question 4 - Retraining with MLflow on SageMaker

A machine learning team uses MLflow on Amazon SageMaker for experiment tracking. They have run multiple training experiments with different hyperparameters and need to register the winning run to the SageMaker Model Registry for subsequent automated deployment via CodePipeline. The team wants this process to be automated and preserve lineage from the original training run.

Which approach should the team take? (Select ONE)

  • A) Manually download the winning model artifact from the MLflow tracking server and upload it to an S3 bucket, then create a model package in SageMaker Model Registry referencing that S3 URI.
  • B) Use the MLflow mlflow.register_model() API to register the winning run's model artifact directly into the SageMaker Model Registry as a new model version within the specified model group.
  • C) Copy the model artifact to a new S3 bucket and update a latest_model.txt pointer file that the deployment pipeline reads at deploy time.
  • D) Export the MLflow experiment metrics to a CSV file and manually create a model card in Amazon SageMaker Model Registry.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: MLflow's managed integration with SageMaker Model Registry allows you to call mlflow.register_model() with the run ID and model path. This creates a model version in the specified SageMaker Model Registry group, preserving lineage from the MLflow experiment run—including parameters, metrics, and source code—through the registry entry.

Distractor Analysis

  • A is Incorrect: Manually downloading and uploading artifacts breaks automated lineage tracking. The Model Registry entry would not automatically link back to the MLflow experiment run, making it harder to reconstruct provenance.
  • C is Incorrect: A pointer file (latest_model.txt) is fragile, lacks the approval status and metadata management of the Model Registry, and does not provide acceptable audit capabilities for production deployments.
  • D is Incorrect: Exporting metrics to CSV does not register the model artifact itself and introduces manual steps that undermine automation and repeatability.

Key Takeaway: MLflow's native register_model() API is the correct path for automatic model registration to SageMaker Model Registry while preserving end-to-end experiment lineage.

Question 5 - Multi-Account Retraining Source Configuration

A healthcare analytics company operates a multi-account AWS environment. The ML retraining pipeline in the production account needs to pull source code from a GitHub Enterprise Server repository hosted in the company's on-premises data center. The security team prohibits storing personal access tokens or SSH keys directly in the pipeline configuration. The solution must integrate with AWS CodePipeline.

Which configuration enables this? (Select ONE)

  • A) Create an AWS CodeCommit repository in the production account and mirror the GitHub Enterprise repository using a scheduled AWS Lambda function.
  • B) Create an AWS CodeConnections connection to the GitHub Enterprise Server and reference that connection ARN in the CodePipeline source stage.
  • C) Store the GitHub Enterprise Server credentials in AWS Secrets Manager and configure CodePipeline to retrieve them directly in the source stage.
  • D) Deploy a self-managed Git proxy on EC2 that CodePipeline polls for changes to trigger the pipeline.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: AWS CodeConnections (formerly CodeStar Connections) is specifically designed for this use case. It establishes an OAuth-based or GitHub App-based connection to repositories hosted outside AWS (including GitHub Enterprise Server) and stores the connection resource in the account. CodePipeline references this connection ARN in its source stage, eliminating the need to store credentials in pipeline definitions.

Distractor Analysis

  • A is Incorrect: Mirroring a repository via Lambda introduces unnecessary complexity and an additional failure point. It also creates a lag between source changes and pipeline triggering, violating the principle of using native, managed integrations.
  • C is Incorrect: While AWS Secrets Manager securely stores credentials, CodePipeline does not natively integrate with Secrets Manager for retrieving source repository credentials in the source stage. This approach also requires additional custom logic.
  • D is Incorrect: A self-managed Git proxy on EC2 represents significant operational overhead—patching, scaling, and maintaining the proxy—when CodeConnections provides a fully managed alternative.

Key Takeaway: AWS CodeConnections is always the correct answer when CodePipeline must connect to any Git repository hosted outside AWS without embedding credentials.

Question 6 - Automated Retraining with CodePipeline

A media streaming company manages their ML infrastructure using AWS CodePipeline and AWS CodeBuild for CI/CD. Their retraining workflow includes the following stages: source code checkout, unit tests, container build, model training, model evaluation, and model registration. The team wants to ensure that a model fails the pipeline if its evaluation metric (F1 score) falls below 0.85 on a held-out validation set.

Where should the evaluation gate be implemented? (Select ONE)

  • A) In the model training script, raise an exception if the F1 score is below 0.85 so the training job fails.
  • B) Configure a CodeBuild stage after model training that runs the evaluation script and uses the exit 1 command when the F1 score is below the threshold, causing the pipeline stage to fail.
  • C) Deploy the model to a SageMaker endpoint with a canary rollout and monitor F1 score in production before approving.
  • D) Configure an Amazon CloudWatch Alarm on the training job's error metric to halt the pipeline if F1 score drops below 0.85.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: In a CodePipeline/CodeBuild-based MLOps workflow, the correct approach is to create a distinct CodeBuild stage that executes the model evaluation script after training completes. If the F1 score does not meet the threshold, the script exits with a non-zero status (exit 1), causing the CodeBuild action to fail and automatically preventing the pipeline from proceeding to model registration or deployment. This creates a hard gate with clear pipeline visibility.

Distractor Analysis

  • A is Incorrect: Raising an exception in the training script conflates training failure with evaluation failure. The training job itself may complete successfully even if the model quality is poor; evaluation is a distinct step that should be separated for clarity and debugging.
  • C is Incorrect: Deploying to production first and then monitoring F1 score defeats the purpose of a pre-deployment gate. This exposes production traffic to a potentially poor model before validation.
  • D is Incorrect: CloudWatch Alarms on training job error metrics relate to infrastructure or job failures, not model quality metrics like F1 score. Model evaluation metrics are not emitted as training job error signals that can halt a pipeline.

Key Takeaway: In CI/CD for ML, separate the training step from the evaluation gate—a CodeBuild step that runs evaluation scripts and fails the pipeline on metric threshold violations is the standard pattern.

Question 7 - Foundation Model Retraining with SageMaker Training Jobs

An enterprise has fine-tuned a Llama 3 foundation model using Amazon SageMaker Training Jobs for a document summarization task. The training data is stored as JSON Lines in Amazon S3, and the training script is containerized in Amazon ECR. The pipeline must automatically execute training weekly, register the fine-tuned model with unique versioning in SageMaker Model Registry, and deploy only after an approval gate is passed.

Which configuration satisfies these requirements? (Select TWO)

  • A) Create an Amazon EventBridge rule with a cron schedule that triggers an AWS CodePipeline pipeline. The pipeline stages include Source (ECR + S3), Build (CodeBuild to validate container), and Train (SageMaker Pipelines step for training using the ECR container and S3 data).
  • B) Configure the SageMaker Training Job to output the model artifact to S3 and register it directly in SageMaker Model Registry with ModelApprovalStatus set to Approved so that deployment can begin immediately.
  • C) Use SageMaker Pipelines with a TrainingStep that references the ECR container and S3 training data. Add a ModelStep to register the trained model in SageMaker Model Registry with ModelApprovalStatus set to PendingManualApproval.
  • D) Store the fine-tuned weights in an S3 bucket and update the production SageMaker endpoint configuration to point to the new S3 path without using the Model Registry.
  • E) Configure CodePipeline's final stage to deploy the model from Model Registry to a SageMaker endpoint only when a manual approval action is completed by an IAM-authorized user.
Click to reveal Answer & Explanation

Correct Answer: C and E

Correct Answer Analysis

  • C is Correct: SageMaker Pipelines provides a native TrainingStep for executing training jobs with specified container images and data inputs, and a ModelStep to register the output model artifact into SageMaker Model Registry. Setting ModelApprovalStatus to PendingManualApproval ensures compliance with the approval gate requirement.
  • E is Correct: CodePipeline supports manual approval actions that pause the pipeline until an IAM-authorized user approves. The final stage then deploys the approved model to a SageMaker endpoint, completing the gated deployment flow.

Distractor Analysis

  • A is Incorrect: While the pipeline structure is reasonable, it does not address the Model Registry registration or the manual approval gating requirement. It also combines CodePipeline and SageMaker Pipelines in a way that does not clearly integrate the approval step.
  • B is Incorrect: Setting ModelApprovalStatus to Approved bypasses the required approval gate and deploys immediately, violating the compliance requirement.
  • D is Incorrect: Updating endpoint configurations directly without using the Model Registry skips versioning and lineage tracking, making audits and rollbacks significantly harder.

Key Takeaway: SageMaker Pipelines TrainingStep + ModelStep with PendingManualApproval integrates retraining with Model Registry versioning, and CodePipeline manual approval actions provide the human gate for production deployment.

Question 8 - Retraining with Drift Detection on SageMaker Endpoint

A financial trading firm runs a real-time risk assessment model on a SageMaker endpoint. The model's input features are derived from market data that changes rapidly. The firm wants to automatically trigger retraining when feature distribution drift is detected on the endpoint's incoming inference requests. The solution must be fully automated and must not require the engineering team to manually monitor dashboards.

Which TWO configurations are required? (Select TWO)

  • A) Enable Amazon SageMaker Model Monitor with a data quality baseline generated from the training dataset. Configure it to run continuously on endpoint data.
  • B) Create an Amazon CloudWatch Alarm on the SageMaker Model Monitor drift metric that transitions to ALARM state when drift exceeds the configured threshold.
  • C) Configure an AWS Config rule that detects changes to the endpoint's input data and triggers a SageMaker Pipeline execution.
  • D) Create an Amazon EventBridge rule that listens for the CloudWatch Alarm state change event from the drift alarm and triggers the retraining SageMaker Pipeline.
  • E) Configure the SageMaker endpoint's Auto Scaling policy to trigger a scale-out event when drift is detected, which causes retraining to occur.
Click to reveal Answer & Explanation

Correct Answer: B and D

Correct Answer Analysis

  • B is Correct: SageMaker Model Monitor emits a CloudWatch metric (e.g., FeatureDistance) when drift between the baseline and incoming data exceeds a threshold. A CloudWatch Alarm on this metric is the correct mechanism to detect drift and signal the event.
  • D is Correct: Amazon EventBridge can listen for CloudWatch Alarm state-change events (CloudWatch Alarm State Change). When the drift alarm enters ALARM state, EventBridge triggers a SageMaker Pipeline execution for retraining—fully automated, no manual dashboards needed.

Distractor Analysis

  • A is Incorrect (in context): While enabling Model Monitor with a data quality baseline is a necessary prerequisite, the option as stated does not specify the alarm or trigger mechanism. For this question's objective, the alarm and EventBridge rule are the actionable triggers; Model Monitor configuration is assumed as a prerequisite. However, in a complete solution, Model Monitor must be enabled—making this option partially correct but not sufficient alone.
  • C is Incorrect: AWS Config tracks configuration changes of AWS resources, not feature drift or data distribution shifts on an ML endpoint. This is an incorrect service for the stated problem.
  • E is Incorrect: Auto Scaling policies respond to infrastructure metrics (e.g., CPU utilization, invocation counts) to adjust compute capacity. They have no capability to detect drift or trigger retraining pipelines.

Key Takeaway: The complete drift-based retraining chain is: Model Monitor (data quality baseline) → CloudWatch Alarm on drift metric → EventBridge rule on alarm state change → SageMaker Pipeline StartPipelineExecution.

Question 9 - Retraining Pipeline with SageMaker Pipelines and CodePipeline Integration

A machine learning team has built a SageMaker Pipelines workflow that handles preprocessing, training, evaluation, and model registration. They also use AWS CodePipeline to manage code changes and deployment workflows. The team wants to ensure that when code changes are pushed to the repository, the SageMaker Pipeline is automatically triggered, and that the CodePipeline waits for the SageMaker Pipeline to complete before proceeding to deployment.

Which configuration achieves this? (Select ONE)

  • A) In CodePipeline, add an AWS CodeBuild action that invokes StartPipelineExecution on the SageMaker Pipeline. Configure the next stage to wait using a manual approval action before deployment.
  • B) In CodePipeline, add an AWS Lambda action that calls StartPipelineExecution. Use a second Lambda action that polls DescribePipelineExecution until the SageMaker Pipeline completes, then returns success to CodePipeline.
  • C) Use an Amazon EventBridge rule to trigger the SageMaker Pipeline when a CodeCommit push occurs, and configure a separate EventBridge rule to trigger the deployment stage when the SageMaker Pipeline completes.
  • D) Configure the SageMaker Pipeline's last step to manually invoke CodePipeline using the AWS SDK, and set CodePipeline to run only when it receives that invocation.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: CodePipeline supports AWS Lambda actions that can call the SageMaker API (StartPipelineExecution). The Lambda function can then poll DescribePipelineExecution to monitor the pipeline's status (e.g., Succeeded or Failed). CodePipeline waits for the Lambda action to complete, and the Lambda returns success only when the SageMaker Pipeline has finished successfully. This creates a synchronous handoff between CodePipeline and SageMaker Pipelines.

Distractor Analysis

  • A is Incorrect: Using CodeBuild to start the pipeline and then using a manual approval action does not wait for the SageMaker Pipeline to complete. The manual approval is a human gate, not a programmatic wait on the pipeline's execution status.
  • C is Incorrect: This approach decouples the two pipelines, which may work but does not guarantee CodePipeline waits for SageMaker Pipelines completion before proceeding. It also introduces additional complexity in coordinating two separate triggers.
  • D is Incorrect: Having the SageMaker Pipeline call back into CodePipeline adds coupling in the wrong direction. It also requires embedding CodePipeline invocation logic in the SageMaker Pipeline, which is an anti-pattern for separation of concerns.

Key Takeaway: When CodePipeline must synchronously wait for a SageMaker Pipeline to finish, use a Lambda action that starts the pipeline and polls DescribePipelineExecution until completion before returning.

Question 10 - Retraining Trigger Based on S3 Manifest File

A logistics company maintains a manifest file in Amazon S3 that lists all raw data files available for model retraining. The manifest is updated at irregular intervals by an upstream data ingestion process. The ML team wants the retraining pipeline to trigger only when the manifest file itself is updated, indicating that new data is ready for processing. The pipeline will read the manifest to determine which files to process.

Which configuration triggers the pipeline with minimum latency after a manifest update? (Select ONE)

  • A) Configure an S3 Event Notification on the manifest file prefix that sends an ObjectUpdated event to Amazon EventBridge, which triggers a Lambda function to start the SageMaker Pipeline.
  • B) Schedule an Amazon EventBridge cron rule to run every hour and check if the manifest file's LastModified timestamp has changed. If changed, start the pipeline.
  • C) Configure an AWS CloudTrail log subscription filter that watches for PutObject API calls on the manifest file and triggers an SNS topic that starts the pipeline.
  • D) Deploy a small EC2 instance running a cron job that uses the AWS CLI to check the manifest's ETag every 5 minutes and start the pipeline when it changes.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Answer Analysis

  • A is Correct: S3 Event Notifications on the specific manifest file prefix provide the lowest latency trigger. When the manifest file is uploaded or overwritten, S3 emits an s3:ObjectCreated:* event (and s3:ObjectRemoved:* for deletions) to Amazon EventBridge. The EventBridge rule then triggers a Lambda function that starts StartPipelineExecution on the SageMaker Pipeline—typically within seconds of the manifest update.

Distractor Analysis

  • B is Incorrect: A scheduled check introduces unnecessary latency (up to 1 hour between checks) and does not provide true event-driven behavior. Even with a shorter schedule, there is a polling interval delay.
  • C is Incorrect: CloudTrail logs are eventual and can have delivery delays of up to 15 minutes for management events. S3 Event Notifications are a more direct and lower-latency mechanism for detecting S3 object changes.
  • D is Incorrect: Managing an EC2 instance with a cron job introduces significant operational overhead (patching, scaling, monitoring). It is also a polling approach rather than an event-driven trigger, adding unnecessary latency.

Key Takeaway: For S3-based triggers, use S3 Event Notifications with EventBridge for lowest latency. Avoid polling mechanisms (schedules, cron jobs) and CloudTrail-based triggers when real-time response is required.