Skill 3.3.6: Manage model versions for repeatability and audits (for example, SageMaker Model Registry, MLflow on SageMaker AI)

Question 1 - SageMaker Model Registry Lineage for Audit Investigation

A financial services company is undergoing a regulatory audit. The compliance team must determine exactly which trained model artifact was serving production traffic at a specific timestamp three months ago during a fraud detection incident.

The company uses Amazon SageMaker Pipelines for training orchestration and Amazon SageMaker Model Registry for model versioning. All training runs are tracked via MLflow on SageMaker AI. The audit must establish the complete lineage chain: endpoint traffic at that timestamp → deployed model version → registered model package → training job that produced it → input dataset URI and hyperparameters.

Which approach satisfies the audit requirement with minimal operational overhead?

A) Query CloudWatch Logs from the model endpoint at that timestamp to find the training job ID embedded in the inference request payloads.
B) Navigate the SageMaker Model Registry version's lineage graph, which links the model package version to the training job, input data, and hyperparameters, and cross-reference the endpoint deployment history for the timestamp.
C) Check the S3 bucket object tags on the model artifact file in the model.tar.gz archive; the S3 object tags record which training job generated the artifact.
D) Review the CloudTrail event history for CreateModel API calls around the timestamp to identify which model artifact was used.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: SageMaker Model Registry maintains immutable lineage tracking for each model package version. The lineage graph connects the registered model version to the exact training job (TrainingJobArn), input data locations, hyperparameters, evaluation metrics, and source pipeline execution. Combined with endpoint deployment history (which records which model package version was deployed at any given time via DescribeEndpointConfig or deployment logs), this provides a complete audit trail from production traffic backward to the source data and training configuration.

Distractor Analysis

A is Incorrect: CloudWatch Logs from an endpoint contain inference invocation logs (input/output payload data), not metadata about which training job produced the model weights. The training job ID is not part of the inference request payload in standard SageMaker deployments.
C is Incorrect: S3 object tags on model artifacts are not automatically applied by SageMaker training jobs. While you could manually implement custom tagging, there is no built-in mechanism that records lineage information on model artifacts in S3, making this an incomplete and non-standard approach.
D is Incorrect: CloudTrail CreateModel events show when a model was created and with what container/image configuration, but they do not provide the end-to-end lineage back to the training job, input data, and hyperparameters. This would only provide partial information.

Key Takeaway: SageMaker Model Registry's built-in lineage tracking is the authoritative source for audit investigations connecting deployed models back to their training runs and input data.

Question 2 - MLflow Experiment Registration to Model Registry

A machine learning team recently migrated to MLflow on SageMaker AI for experiment tracking. They run hyperparameter tuning jobs that each produce between 50–100 candidate models. After each tuning job completes, the data science lead reviews evaluation metrics and selects the best-performing run for production deployment.

The current workflow requires manually copying the winning model artifact from the MLflow tracking server's artifact store to the SageMaker Model Registry. The team wants to automate the registration of the winning run into the registry while preserving complete lineage back to the MLflow experiment.

Which TWO steps should the engineer configure? (Select TWO)

A) Use MLflow's mlflow.register_model() API with the run ID of the winning model to create a model version in the SageMaker Model Registry automatically.
B) Create a scheduled SageMaker Pipeline that re-trains all models daily and registers the most recent one to the registry.
C) Configure an EventBridge rule that triggers when the MLflow tracking server writes a new metric to CloudWatch, then invoke a Lambda function to copy the model artifact to S3.
D) Store the MLflow run ID and experiment ID as metadata fields when registering the model package version in SageMaker Model Registry.
E) Use SageMaker Model Monitor to detect when a better model has been trained and automatically register it.

Click to reveal Answer & Explanation

Correct Answers: A and D

Correct Option Analysis

A is Correct: MLflow on SageMaker AI integrates natively with SageMaker Model Registry. The mlflow.register_model() API call registers the model artifact produced by a specific MLflow run directly into the registry, creating a new model package version without manual copying.
D is Correct: Storing the MLflow run ID and experiment ID as custom metadata on the registered model package version in SageMaker Model Registry preserves the lineage link between the registry entry and the MLflow experiment tracking system, enabling full auditability.

Distractor Analysis

B is Incorrect: Scheduling daily re-training of all models is wasteful and unnecessary when the requirement is to register an already-trained winning run. It also doesn't address automatic registration—it just trains more models.
C is Incorrect: MLflow tracking servers on SageMaker AI do not automatically emit metrics to CloudWatch in a way that would trigger this workflow. Even if they did, this approach still requires a custom copy step that doesn't use the native MLflow-to-registry integration.
E is Incorrect: SageMaker Model Monitor detects data drift, prediction quality degradation, and bias in deployed models—it does not identify when a better model has been trained in an experiment pipeline.

Key Takeaway: MLflow on SageMaker AI provides native integration with SageMaker Model Registry through mlflow.register_model(), and storing experiment/run IDs as registry metadata preserves end-to-end lineage.

Question 3 - Model Approval Status for Production Deployment

A healthcare AI company manages diagnostic models through SageMaker Model Registry. They want to enforce a governance workflow where newly trained models must pass evaluation checks and receive manual approval from a clinical compliance officer before they can be deployed to production endpoints.

The deployment pipeline is implemented in AWS CodePipeline with a deploy stage that uses SageMaker Deployment Guardrails.

Which configuration ensures that only approved model versions reach production with the strongest governance enforcement?

A) Tag each model with a "production-ready" label in S3 after manual review, and have the pipeline check for this tag before deployment.
B) Set the model package version status to Approved in SageMaker Model Registry after compliance review, and configure the deploy stage in CodePipeline to read the approval status and fail if it is not Approved.
C) Send an email notification to the compliance officer via SNS after each training run; the officer manually deploys the model using the SageMaker console.
D) Use IAM policy conditions on the endpoint to deny deployment of any model that does not have a specific S3 tag.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: SageMaker Model Registry's ModelApprovalStatus field is designed exactly for this governance pattern. The registry enforces a workflow where models start as PendingManualApproval and must be explicitly changed to Approved by an authorized user. CodePipeline can query the registry API (DescribeModelPackage) in a custom action or via a CodeBuild step to verify the status before triggering the SageMaker deployment. This creates a hard gate in the pipeline that cannot be bypassed.

Distractor Analysis

A is Incorrect: S3 object tags are not an enforced metadata mechanism for model governance. Any pipeline step with S3 write access could theoretically modify tags, and this approach separates the approval record from the model registry where lineage and metadata live.
C is Incorrect: Manual deployment via the console bypasses the automated CI/CD pipeline entirely, introduces human error risk, and creates a completely ungoverned path to production without pipeline-level enforcement.
D is Incorrect: IAM policy conditions on endpoints can control who can deploy models, but they cannot enforce model-specific quality or approval status. This approach also lacks integration with the registry's approval workflow.

Key Takeaway: SageMaker Model Registry approval status (Approved/PendingManualApproval/Rejected) is the native governance mechanism that CI/CD pipelines should read as a deployment gate for audited production releases.

Question 4 - MLflow Tracking Server as Source of Truth

A retail company's data science team uses MLflow on SageMaker AI to track experiments across multiple projects. The team currently has a manually maintained spreadsheet that records the best hyperparameters, metrics, and model artifact locations for each project.

The CTO wants a manageable, low-overhead system that provides reproducible experiment records for internal audits, including the ability to answer questions like "What hyperparameters and dataset produced model X?" and "How does model X compare to other runs in the same experiment?"

Which approach should the ML engineer recommend?

A) Continue using the spreadsheet but require data scientists to take screenshots of the SageMaker training console for each run.
B) Use MLflow's experiment tracking to log parameters, metrics, input dataset URIs, and artifacts for each run; organize runs by experiment, and use the tracking UI/API for audit lookups.
C) Write each run's parameters and metrics to a DynamoDB table using a Lambda function triggered after each training job.
D) Store all hyperparameters in the model filename and store metrics in S3 object tags.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: MLflow on SageMaker AI is a fully managed tracking server that automatically records experiment runs, including hyperparameters, metrics, input dataset references, source code versions, and produced artifacts. It provides a built-in UI and API for comparing runs within an experiment and querying historical results. This is the native, low-overhead solution for experiment reproducibility and audit trails.

Distractor Analysis

A is Incorrect: Spreadsheets combined with screenshots are not a scalable or auditable solution. They lack automation, cannot capture all metadata, and are prone to human error and versioning problems.
C is Incorrect: Building a custom DynamoDB-based tracking system introduces significant operational overhead when MLflow already provides this capability natively. It also lacks features like automatic artifact linking and experiment comparison views.
D is Incorrect: Encoding metadata in filenames and S3 object tags is brittle, does not scale, and loses the structured querying and comparison capabilities that a proper tracking server provides.

Key Takeaway: MLflow on SageMaker AI provides built-in experiment tracking with automated logging of parameters, metrics, and artifacts—eliminating the need for custom tracking solutions.

Question 5 - Multi-Account Model Registry Access

A large enterprise runs its ML teams in separate AWS accounts (development, staging, and production). Models are trained in the development account using SageMaker Pipelines. After approval, models must be accessible for deployment from the production account.

The compliance team requires that the exact same model package version approved in development is what gets deployed in production, with full lineage preserved across accounts.

Which configuration satisfies this requirement with minimal operational overhead?

A) Copy the model artifact (model.tar.gz) from the development account's S3 bucket to the production account's S3 bucket, then create a new model in production pointing to the copied artifact.
B) Use SageMaker Model Registry's cross-account sharing capability to share the model package group with the production account, allowing production to deploy the exact approved version by referencing its ARN.
C) Re-train the model in the production account using the same data and hyperparameters, then register the new model in production.
D) Export the model artifact to Amazon ECR, push the container to a shared ECR repository, and reference the ECR image in the production account's model definition.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: SageMaker Model Registry supports cross-account sharing of model packages and model groups via AWS Resource Access Manager (RAM) or resource-based policies. By sharing the registry from the development account, the production account can deploy the identical model package version using its ARN, preserving the full lineage, metadata, and approval status without duplicating artifacts or creating a new registry entry.

Distractor Analysis

A is Incorrect: Copying the model artifact between accounts creates a disconnected copy. The production model loses the lineage back to the original training job and registry metadata in the development account, breaking the audit chain.
C is Incorrect: Re-training in the production account produces a different model artifact (due to non-determinism in training) and creates a duplicate registry entry, defeating the purpose of version consistency across environments.
D is Incorrect: Amazon ECR stores container images, not model weights or artifacts. Even if you package the model into an inference container, this approach still bypasses the Model Registry and loses the registry-level metadata and lineage.

Key Takeaway: SageMaker Model Registry supports cross-account sharing, enabling the exact same model package version to be deployed across development, staging, and production accounts with preserved lineage.

Question 6 - Model Version Rollback with Registry

An e-commerce company runs a product recommendation model behind a single SageMaker real-time endpoint. They use SageMaker Model Registry and SageMaker Deployment Guardrails for model releases.

After deploying a new model version, the team observes a significant increase in recommendation click-through rate drop-off. The on-call engineer needs to quickly revert to the previous model version while preserving the deployment history for the incident review.

What should the engineer do?

A) Delete the current endpoint and create a new endpoint pointing to the previous model.
B) Trigger an automatic rollback by configuring an alarm on the click-through rate metric; if the alarm fires during the baking period, SageMaker automatically reverts to the previous version.
C) Manually update the production endpoint configuration to reference the previous model package version ARN from Model Registry, creating a new endpoint configuration revision.
D) Restore the previous model weights from an S3 backup bucket and overwrite the current model artifact.

Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

C is Correct: If the deployment has already completed (outside the baking period or no guardrail was configured), the manual rollback path is to update the endpoint configuration to reference the previous model package version ARN from SageMaker Model Registry. This approach is clean, uses the registry's versioned artifacts, and maintains a clear audit trail of which version was deployed when.

Distractor Analysis

A is Incorrect: Deleting and recreating the endpoint causes unnecessary downtime and risks losing the endpoint's configuration, scaling policies, and other settings. Updating the endpoint configuration in place is the correct approach.
B is Incorrect: An automatic rollback with a baking period must be configured before the deployment, not after. Once the baking period has expired and the deployment has completed, the guardrail rollback mechanism no longer applies. This option is also hypothetical and does not provide an immediate recovery path.
D is Incorrect: Overwriting model artifacts in S3 is a dangerous practice. It breaks the immutability of versioned artifacts, destroys the lineage audit trail, and may inadvertently affect other consumers of the same artifact.

Key Takeaway: For manual rollbacks after deployment completion, update the endpoint configuration to reference the previous model package version ARN from SageMaker Model Registry—never overwrite artifacts.

Question 7 - MLflow Auto-Registration in Pipeline

An autonomous driving perception team uses AWS CodePipeline to orchestrate their ML lifecycle. The pipeline includes:

A CodeCommit source stage
A CodeBuild stage that runs a SageMaker training job
A CodeBuild stage that runs model evaluation
A manual approval gate
A CodeBuild stage that deploy the approved model

Currently, after the evaluation stage passes, a data scientist manually runs mlflow.register_model() from their local machine to register the best run. The team wants to integrate MLflow auto-registration into the pipeline so that the winning model is automatically registered after evaluation without manual intervention.

Which TWO steps achieve this? (Select TWO)

A) Add a new CodeBuild stage after the evaluation stage that calls the MLflow tracking server API using the run ID captured from the training job output to register the model into SageMaker Model Registry.
B) Configure an EventBridge rule that matches SageMaker training job completion events and automatically registers any trained model to the registry.
C) Use the mlflow.autolog() feature in the training script, which automatically registers every trained model to SageMaker Model Registry without any additional pipeline stage.
D) Store the MLflow run ID as an output artifact from the training CodeBuild stage, then use it in the registration step.
E) Remove the evaluation stage entirely and register models directly after training to reduce pipeline complexity.

Click to reveal Answer & Explanation

Correct Answers: A and D

Correct Option Analysis

A is Correct: Adding a CodeBuild stage that invokes the MLflow tracking server API (mlflow.register_model()) with the specific run ID of the winning model is the correct way to automate registration within the pipeline. The pipeline's ordering ensures registration only happens after evaluation passes.
D is Correct: Capturing the MLflow run ID as a pipeline artifact (e.g., in a JSON file stored in S3 or passed between CodePipeline stages) enables the registration stage to reference the correct run without manual data entry. This maintains the lineage between the training run and the registered model version.

Distractor Analysis

B is Incorrect: Registering every trained model automatically without evaluation or quality gating would flood the registry with potentially poor-performing models and bypass the governance process. Registration should occur only after the evaluation stage confirms the model meets quality thresholds.
C is Incorrect: mlflow.autolog() automatically logs parameters and metrics during model training, but it does not automatically register models to the SageMaker Model Registry. Model registration is a separate explicit API call.
E is Incorrect: Removing the evaluation stage eliminates a critical quality gate in the pipeline and would allow substandard models to be registered and potentially deployed.

Key Takeaway: In CI/CD pipelines for ML, MLflow model registration should be an explicit pipeline stage that executes after evaluation passes, using the run ID captured from the training stage.

Question 8 - Audit Trail for Model Promotion Decisions

A global logistics company uses SageMaker Model Registry to manage thousands of models across multiple business units. The compliance department requires a complete audit trail showing:

Who approved each model version for production
When approval was granted
What evaluation metrics were associated with the approval decision

The ML platform team needs to implement this with minimal custom development.

Which approach provides the required audit trail?

A) Enable AWS CloudTrail for SageMaker Model Registry API calls, and store evaluation metrics as model package metadata at registration time using the CustomerMetadataMap field.
B) Create a custom approval workflow using AWS Step Functions that records approvals in a DynamoDB table.
C) Have approvers send an email to a shared mailbox; a Lambda function parses the email and stores approval records in S3.
D) Manually maintain an approval log in a shared document that approvers update after reviewing models.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

A is Correct: AWS CloudTrail automatically captures all SageMaker Model Registry API calls, including UpdateModelPackageStatus events that record who changed the approval status and when. The CustomerMetadataMap field on model package versions allows you to store evaluation metrics alongside the version at registration time. Together, these provide a complete, automated audit trail without custom infrastructure.

Distractor Analysis

B is Incorrect: Building a custom Step Functions approval workflow with DynamoDB introduces unnecessary complexity when CloudTrail already provides the audit trail and Model Registry provides the approval status field natively. This approach would require maintaining separate infrastructure.
C is Incorrect: Email-based approval parsing is fragile, introduces latency, and does not integrate with the model registry's native approval workflow. It also doesn't automatically link approval decisions to specific model versions in the registry.
D is Incorrect: Manual document logs are not a reliable audit trail, prone to human error, and scale poorly with thousands of models. They also cannot be automatically queried.

Key Takeaway: CloudTrail + SageMaker Model Registry approval status + CustomerMetadataMap for metrics provides a complete audit trail with zero custom code.

Question 9 - Model Version Naming Strategy

A software company is implementing a Model Registry-based versioning system for their first production ML models. They need a consistent, scalable versioning strategy that supports:

Multiple models in production simultaneously
Clear identification of which model group a version belongs to
Compliance with major-version audit requirements (e.g., "All v2 models must be traceable to the v2 training pipeline")

Which approach should the ML engineer recommend?

A) Use a single model group named "ProductionModel" and rely on model package version numbers (1, 2, 3...) to differentiate between different models.
B) Create one model group per logical model (e.g., "recommendation-model", "fraud-detection-model"), with each trained version automatically registered as an incrementing version within its group. Use the CustomerMetadataMap to record pipeline version and dataset version information.
C) Use S3 bucket folders to organize models by version (e.g., s3://models/v1/model.tar.gz, s3://models/v2/model.tar.gz) and skip Model Registry entirely for simplicity.
D) Create a new model group for each training run so that every version has its own group, making it easier to query individual training runs.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: SageMaker Model Registry is designed to organize models by model groups, where each group represents a logical model and contains versioned iterations of that model. This aligns with the requirement for multiple models in production and clear identification. Using CustomerMetadataMap to record pipeline version and dataset version enables compliance queries without breaking the registry's built-in versioning.

Distractor Analysis

A is Incorrect: Using a single model group for all models conflates different models under one versioning namespace. This makes it impossible to distinguish between recommendation model v2 and fraud detection model v2, and breaks the registry's group-based organization model.
C is Incorrect: Skipping Model Registry entirely loses approval workflows, lineage tracking, and the deployment integration that the registry provides. S3 folder organization alone cannot support the audit and governance requirements.
D is Incorrect: Creating a new model group for each training run defeats the purpose of versioning within a group. This would create thousands of fragmented groups, making it impossible to track the evolution of a single model over time.

Key Takeaway: Use one SageMaker Model Registry group per logical model, with automatic version increments within the group. Store metadata like pipeline version and dataset version in CustomerMetadataMap for compliance queries.

Question 10 - Cross-Team Model Discovery and Reuse

A large organization has multiple data science teams, each building models for different business units. The ML platform team wants to enable model discovery and reuse across teams using SageMaker Model Registry.

A data scientist on the pricing team needs to find a customer-segmentation model developed by the marketing team to use as a feature extractor in their pricing model.

Which approach enables the pricing team to discover and reuse the marketing team's model with minimal friction?

A) Search across all model groups in SageMaker Model Registry by model name and description, identify the marketing team's segmentation model, note the model package ARN, and reference that ARN in the pricing model's training script for feature extraction.
B) Ask the marketing team to export their model weights to a shared S3 bucket with a README file describing how to use the model.
C) Create a public endpoint for the marketing model and have the pricing team call the endpoint during training to extract features.
D) Require the pricing team to re-implement the segmentation model from scratch using the marketing team's published research document.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

A is Correct: SageMaker Model Registry serves as a centralized catalog where model groups, descriptions, and metadata are searchable across the organization. By finding the model group and its approved version ARN, the pricing team can reference the exact same model artifact in their training pipeline (e.g., downloading the model from the registry for feature extraction before training), ensuring reproducibility and version consistency.

Distractor Analysis

B is Incorrect: Ad-hoc S3 exports lose the governance, versioning, and approval status that the registry provides. The pricing team would have no way to verify which version was approved or whether the artifact was tampered with.
C is Incorrect: Creating and maintaining a separate public endpoint for feature extraction during training introduces unnecessary operational overhead, latency, and cost. Direct artifact access from the registry is more efficient for offline feature extraction during training.
D is Incorrect: Re-implementing models from documentation defeats the purpose of model reuse and introduces inconsistency. This approach is costly, error-prone, and fails to leverage the organization's existing approved model assets.

Key Takeaway: SageMaker Model Registry functions as an enterprise-wide model catalog, enabling teams to discover, reference, and reuse approved model versions by ARN without manual artifact transfers.

Key Themes Across Questions

Theme	Best Practice
Lineage Tracking	SageMaker Model Registry automatically maintains immutable lineage links from model packages back to training jobs, datasets, and hyperparameters
MLflow Integration	MLflow on SageMaker AI tracks experiment runs and can register winning runs directly into the Registry via API
Approval Governance	Use Registry approval status (`Approved`/`PendingManualApproval`) as a pipeline gate before deployment
Cross-Account Sharing	Share model groups and packages via RAM or resource policies to deploy the exact same version across environments
Rollback	Reference previous model package version ARN from the Registry when manual rollback is needed after deployment
Audit Trail	CloudTrail + Registry approval events + `CustomerMetadataMap` provide complete auditability without custom code
Versioning Strategy	One model group per logical model; store pipeline/dataset version in metadata for compliance queries
Model Discovery	Registry serves as a searchable enterprise catalog for cross-team model reuse