Skill 3.3.6: Manage model versions for repeatability and audits
| Question | Answer & Explanation |
|---|---|
| ### Question 1 - Integrating MLflow with SageMaker Model Registry A machine learning team is using MLflow on Amazon SageMaker AI to track experiment runs, hyperparameters, and evaluation metrics for a predictive maintenance model. The team wants to automatically register a successfully trained model into the Amazon SageMaker Model Registry directly from the MLflow tracking environment to ensure lineage is preserved with the least operational overhead. Which approach satisfies these requirements with the LEAST operational overhead? - A) Export the model artifacts from MLflow to an Amazon S3 bucket, and manually create a model package in the SageMaker Model Registry using the SageMaker Python SDK. - B) Use the mlflow.sagemaker.register_model function within the training script to automatically create a model package version in the SageMaker Model Registry.- C) Configure an AWS Lambda function triggered by an S3 event on the MLflow artifact bucket to invoke the CreateModelPackage API via AWS CLI.- D) Deploy an AWS Step Functions workflow that polls the MLflow tracking server for completed runs and pushes the metrics to the SageMaker Model Registry. |
Correct Answer: B Correct Answer Analysis: - B is Correct: MLflow on SageMaker AI natively supports integration with the SageMaker Model Registry. By using the mlflow.sagemaker.register_model API (or similar integration methods depending on the specific MLflow version configuration on SageMaker), you can seamlessly register the model artifacts from an MLflow run directly into the SageMaker Model Registry, preserving all lineage, parameters, and metrics automatically.Distractor Analysis: - A is Incorrect: Manually exporting artifacts and creating model packages introduces unnecessary operational overhead and breaks the automated lineage tracking between the experiment run and the registered model. - C is Incorrect: Triggering a Lambda function via S3 events requires writing and maintaining custom integration code, which is more complex than using the native MLflow-SageMaker integration. - D is Incorrect: Using a polling mechanism with Step Functions adds unnecessary architectural complexity and cost compared to a native API call within the training script. > Key Takeaway: MLflow on SageMaker AI provides native API support to register models directly into the SageMaker Model Registry, maintaining end-to-end lineage without requiring custom ETL or orchestration code. |
| ### Question 2 - Gating Deployment on Model Approval Status An enterprise data science team uses Amazon SageMaker Pipelines for their MLOps workflow. The pipeline trains a model, evaluates it, and registers it into the Amazon SageMaker Model Registry. The DevOps team has a strict policy: a model can only be deployed to a production SageMaker endpoint if it has been manually verified and marked as "Approved" by an internal compliance board. The CI/CD pipeline must not automate the approval process. How should the deployment stage be configured to enforce this compliance requirement with the least operational overhead? - A) Configure the deployment stage to query the Model Registry using the DescribeModelPackage API and only proceed if ModelApprovalStatus is Approved.- B) Use an AWS Lambda function to automatically approve the model version if the evaluation metrics exceed a defined threshold, then deploy. - C) Store the model evaluation metrics in Amazon DynamoDB and have the deployment stage query DynamoDB to verify human approval. - D) Configure Amazon EventBridge to automatically trigger the CodePipeline deployment stage as soon as a new model version is created, regardless of status. |
Correct Answer: A Correct Answer Analysis: - A is Correct: The SageMaker Model Registry maintains a ModelApprovalStatus (e.g., PendingManualApproval, Approved, Rejected) for each model version. A deployment pipeline can be configured to gate deployment by checking this status using the DescribeModelPackage API, ensuring that only manually approved models reach production.Distractor Analysis: - B is Incorrect: Automatically approving the model based on metrics bypasses the manual verification requirement mandated by the compliance board. - C is Incorrect: Using DynamoDB to track human approval introduces redundant data storage and custom logic, duplicating the built-in approval functionality already provided by the SageMaker Model Registry. - D is Incorrect: Triggering the deployment immediately upon model version creation bypasses the required manual approval gate. > Key Takeaway: The SageMaker Model Registry's ModelApprovalStatus provides a native gating mechanism for CI/CD pipelines, allowing human-in-the-loop approvals before production deployment. |
| ### Question 3 - Auditing Model Lineage for Regulatory Compliance A financial services company must comply with strict regulatory requirements that mandate the ability to trace any deployed predictive model back to its exact training data, hyperparameters, and the specific SageMaker Processing job that transformed the data. The team uses Amazon SageMaker Pipelines and the SageMaker Model Registry. Which configuration provides the necessary audit trail with the least effort? - A) Manually document the training job name and S3 data paths in a README file stored alongside the model artifacts in Amazon S3. - B) Enable SageMaker Model Registry lineage tracking, which automatically associates the model package with the specific pipeline execution step, training job, and processing job artifacts. - C) Use Amazon CloudTrail to track all API calls made to CreateTrainingJob and CreateModelPackage and cross-reference timestamps during an audit.- D) Export SageMaker Pipelines execution logs to Amazon OpenSearch Service and build a custom dashboard to trace model lineage. |
Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker Pipelines and the Model Registry natively support lineage tracking. When a model is registered via the pipeline, the Model Registry automatically captures the lineage, linking the model package to the specific pipeline execution, the training job, the processing job, and the input data locations. This provides a ready-to-use audit trail. Distractor Analysis: - A is Incorrect: Manual documentation is error-prone, scales poorly, and does not meet the automated repeatability standards required for enterprise regulatory compliance. - C is Incorrect: While CloudTrail tracks API calls, using it to reconstruct complex lineage relationships across multiple jobs and pipeline steps requires significant manual correlation effort and lacks context. - D is Incorrect: Exporting logs to OpenSearch and building a custom dashboard is a highly operationally heavy approach for something that SageMaker natively tracks and visualizes in the Studio UI. > Key Takeaway: SageMaker natively captures end-to-end ML lineage across Pipelines, Processing jobs, Training jobs, and the Model Registry, enabling effortless regulatory audits and reproducibility. |
| ### Question 4 - Organizing Models in the SageMaker Model Registry An AI engineering team is building multiple distinct models: one for predicting customer churn (XGBoost), one for image classification (ResNet), and a fine-tuned Llama 2 model for customer service chatbots. The team needs to organize these models in the Amazon SageMaker Model Registry to ensure that CI/CD pipelines retrieve the correct model type and version for their respective endpoints. How should the engineer structure the Model Registry to achieve this with the least confusion? - A) Create a single model group named "ProductionModels" and register all models as sequential versions (v1, v2, v3) within that group. - B) Create separate model groups for each model type (e.g., churn-xgboost-group, image-resnet-group, chatbot-llama-group) and register versions within their respective groups.- C) Store all models in the same Amazon S3 bucket and use S3 metadata tags to differentiate them, bypassing the Model Registry entirely. - D) Create a separate AWS account for each model type and use the Model Registry within each account to store only one version at a time. |
Correct Answer: B Correct Answer Analysis: - B is Correct: A model group in the SageMaker Model Registry is designed to group versions of the same model or models serving the same purpose. Creating distinct model groups for distinct model use cases allows CI/CD pipelines to query a specific group and pull the latest approved version without ambiguity. Distractor Analysis: - A is Incorrect: Putting entirely different models (XGBoost, ResNet, Llama) into the same model group breaks the semantic meaning of a model group and makes querying the latest version highly ambiguous and error-prone. - C is Incorrect: Bypassing the Model Registry removes the ability to track approval status, lineage, and deployment metrics natively. - D is Incorrect: Using separate AWS accounts for each model type is an extreme over-architecting of the solution, introducing massive overhead in networking, billing, and cross-account deployment. > Key Takeaway: In the SageMaker Model Registry, a "Model Group" represents a family of versions for a specific ML use case or model architecture, enabling organized version control and unambiguous deployment retrieval. |
| ### Question 5 - Automating Deployment Triggers upon Approval A DevOps team has set up a CI/CD pipeline using AWS CodePipeline to deploy models to a SageMaker Asynchronous endpoint. To ensure quality control, a data scientist must manually approve new model versions in the Amazon SageMaker Model Registry before they are deployed. The team wants the CI/CD pipeline to initiate the deployment automatically only after the data scientist updates the approval status. Which AWS service can be used to reliably trigger the CodePipeline execution based on this approval event? - A) Configure an Amazon EventBridge rule to listen for the SageMaker Model Package State Change event where the status is Approved, and use it to start the CodePipeline execution.- B) Use an AWS Lambda function that polls the Model Registry every 5 minutes to check for newly approved models and manually triggers CodePipeline. - C) Configure Amazon CloudWatch Logs to monitor API calls to UpdateModelPackage and trigger an SNS notification that starts CodePipeline.- D) Use AWS CloudTrail to detect the approval event and send an alert to an administrator who manually clicks "Release Change" in CodePipeline. |
Correct Answer: A Correct Answer Analysis: - A is Correct: Amazon SageMaker emits events to Amazon EventBridge when the status of a model package changes. By creating an EventBridge rule that filters for events where the model package approval status changes to Approved, you can seamlessly trigger an event-driven execution of a CodePipeline deployment, fulfilling the automation requirement with zero polling overhead.Distractor Analysis: - B is Incorrect: Polling the Model Registry with Lambda is an anti-pattern. It consumes unnecessary compute resources and introduces latency (up to 5 minutes) compared to an event-driven approach. - C is Incorrect: CloudWatch Logs is for log ingestion and analysis, not for routing AWS service state change events. EventBridge is the correct service for AWS event routing. - D is Incorrect: Requiring manual intervention to release the change defeats the purpose of automated CI/CD triggers upon approval. > Key Takeaway: Amazon EventBridge is the native mechanism to create event-driven MLOps workflows in AWS, such as triggering a deployment pipeline immediately when a model version is approved in the SageMaker Model Registry. |
| ### Question 6 - Enterprise Security for MLflow Tracking Server An enterprise machine learning team is deploying MLflow on Amazon SageMaker AI to track experiments across multiple teams. The security team mandates that all communications to the MLflow tracking server must be encrypted in transit using TLS, and the tracking server must not be accessible directly from the public internet. Access should be restricted to authenticated IAM principals within the VPC. Which configuration satisfies these security and compliance requirements? - A) Deploy the MLflow tracking server in a private subnet within a VPC, configure a VPC endpoint (PrivateLink) for SageMaker, and ensure security groups only allow traffic from specific IAM roles over HTTPS. - B) Deploy the MLflow tracking server in a public subnet, enable TLS on the Application Load Balancer, and restrict access using an IP allowlist. - C) Run the MLflow tracking server locally on each data scientist's laptop and sync the tracking data to an S3 bucket daily. - D) Deploy the MLflow tracking server on an Amazon EC2 instance in a public subnet and use AWS WAF to block traffic from outside the corporate VPN CIDR block. |
Correct Answer: A Correct Answer Analysis: - A is Correct: When using MLflow on SageMaker, you can configure the tracking server to run within a VPC. By placing it in a private subnet and utilizing AWS PrivateLink (VPC Endpoints) for SageMaker, you ensure traffic stays entirely within the AWS network, encrypts data in transit (TLS), and is inaccessible from the public internet. Distractor Analysis: - B is Incorrect: Deploying in a public subnet exposes the server to the internet, even with an IP allowlist, violating the requirement to not be accessible from the public internet. - C is Incorrect: Local execution on laptops does not provide a centralized, secure tracking server for the enterprise and breaks the MLOps workflow. - D is Incorrect: Managing your own EC2 instance for MLflow contradicts the managed "MLflow on SageMaker AI" requirement, and using WAF does not fully isolate the server from public internet routing. > Key Takeaway: For enterprise ML tracking, deploy MLflow on SageMaker within a private VPC and use VPC Endpoints (PrivateLink) to ensure traffic is encrypted, private, and isolated from the public internet. |
| ### Question 7 - Rolling Back to a Previous Model Version A production SageMaker real-time endpoint is serving traffic for an e-commerce recommendation engine using model version 5 from a specific SageMaker Model Registry group. The newly deployed version 6 is causing degraded performance and increased latency. The operations team needs to immediately revert the endpoint to the stable version 5 with the least operational overhead and minimal disruption to traffic. Which action achieves this rollback? - A) Update the existing SageMaker endpoint configuration to point to the model artifact S3 URI of version 5 and save the configuration. - B) Use the UpdateEndpoint API to deploy a new endpoint configuration that references the model package ARN of the approved version 5.- C) Delete the endpoint and recreate it from scratch using the SageMaker Model Registry version 5. - D) Download the artifacts of version 5 to a local machine, create a new Docker container, and deploy it to an Amazon ECS cluster behind an Application Load Balancer. |
Correct Answer: B Correct Answer Analysis: - B is Correct: Using the UpdateEndpoint API (or the SageMaker console) to deploy a new endpoint configuration that references the specific Model Registry package ARN for version 5 allows SageMaker to natively handle the blue/green deployment for the rollback. This ensures zero downtime and utilizes the versioned artifacts stored in the registry.Distractor Analysis: - A is Incorrect: While updating the endpoint configuration is conceptually correct, directly pointing to an S3 URI bypasses the Model Registry's managed metadata, container image definitions, and environment variables stored within the Model Package. It is better to reference the Model Package ARN. - C is Incorrect: Deleting and recreating the endpoint causes downtime and disrupts production traffic, violating the minimal disruption constraint. - D is Incorrect: Deploying to ECS using custom containers introduces massive operational overhead and complexity compared to using SageMaker's native hosting capabilities. > Key Takeaway: SageMaker endpoints can be seamlessly rolled back by using the UpdateEndpoint API to target a previous, approved model version (identified by its Model Package ARN) from the Model Registry, leveraging native blue/green deployment. |
| ### Question 8 - Versioning Fine-Tuned Foundation Models An AI team has fine-tuned a foundation model (Llama 3) using Amazon SageMaker JumpStart for a medical document summarization task. The team wants to catalog this fine-tuned model in the SageMaker Model Registry so they can track performance across different fine-tuning datasets and deploy the best version to a SageMaker endpoint. Which step is required to successfully register this fine-tuned model into the Model Registry? - A) Convert the fine-tuned model weights into a SageMaker-compatible .tar.gz artifact, upload it to Amazon S3, and create a model package specifying the appropriate inference container image.- B) Push the fine-tuned weights directly to Amazon ECR as a Docker image and register the ECR repository ARN in the Model Registry. - C) Register the model by providing the original base model's JumpStart model ID, as the Model Registry does not support custom fine-tuned artifacts. - D) Deploy the fine-tuned model to an endpoint first, and the Model Registry will automatically capture the running endpoint configuration as a new version. |
Correct Answer: A Correct Answer Analysis: - A is Correct: To register any model in the SageMaker Model Registry, you need the model artifacts packaged as a .tar.gz file in Amazon S3, along with the inference container image URI (either a SageMaker pre-built container or a custom one). For a fine-tuned foundation model, the resulting weights must be packaged this way to be registered as a model package version.Distractor Analysis: - B is Incorrect: While you need an inference container image (which can be a pre-built SageMaker DLC), the actual model weights (artifacts) are not Docker images; they are separate files stored in S3 and loaded by the container at runtime. - C is Incorrect: The SageMaker Model Registry fully supports custom fine-tuned artifacts. You must register the new weights, not just the base model ID. - D is Incorrect: The Model Registry does not automatically capture running endpoint configurations. Models must be explicitly registered via the SDK, Pipelines, or the console. > Key Takeaway: Registering a model in the SageMaker Model Registry requires linking an S3 model artifact ( .tar.gz) with a compatible container image URI, capturing the exact state of fine-tuned weights for reproducible deployments. |
| ### Question 9 - Automating Registration via SageMaker Pipelines A machine learning engineer is building an automated retraining pipeline using Amazon SageMaker Pipelines. The pipeline processes new data, trains an XGBoost model, and evaluates its F1 score. To ensure the Model Registry is not cluttered with poor-performing models, the engineer wants the pipeline to only register a new model version if the F1 score is strictly greater than 0.85. How can this conditional logic be implemented natively within the SageMaker Pipeline? - A) Use a ConditionStep in the pipeline that evaluates the F1 score from the EvaluationStep properties, and triggers a RegisterModelStep only if the condition is met.- B) Configure the TrainingStep to throw an exception if the F1 score is less than 0.85, causing the pipeline to fail before reaching registration.- C) Use an AWS Lambda function to check the F1 score and manually invoke the RegisterModel API outside of the SageMaker Pipeline.- D) Use the FailStep to halt the pipeline immediately after training, and require a manual review before adding it to the Model Registry. |
Correct Answer: A Correct Answer Analysis: - A is Correct: Amazon SageMaker Pipelines supports ConditionStep. You can define a condition (e.g., F1 > 0.85) that evaluates the metrics output from an EvaluationStep. Based on the boolean result, the pipeline can selectively execute downstream steps, such as a RegisterModelStep, ensuring only quality models are versioned in the registry.Distractor Analysis: - B is Incorrect: Throwing an exception to fail the pipeline is a hard stop. It prevents cleanup steps or other branches from running and does not provide graceful handling of low-performing models. - C is Incorrect: While AWS Lambda can be used in a pipeline via LambdaStep, it introduces unnecessary custom code and operational overhead when ConditionStep natively supports this exact use case.- D is Incorrect: Requiring manual intervention defeats the purpose of an automated CI/CD pipeline and conditional registration logic. > Key Takeaway: Use ConditionStep in SageMaker Pipelines to evaluate model metrics and gate downstream actions like RegisterModelStep, ensuring only models that meet your quality thresholds are added to the Model Registry. |
| ### Question 10 - Comparing SageMaker Model Registry and MLflow Model Registry A data science team is using MLflow on SageMaker AI for experiment tracking. They want to manage model versions and deployments using Amazon SageMaker Model Registry to leverage native SageMaker deployment capabilities (like deployment guardrails and shadow testing). However, they also want to maintain their tracking lineage in MLflow. Which TWO actions should the team take to seamlessly connect MLflow experiments with the SageMaker Model Registry? (Select TWO) - A) Use the mlflow.sagemaker.register_model function to register the MLflow run's artifacts directly into a SageMaker Model Registry model group.- B) Ensure the MLflow tracking server has the necessary IAM permissions to call the CreateModelPackage and CreateModelPackageGroup SageMaker APIs.- C) Manually copy the MLflow model artifacts from the MLflow tracking S3 bucket to a new S3 bucket managed by SageMaker before registering. - D) Deploy the MLflow model directly from the MLflow registry to an Amazon EC2 instance, bypassing SageMaker hosting entirely. - E) Use Amazon EventBridge to monitor MLflow experiment runs and automatically push the artifacts to the SageMaker Model Registry via AWS Step Functions. |
Correct Answers: A and B Correct Answer Analysis: - A is Correct: The mlflow.sagemaker.register_model API (or the equivalent MLflow on SageMaker integration method) natively bridges MLflow and the SageMaker Model Registry, creating a model package version that points to the MLflow artifact while preserving lineage.- B is Correct: For the MLflow tracking server to successfully interact with the SageMaker Model Registry and create model packages, the IAM execution role attached to the MLflow SageMaker environment must have permissions for SageMaker Model Registry actions (e.g., sagemaker:CreateModelPackage).Distractor Analysis: - C is Incorrect: Copying artifacts manually breaks the automated lineage tracking and introduces unnecessary operational overhead. The native integration points the Model Registry directly to the MLflow artifacts. - D is Incorrect: The requirement explicitly asks to use SageMaker Model Registry and native SageMaker deployment capabilities (deployment guardrails, shadow testing), which require hosting on SageMaker endpoints, not EC2. - E is Incorrect: Using EventBridge and Step Functions to monitor MLflow and push to SageMaker is a complex, over-engineered workaround when a direct API integration exists. > Key Takeaway: MLflow on SageMaker integrates directly with the SageMaker Model Registry via native APIs, provided the execution role has the correct IAM permissions, allowing seamless tracking and deployment without custom data movement. |