Skill 3.3.10: Configure FM deployment automation with fine-tuned model versioning
| Question | Answer & Explanation |
|---|---|
| ### Question 1 - Automating Bedrock Fine-Tuned Model Deployment & Rollback An enterprise AI team has fine-tuned a foundation model using Amazon Bedrock. They need to automate the deployment of this fine-tuned model for inference and ensure that previous versions are retained. The CI/CD pipeline must provision the model and route traffic with the least operational overhead, while retaining the ability to instantly shift traffic back to the previous version if latency degrades. Which ONE approach fulfills these requirements? - A) Deploy the fine-tuned model to an Amazon SageMaker real-time endpoint and use AWS CodeDeploy to manage the traffic shifting. - B) Create an Amazon Bedrock Provisioned Throughput for the fine-tuned model and use an alias to route traffic, updating the alias target to the new model version during deployment. - C) Host the fine-tuned model on an Amazon EC2 instance with an Application Load Balancer and manually update the target group weights. - D) Use an AWS Lambda function to invoke the fine-tuned model directly and maintain a mapping of model versions in an Amazon DynamoDB table. |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock Provisioned Throughput allows you to allocate capacity for fine-tuned models. By using an alias, you can map a stable endpoint reference to a specific model version. During a deployment, you update the alias to point to the new model version. If issues arise, you simply point the alias back to the previous version, achieving instant rollback with minimal operational overhead. Distractor Analysis: - A is Incorrect: Deploying a Bedrock fine-tuned model to a SageMaker endpoint is not a native Bedrock pattern; SageMaker endpoints are typically used for models trained or hosted directly within SageMaker, introducing unnecessary operational overhead. - C is Incorrect: Hosting the model on EC2 instances requires managing the underlying compute, OS, and inference containers, which violates the "least operational overhead" constraint. - D is Incorrect: Using Lambda and DynamoDB to route traffic to model versions is an anti-pattern that introduces significant custom logic and operational complexity, and Lambda is not suited for hosting or routing long-running inference traffic. > Exam Tip: Bedrock Provisioned Throughput aliases provide a native, low-overhead mechanism for version routing and instant rollback for fine-tuned FMs. |
| ### Question 2 - SageMaker Pipelines Model Registration for FMs A machine learning team uses SageMaker Pipelines to fine-tune a Hugging Face foundation model. The pipeline must automatically register the fine-tuned model as a new version in a registry only if its F1 score on a hold-out dataset exceeds 0.85. The solution must minimize operational overhead and avoid custom Lambda functions for condition checking. Which TWO steps should the engineer include in the SageMaker Pipeline to achieve this? (Select TWO) - A) Add a Condition step that evaluates the evaluation metric against the 0.85 threshold. - B) Add a RegisterModel step that depends on the Condition step's success. - C) Add an AWS Lambda step that parses the evaluation report and manually calls the SageMaker Model Registry API. - D) Add a CreateModel step that runs unconditionally to package the model for deployment. - E) Add a Fail step that triggers if the F1 score is above 0.85 to prevent deployment. |
Correct Answers: A and B Correct Answer Analysis: - A is Correct: SageMaker Pipelines natively supports a Condition step, which allows you to define rules (e.g., GreaterThan) based on properties from previous steps (like an evaluation step). This avoids custom code for gating logic.- B is Correct: The RegisterModel step natively packages the model artifacts and registers them as a new version in the SageMaker Model Registry. By making its execution dependent on the Condition step's true outcome, the model is only registered if the metric threshold is met.Distractor Analysis: - C is Incorrect: Using a Lambda step to parse metrics and call the Registry API introduces unnecessary custom code and operational overhead compared to the native Condition and RegisterModel steps.- D is Incorrect: While a CreateModel step is useful for creating an endpoint, registering the model conditionally is the specific requirement here. Creating the model unconditionally does not enforce the quality gate.- E is Incorrect: A Fail step would intentionally break the pipeline if the score was good, which is the opposite of the desired behavior.> Exam Tip: Use SageMaker Pipelines Condition steps to gate RegisterModel steps on evaluation metrics, ensuring only validated models are versioned in the Model Registry. |
| ### Question 3 - CI/CD Deployment of SageMaker Registry Models A CI/CD pipeline in AWS CodePipeline must deploy a newly approved fine-tuned model version from the SageMaker Model Registry to a SageMaker real-time endpoint. The deployment must minimize downtime and automatically route traffic using a blue/green deployment strategy without writing custom traffic-shifting scripts. Which ONE action meets these requirements with the least operational overhead? - A) Use an AWS CodeBuild stage to run a Python script using boto3 to call UpdateEndpointWeightsAndCapacities for manual traffic shifting.- B) Use an AWS CodeDeploy stage to manage the blue/green deployment of the SageMaker endpoint. - C) Use an AWS CodeBuild stage to invoke the UpdateEndpoint API with a BlueGreenDeploymentConfig specifying the new model.- D) Configure an AWS Lambda function to terminate the old endpoint configuration and create a new one. |
Correct Answer: C Correct Answer Analysis: - C is Correct: The SageMaker UpdateEndpoint API natively supports blue/green deployments via BlueGreenDeploymentConfig. By invoking this API from a CodeBuild stage within CodePipeline, you leverage SageMaker's managed Deployment Guardrails to automatically provision the new fleet, shift traffic, and terminate the old fleet without custom scripting.Distractor Analysis: - A is Incorrect: UpdateEndpointWeightsAndCapacities is used for shifting traffic between variants, but managing a full blue/green deployment lifecycle this way requires heavy custom orchestration and scripting, violating the "least operational overhead" constraint.- B is Incorrect: AWS CodeDeploy is used for deploying applications to EC2, ECS, and Lambda, but it is not the native mechanism for managing SageMaker endpoint blue/green deployments. - D is Incorrect: Manually deleting and recreating endpoints via Lambda causes downtime and requires extensive custom error handling, failing the "minimize downtime" and "least overhead" constraints. > Exam Tip: SageMaker's UpdateEndpoint API with BlueGreenDeploymentConfig provides native, zero-downtime deployments triggered easily from a CI/CD pipeline's CodeBuild stage. |
| ### Question 4 - Event-Driven Retraining and Auto-Registration A retail company fine-tunes a foundation model on SageMaker when new product catalog data arrives in an Amazon S3 bucket. The retraining must be triggered automatically, and the resulting model must be versioned and await manual approval before any deployment occurs. The architecture must be fully automated up to the registration point. Which TWO components should the engineer configure to fulfill these requirements? (Select TWO) - A) Configure an Amazon EventBridge rule to listen for S3 ObjectCreated events and use it to trigger StartPipelineExecution for the SageMaker Pipeline.- B) Configure an AWS CodePipeline schedule to run the SageMaker training pipeline daily. - C) In the SageMaker Pipeline, configure a RegisterModel step with ApprovalStatus="PendingManualApproval".- D) Configure the SageMaker Pipeline to automatically deploy the model to a staging endpoint upon successful training. - E) Use an S3 Event Notification to directly invoke an AWS Lambda function that executes the training script on Amazon EC2. |
Correct Answers: A and C Correct Answer Analysis: - A is Correct: Amazon EventBridge can capture S3 event notifications (e.g., when new data is uploaded) and route them to trigger a SageMaker Pipeline execution. This creates an event-driven retraining mechanism that reacts only when new data arrives. - C is Correct: Setting the ApprovalStatus to PendingManualApproval in the RegisterModel step ensures the model is versioned in the Model Registry but will not be deployed automatically. A human must explicitly approve the version before it can proceed to production.Distractor Analysis: - B is Incorrect: A daily schedule wastes compute resources when no new data has arrived and might delay retraining if data arrives mid-day, violating the requirement for an event-driven trigger upon data arrival. - D is Incorrect: The scenario explicitly requires the model to await manual approval before deployment. Automatically deploying to a staging endpoint bypasses this control flow. - E is Incorrect: Using Lambda and EC2 to run training scripts introduces massive operational overhead compared to using SageMaker Pipelines natively. > Exam Tip: Event-driven retraining uses EventBridge to trigger SageMaker Pipelines, and PendingManualApproval in the RegisterModel step ensures a human gates the deployment. |
| ### Question 5 - Canary Rollbacks for FM Endpoints An enterprise is deploying a fine-tuned foundation model to a SageMaker endpoint. They require a canary deployment strategy where 10% of traffic is shifted to the new model version for 5 minutes. If CloudWatch alarms detect an increase in 5XX errors during this baking period, the deployment must automatically roll back. Which ONE configuration achieves this automated rollback behavior? - A) Configure a linear traffic shift with a manual rollback trigger from a developer's AWS CLI terminal. - B) Configure a canary traffic shift with an AutoRollbackConfig that includes a CloudWatch alarm monitoring Invocation5XXErrors.- C) Configure an all-at-once traffic shift and rely on Amazon EventBridge to revert the endpoint configuration if errors occur. - D) Configure a rolling update on the endpoint instances and enable Detailed Monitoring in CloudWatch. |
Correct Answer: B Correct Answer Analysis: - B is Correct: SageMaker Deployment Guardrails support canary deployments that shift a small percentage of traffic (e.g., 10%) to the new fleet for a baking period. By linking a CloudWatch alarm (e.g., monitoring Invocation5XXErrors) to the AutoRollbackConfig, SageMaker will automatically trigger a rollback and revert traffic to the blue fleet if the alarm trips during the canary window.Distractor Analysis: - A is Incorrect: A linear traffic shift moves traffic in even steps, not a single 10% slice, and manual CLI rollback violates the requirement for automated rollback. - C is Incorrect: An all-at-once traffic shift exposes 100% of traffic to the new model immediately, defeating the purpose of a canary deployment, and EventBridge does not natively manage endpoint rollbacks without custom logic. - D is Incorrect: A rolling update replaces capacity in batches but does not perform blue/green canary traffic shifting, and Detailed Monitoring alone does not trigger a rollback. > Exam Tip: Automated rollbacks in SageMaker require both a traffic-shifting strategy (like Canary) and an AutoRollbackConfig linked to CloudWatch Alarms. |
| ### Question 6 - Evaluating Generative FMs in CI/CD A CI/CD pipeline for an FM application needs to test a newly fine-tuned generative model before deployment. Because the model is generative, exact string matching is not possible. The pipeline must evaluate the model against a golden dataset and block deployment if the response quality score drops below the baseline. The evaluation must run with minimal infrastructure management. Which ONE approach fits these requirements? - A) Deploy the model to an EC2 instance and run custom Python scripts to perform exact string matching against unit tests. - B) Use Amazon Bedrock Model Evaluation jobs within the CI/CD pipeline to evaluate the model against a custom dataset and gate the deployment on the resulting metrics. - C) Deploy the model directly to production using a canary shift and monitor user feedback via an AWS Lambda function. - D) Run the application's existing unit tests against the inference API code to ensure the API responds with a 200 OK status. |
Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock Model Evaluation (or SageMaker Model Evaluation) allows you to run curated "golden prompt datasets" through the model and score the responses using metrics like faithfulness, toxicity, or LLM-as-a-judge. By invoking this as a step in the CI/CD pipeline, you can programmatically gate the deployment based on the evaluation metrics without managing underlying compute. Distractor Analysis: - A is Incorrect: Exact string matching is fundamentally incompatible with generative AI models, as responses naturally vary between runs. Furthermore, managing EC2 instances violates the minimal infrastructure constraint. - C is Incorrect: Deploying directly to production and relying on user feedback bypasses the CI/CD testing phase entirely and exposes users to a potentially degraded model. - D is Incorrect: Unit tests on the API code verify infrastructure and routing, but they do not evaluate the semantic quality or accuracy of the model's generated responses. > Exam Tip: For generative AI, CI/CD quality gates must use evaluation frameworks (like Bedrock Model Evaluation) rather than deterministic unit tests. |
| ### Question 7 - Auditing Fine-Tuned FM Lineage An investigation must establish exactly which fine-tuned foundation model version served traffic during a past incident, and which specific training job and dataset produced it. The team needs a centralized service to record metadata, lineage, and approval status for repeatability and audits. Which ONE AWS service supports this requirement natively for SageMaker fine-tuned models? - A) Amazon ECR image tags for the inference containers. - B) AWS CloudTrail logs for the InvokeEndpoint API calls.- C) Amazon SageMaker Model Registry. - D) Amazon EventBridge archive. |
Correct Answer: C Correct Answer Analysis: - C is Correct: SageMaker Model Registry catalogs trained models as versions inside a model group. It natively records metadata, lineage (links back to the exact training job and S3 data URIs), and approval status. This allows auditors to trace exactly which model version was approved and deployed, and how it was produced. Distractor Analysis: - A is Incorrect: ECR image tags track the serving container environment (e.g., PyTorch image version), not the specific model weights or the training job lineage. - B is Incorrect: CloudTrail logs record API invocations (who called the endpoint and when), but they do not store the lineage of the model artifacts or their training metadata. - D is Incorrect: EventBridge archives capture event bus traffic, but they are not a structured registry for model versioning and lineage tracking. > Exam Tip: SageMaker Model Registry is the single source of truth for model versions, lineage, and approval status, enabling full auditability of ML deployments. |
| ### Question 8 - Versioning FMs and Prompts in CI/CD A generative AI application uses a fine-tuned foundation model alongside a complex prompt template. The prompt template frequently changes to adjust behavior. The CI/CD pipeline must version both the fine-tuned model and the prompt so that an audit can reconstruct exactly what configuration served traffic at any given time. Which TWO actions should the engineer take to implement versioning for both artifacts? (Select TWO) - A) Store the prompt template as an environment variable in the AWS Lambda function configuration. - B) Use Amazon Bedrock Prompt Management to create, version, and deploy the prompt template. - C) Hardcode the prompt template in the application source code on AWS CodeCommit. - D) Register the fine-tuned model as a versioned model package in the SageMaker Model Registry. - E) Store the prompt template as a plaintext file in an Amazon S3 bucket with object versioning enabled. |
Correct Answers: B and D Correct Answer Analysis: - B is Correct: Amazon Bedrock Prompt Management allows prompts to be stored as versioned resources with variables. The application refers to a specific prompt version (or an alias), ensuring prompt changes are explicitly released and auditable, rather than edited in place. - D is Correct: Registering the fine-tuned model in the SageMaker Model Registry ensures the model weights and artifacts are versioned, carrying an approval status that the pipeline can read before deployment. Distractor Analysis: - A is Incorrect: Storing prompts as Lambda environment variables makes versioning and auditing difficult, as changes require updating the Lambda configuration rather than routing through a managed versioning system. - C is Incorrect: Hardcoding prompts in source code couples the prompt lifecycle to the application code lifecycle, which violates the separation of concerns and makes prompt-specific auditing difficult. - E is Incorrect: While S3 versioning preserves file history, it lacks the native integration, variable templating, and deployment aliasing features of Bedrock Prompt Management for prompt lifecycle management. > Exam Tip: Both prompts and model weights are behavioral artifacts that must be versioned. Use Bedrock Prompt Management for prompts and SageMaker Model Registry for models. |
| ### Question 9 - Connecting External Repositories for FM CI/CD A company is using a third-party GitHub Enterprise repository to store their CI/CD pipeline code and configuration files for fine-tuning foundation models on Amazon SageMaker. The AWS CodePipeline needs to trigger on every commit to the repository without storing static credentials (like Personal Access Tokens) in the pipeline definition or AWS Secrets Manager. Which ONE AWS service provides the secure, managed link between CodePipeline and the external Git repository? - A) AWS CodeCommit. - B) AWS CodeConnections. - C) AWS CodeDeploy. - D) Amazon EventBridge. |
Correct Answer: B Correct Answer Analysis: - B is Correct: AWS CodeConnections (formerly AWS CodeStar Connections) provides a managed, secure connection resource between AWS services (like CodePipeline) and third-party code repositories hosted outside AWS (e.g., GitHub Enterprise, GitLab, Bitbucket). It establishes an OAuth or GitHub App-based connection handshake, allowing the pipeline to detect commits and fetch source code without embedding static credentials. Distractor Analysis: - A is Incorrect: AWS CodeCommit is an AWS-native Git repository service. It does not connect to third-party external repositories like GitHub Enterprise. - C is Incorrect: AWS CodeDeploy automates application deployments to compute targets; it does not manage source control connections. - D is Incorrect: While EventBridge can trigger pipelines, it does not establish the secure authentication and code-fetching link required for CodePipeline to read from an external Git repository. > Exam Tip: Use AWS CodeConnections to securely integrate AWS CodePipeline with third-party Git providers without managing static PATs. |
| ### Question 10 - Updating RAG Systems with Fine-Tuned Embedding Models An enterprise has a Retrieval-Augmented Generation (RAG) system using Amazon Bedrock Knowledge Bases. The team fine-tuned a new embedding model and wants to update the vector index. The system must reflect the new embeddings while minimizing the operational overhead of rebuilding the index infrastructure. The team needs a fully automated way to refresh the embeddings based on the fine-tuned model. Which ONE action fulfills these requirements? - A) Manually copy the existing OpenSearch Serverless index to a new collection and update the application endpoint. - B) Trigger an incremental ingestion job ( StartIngestionJob) in the Bedrock Knowledge Base, which will re-parse and re-embed the documents using the fine-tuned model.- C) Run an AWS Glue crawler to append the new embeddings to the existing OpenSearch index. - D) Update the Bedrock Knowledge Base data source to point to a new S3 bucket containing the same documents, which will automatically trigger a background re-index. |
Correct Answer: B Correct Answer Analysis: - B is Correct: When an embedding model is fine-tuned or changed, the existing vectors must be regenerated. In Amazon Bedrock Knowledge Bases, starting a new ingestion job ( StartIngestionJob) will re-process the data source. Since the embedding model has been updated in the Knowledge Base configuration, the ingestion job will parse, chunk, and re-embed the documents, updating the vector index with the new embeddings in a managed, automated fashion.Distractor Analysis: - A is Incorrect: Copying the index to a new collection does not regenerate the vectors; it merely duplicates the old, stale vectors produced by the previous embedding model. - C is Incorrect: AWS Glue crawlers are used for discovering and cataloging data schemas, not for orchestrating vector embedding generation or updating OpenSearch indexes with Bedrock. - D is Incorrect: Pointing to a new S3 bucket with the same documents is an anti-pattern. It does not inherently trigger a background re-index without an ingestion job, and it complicates data management without providing the automated embedding refresh of a native ingestion job. > Exam Tip: Changing an embedding model in a RAG system requires regenerating vectors. Trigger a Bedrock Knowledge Bases StartIngestionJob to re-process and re-embed the corpus using the new model. |