Skill 3.3.4: Configure automated testing strategies within CI/CD pipelines for traditional ML and AI workloads

---

Question	Answer & Explanation
### Question 1 - Separating Code and Model Tests in MLOps A company is building an MLOps pipeline using AWS CodePipeline and Amazon SageMaker Pipelines. The team must separate the testing of their Python data preprocessing scripts from the evaluation of the trained model's performance. They want to achieve this with the least operational overhead. Which TWO approaches should they implement to properly separate these testing domains? (Select TWO) - A) Use AWS CodeBuild to execute unit tests for the Python preprocessing scripts during the pipeline's Build stage. - B) Use an Amazon SageMaker Processing job within SageMaker Pipelines to evaluate the model's accuracy on a held-out validation dataset. - C) Use AWS CodeDeploy to run pytest on the trained model artifacts before deployment. - D) Use Amazon EventBridge to schedule automated model evaluation jobs independently of the pipeline execution.	Correct Answers: A and B Correct Options Analysis: - A is Correct: AWS CodeBuild is the standard service for running unit tests on application and pipeline code (like Python data preprocessing scripts). It executes deterministically and fails fast before any expensive ML compute resources are provisioned. - B is Correct: Amazon SageMaker Processing jobs are designed to run model evaluation scripts on managed compute infrastructure. When integrated into SageMaker Pipelines, this natively evaluates the model's performance metrics on a validation dataset, separating the model test from the code test. Distractor Analysis: - C is Incorrect: AWS CodeDeploy is used for deploying applications to compute services (like EC2, ECS, Lambda) and managing blue/green deployments, not for running unit tests like pytest on model artifacts. - D is Incorrect: Amazon EventBridge is an event router service. While it can trigger pipelines, it cannot run the evaluation tests itself, nor does it provide a native mechanism to evaluate model performance metrics. > Key Takeaway: In MLOps, code tests (unit tests for preprocessing logic) belong in CodeBuild, whereas model tests (accuracy, F1 score on holdout data) belong in SageMaker Processing jobs orchestrated by SageMaker Pipelines.
### Question 2 - Generative AI Response Quality Testing A company is deploying an LLM application using Amazon Bedrock. They need an automated testing strategy in their CI/CD pipeline to ensure that a new prompt version does not degrade response quality. The testing approach must handle the non-deterministic nature of LLM outputs and minimize operational overhead. Which approach should the engineering team implement? - A) Run a script in AWS CodeBuild to assert that the exact string output of the new prompt matches the expected baseline string. - B) Use Amazon Bedrock Model Evaluation to run a curated prompt dataset and compare aggregate semantic scores against the previous prompt version. - C) Deploy the new prompt to a canary endpoint and monitor CloudWatch latency metrics to determine if the prompt is performing well. - D) Use Amazon SageMaker Ground Truth to manually label the LLM outputs before approving the pipeline stage.	Correct Answer: B Correct Option Analysis: - B is Correct: Amazon Bedrock Model Evaluation supports automatic evaluation using LLM-as-a-judge or task-specific metrics (like faithfulness or toxicity). By running a curated prompt dataset through both the candidate and baseline prompt versions, the pipeline can compare aggregate semantic scores and block deployment if the candidate version's quality drops below a defined threshold. This is specifically designed for non-deterministic LLM outputs. Distractor Analysis: - A is Incorrect: LLM responses are non-deterministic; they vary between runs even for identical prompts. Exact string assertions will cause false negatives and flaky pipeline builds. - C is Incorrect: CloudWatch latency metrics measure infrastructure and network responsiveness, not the semantic quality or correctness of the LLM's generated text. - D is Incorrect: Using SageMaker Ground Truth introduces a manual human-in-the-loop labeling process, which violates the requirement for an automated CI/CD pipeline and increases operational overhead significantly. > Key Takeaway: For generative AI, exact string matching fails due to non-determinism. Use automated evaluation frameworks (like Bedrock Model Evaluation) to score responses against semantic criteria rather than asserting exact outputs.
### Question 3 - Gating Deployment on Model Metrics A data science team uses SageMaker Pipelines to orchestrate their ML workflow. The team needs to ensure that a newly trained model is automatically registered in the SageMaker Model Registry but blocked from being deployed to production unless its F1 score is strictly greater than 0.85. The solution must enforce this gate automatically with the least operational overhead. Which configuration achieves this? - A) Configure an Amazon EventBridge rule to check the F1 score and trigger the deployment stage if it passes. - B) Add a ConditionStep in SageMaker Pipelines that checks the evaluation metrics, followed by a RegisterModel step that only executes if the condition passes. - C) Use an AWS Lambda function in AWS CodePipeline to manually query the SageMaker API for the F1 score and transition the pipeline state. - D) Set up an Amazon CloudWatch Alarm on the F1 score to halt the pipeline execution.	Correct Answer: B Correct Option Analysis: - B is Correct: SageMaker Pipelines natively supports a ConditionStep. This step can evaluate the metrics output by a prior evaluation step (e.g., checking if F1Score > 0.85). If the condition evaluates to true, it allows the downstream RegisterModel step to execute. If false, the pipeline skips registration. This is a fully native, serverless approach with zero custom code overhead. Distractor Analysis: - A is Incorrect: Amazon EventBridge is an event router and cannot inherently parse ML evaluation metrics or manage SageMaker pipeline state transitions based on metric thresholds. - C is Incorrect: Writing a custom AWS Lambda function introduces unnecessary operational overhead and custom code maintenance when SageMaker Pipelines offers a native ConditionStep for exactly this purpose. - D is Incorrect: CloudWatch Alarms are used for runtime monitoring of deployed endpoints (like latency or 5XX errors), not for gating internal pipeline workflow steps based on batch evaluation metrics. > Key Takeaway: Use SageMaker Pipelines ConditionStep to automatically gate model registration and deployment based on evaluation metrics, eliminating the need for custom Lambda functions.
### Question 4 - Testing RAG System Updates A company is using an Amazon Bedrock Knowledge Base backed by OpenSearch Serverless for their RAG application. They are updating the chunking strategy and want to test the impact of this change on retrieval accuracy before pushing it to production. The testing must be fully automated within their AWS CodePipeline with the least operational overhead. Which approach is the most appropriate? - A) Create a duplicate OpenSearch Serverless collection and manually run search queries to compare results. - B) Trigger an AWS Lambda function to ingest a test document into OpenSearch and measure indexing latency. - C) Configure an Amazon Bedrock Model Evaluation job to evaluate the retrieval quality of the updated knowledge base against a dataset of test queries and expected responses. - D) Enable CloudWatch Logs on the OpenSearch Serverless collection and check for indexing errors during ingestion.	Correct Answer: C Correct Option Analysis: - C is Correct: Amazon Bedrock Model Evaluation supports evaluating RAG applications (Knowledge Bases). It can automatically run a dataset of test queries against the updated knowledge base, retrieve the contexts, and score the retrieval and generation quality using metrics like answer relevance and faithfulness. This native integration requires minimal operational overhead. Distractor Analysis: - A is Incorrect: Manually duplicating a collection and running search queries is a highly manual process that cannot be easily automated in a CI/CD pipeline and incurs high operational overhead. - B is Incorrect: Measuring indexing latency via Lambda only tests the performance of the ingestion process, not the semantic accuracy or retrieval quality of the RAG system. - D is Incorrect: Checking CloudWatch Logs for indexing errors verifies system health, not the accuracy of the chunking strategy or the quality of vector retrieval. > Key Takeaway: When updating RAG components like chunking strategies, use Amazon Bedrock Model Evaluation to assess retrieval and generation quality natively against a baseline dataset.
### Question 5 - Testing FMs for Bias and Explainability An enterprise is fine-tuning a foundation model on Amazon SageMaker JumpStart. They need to establish a CI/CD pipeline that tests the fine-tuned model for bias and explainability before it is made available to developers. The pipeline must fail if bias exceeds a defined threshold. Which native AWS service or feature should be used to implement this automated test with minimal custom code? - A) Amazon Comprehend to analyze the fine-tuning dataset for toxic phrases before training. - B) Amazon SageMaker Clarify to run bias and explainability evaluations on the fine-tuned model's outputs. - C) AWS CodeBuild with a custom Python script using the Natural Language Toolkit (NLTK) library to analyze outputs. - D) Amazon Macie to scan the model artifacts for sensitive data before deployment.	Correct Answer: B Correct Option Analysis: - B is Correct: Amazon SageMaker Clarify is purpose-built to detect bias in machine learning models and provide explainability for model predictions. It can be integrated into a SageMaker Pipeline as a Processing step to automatically evaluate the model's outputs against bias metrics, failing the pipeline if thresholds are breached, all with minimal custom code. Distractor Analysis: - A is Incorrect: Amazon Comprehend analyzes raw text for entities, sentiment, and key phrases, but it is not designed to run statistical bias detection or explainability metrics on machine learning model artifacts. - C is Incorrect: Writing a custom script in CodeBuild introduces significant operational overhead and custom code maintenance compared to using the purpose-built SageMaker Clarify container. - D is Incorrect: Amazon Macie is a data security service that scans S3 buckets for sensitive data (like PII); it does not evaluate model behavior or bias. > Key Takeaway: For automated bias and explainability testing in CI/CD, integrate Amazon SageMaker Clarify as an evaluation step rather than writing custom natural language processing scripts.
### Question 6 - Orchestrating Code and ML Workflows A machine learning team wants to automate their end-to-end ML workflow. The pipeline must handle both the unit testing of their Python code and the training of the model. They want to ensure that unit tests are executed before any expensive model training resources are provisioned, minimizing unnecessary compute costs. How should they design the CI/CD pipeline integration? - A) Use SageMaker Pipelines for the entire workflow, adding a Processing job at the start to run unit tests. - B) Trigger an AWS CodePipeline execution on code commit, use AWS CodeBuild for unit tests, and then trigger a SageMaker Pipeline for training and deployment. - C) Use Amazon EventBridge to trigger a SageMaker Training Job, and use SageMaker Model Monitor to run unit tests. - D) Deploy the code using AWS CodeDeploy to an EC2 instance, run the unit tests, and then trigger a SageMaker Pipeline.	Correct Answer: B Correct Option Analysis: - B is Correct: AWS CodePipeline is the orchestration layer. By using AWS CodeBuild as the first stage, the team can execute fast, cheap unit tests on their code. If the tests fail, the pipeline halts immediately. Only if the tests pass does CodePipeline trigger the downstream SageMaker Pipeline, thus avoiding the cost of provisioning expensive ML training instances on broken code. Distractor Analysis: - A is Incorrect: While a SageMaker Processing job can run tests, it provisions ML-optimized compute instances which are more expensive and slower to start than AWS CodeBuild. Running unit tests here wastes resources. - C is Incorrect: EventBridge triggers events but cannot run unit tests. SageMaker Model Monitor is for post-deployment monitoring of production endpoints, not pre-deployment unit testing. - D is Incorrect: Using AWS CodeDeploy to deploy code to EC2 instances just to run unit tests introduces massive operational overhead and unnecessary infrastructure management. > Key Takeaway: Separate cheap, fast code validation (AWS CodeBuild) from expensive ML compute (SageMaker Pipelines) by orchestrating them sequentially in AWS CodePipeline to minimize costs.
### Question 7 - Automated Agent Deployment Testing An enterprise is deploying an AI agent built on Amazon Bedrock Agents. The agent's configuration includes a new action group that calls an internal API. The CI/CD pipeline must test that the agent correctly invokes the new action group and returns the correct structured response before deployment. Which testing strategy meets this requirement with the least operational overhead while verifying the specific API invocation? - A) Deploy the agent to a staging environment and manually invoke the action group via the AWS CLI. - B) Configure an AWS Lambda function in AWS CodePipeline to invoke the Bedrock InvokeAgent API and parse the trace data to verify the correct action group invocation. - C) Use Amazon Bedrock Model Evaluation to assess the agent's semantic text generation quality using a prompt dataset. - D) Enable AWS CloudTrail to log the API calls made by the agent and manually inspect the logs post-deployment.	Correct Answer: B Correct Option Analysis: - B is Correct: To specifically verify that an Agent correctly routes a request to a specific action group and invokes the right API, the pipeline must inspect the agent's trace data. Using a lightweight AWS Lambda function to call InvokeAgent with a test prompt and parse the trace object in the response is the standard, programmatic way to validate agent routing logic in a CI/CD pipeline. Distractor Analysis: - A is Incorrect: Manual CLI invocation violates the requirement for an automated CI/CD pipeline testing strategy. - C is Incorrect: Bedrock Model Evaluation is designed to evaluate LLM text generation, toxicity, and semantic quality, not to verify the specific routing of Bedrock Agent action groups or API invocations. - D is Incorrect: Manually inspecting CloudTrail logs post-deployment is a reactive, manual process that does not gate the deployment in the CI/CD pipeline. > Key Takeaway: To automate testing of Bedrock Agent action group routing, use a Lambda function to call InvokeAgent and programmatically assert against the returned trace data.
### Question 8 - Regression Testing for Recommendation Models A retail company updates its product recommendation model weekly. Before deploying the new model version, the team needs to ensure it does not regress on critical user segments compared to the currently deployed model. The test must be automated within the AWS CodePipeline. Which AWS service is best suited to run this batch evaluation against historical data to gate the deployment? - A) Amazon SageMaker Processing job running a custom evaluation script that compares predictions on a held-out test set. - B) Amazon SageMaker Model Monitor using the Bias analysis type on the production endpoint. - C) AWS CodeBuild running unit tests against the recommendation algorithm's Python code. - D) Amazon QuickSight to visualize the differences between the model outputs.	Correct Answer: A Correct Option Analysis: - A is Correct: An Amazon SageMaker Processing job can take the new model and the current deployed model, run batch predictions against a held-out validation dataset, and execute a custom script to compare their metrics (e.g., precision@k). If the new model underperforms, the pipeline can be configured to halt. This is the standard approach for batch model regression testing. Distractor Analysis: - B is Incorrect: SageMaker Model Monitor analyzes live production traffic in real-time to detect data drift or model quality issues; it is not designed for pre-deployment batch regression testing against historical data. - C is Incorrect: AWS CodeBuild runs unit tests for code logic, but model regression testing requires deploying both models to compute resources to generate predictions, which SageMaker Processing handles natively. - D is Incorrect: Amazon QuickSight is a BI tool for visualizing data; it cannot execute model predictions, compute evaluation metrics, or gate a CI/CD pipeline. > Key Takeaway: Pre-deployment regression testing against historical data is best handled by an Amazon SageMaker Processing job executing evaluation scripts, whereas Model Monitor is for post-deployment runtime analysis.
### Question 9 - Automated Model Approval in Registry A machine learning team uses SageMaker Pipelines for their training workflow. The pipeline registers the model in the SageMaker Model Registry. The team needs to ensure that the model is automatically registered with an 'Approved' status only if its accuracy metric exceeds 0.90, reducing the need for manual reviews. Which configuration achieves this with the least operational overhead? - A) Add a RegisterModel step directly after the training step, and use AWS CodePipeline to approve it. - B) Create a ConditionStep that checks the evaluation metric, and if true, proceeds to a RegisterModel step configured with ModelApprovalStatus="Approved". - C) Use an AWS Lambda function to query the model registry and change the approval status post-registration. - D) Set up an Amazon EventBridge rule to trigger an AWS Lambda function to approve the model after registration.	Correct Answer: B Correct Option Analysis: - B is Correct: SageMaker Pipelines natively supports the ConditionStep which can evaluate the output metrics of an evaluation step. If the condition (Accuracy > 0.90) is met, it routes the workflow to the RegisterModel step. The RegisterModel step natively accepts a ModelApprovalStatus parameter, allowing the pipeline to register the model as Approved automatically without any custom code. Distractor Analysis: - A is Incorrect: Registering the model directly without checking the metric first means unqualified models will be registered. Additionally, AWS CodePipeline cannot natively read SageMaker metrics to approve a model without complex custom integrations. - C is Incorrect: Using a Lambda function to change the approval status after registration introduces unnecessary custom code and operational overhead when the RegisterModel step can do it natively. - D is Incorrect: Relying on EventBridge and Lambda to approve the model post-registration is a complex, event-driven workaround that is superseded by the native RegisterModel step's ModelApprovalStatus parameter. > Key Takeaway: The SageMaker Pipelines RegisterModel step can automatically set the ModelApprovalStatus to Approved or PendingManualApproval, enabling fully automated CI/CD gating without custom API calls.
### Question 10 - Testing Out-of-Domain Prompts for Hallucinations A generative AI application is being deployed using AWS CodePipeline. The engineering team wants to ensure that the updated model configuration does not cause the LLM to hallucinate when asked out-of-domain questions. The testing approach must use a standardized set of prompts and automatically grade the responses based on a rubric before allowing deployment. Which approach is the most appropriate? - A) Use Amazon Bedrock Guardrails to automatically block hallucinations, bypassing the need for pre-deployment testing. - B) Configure an Amazon Bedrock Model Evaluation job using LLM-as-a-judge to grade the responses of the candidate model against the standardized prompts based on a rubric. - C) Configure an AWS Lambda function to compare the cosine similarity of the candidate response with a single expected response. - D) Enable SageMaker Model Monitor to detect hallucinations after the model is deployed to production.	Correct Answer: B Correct Option Analysis: - B is Correct: Amazon Bedrock Model Evaluation allows teams to define a dataset of out-of-domain prompts. By utilizing the LLM-as-a-judge evaluation method, the pipeline can automatically grade the candidate model's responses against a specific rubric (e.g., "Does the model admit it lacks knowledge?"). If the scores fall below a threshold, the pipeline gates the deployment. Distractor Analysis: - A is Incorrect: While Bedrock Guardrails can filter content and block topics, they do not eliminate the need for pre-deployment testing of model behavior and prompt configurations. Guardrails are a runtime mitigation, not a CI/CD test. - C is Incorrect: Cosine similarity is a rigid vector comparison that is ill-suited for grading nuanced LLM behaviors like admitting a lack of knowledge, where responses may vary significantly in text but share the same semantic meaning. - D is Incorrect: SageMaker Model Monitor is used for post-deployment monitoring of production traffic, not for pre-deployment CI/CD gating against a standardized test set. > Key Takeaway: To test for specific generative behaviors like hallucination in CI/CD, use Bedrock Model Evaluation with LLM-as-a-judge to grade responses against a rubric rather than relying on similarity scores or post-deployment monitoring.