Skill 3.3.9: Implement AI model testing frameworks, including prompt testing

Question	Answer & Explanation
### Question 1 - GenAI Response Quality Testing in CI/CD An AI engineering team is setting up a CI/CD pipeline to deploy updates to a generative AI application using Amazon Bedrock. The team needs to ensure that a new version of the fine-tuned model does not degrade the quality of responses compared to the currently deployed version. Due to the non-deterministic nature of generative AI, exact string matching is not feasible. The pipeline must block deployment if the response quality drops below a defined threshold, with the least operational overhead. Which ONE testing approach fulfills these requirements? (Select ONE) - A) Run a fixed golden prompt dataset through both versions, use an LLM-as-a-judge to evaluate responses, and block deployment if the average score drops below the baseline. - B) Assert that each response from the new version matches the expected output string stored in the test suite. - C) Deploy the new version to a staging endpoint and manually review the responses for a week before promoting to production. - D) Monitor the CPU utilization and latency of the new endpoint during a canary deployment to determine response quality.	Correct Answer: A Correct Answer Analysis: - A is Correct: Generative AI models are non-deterministic, meaning responses vary across runs. To test response quality in an automated CI/CD pipeline, you run a curated "golden prompt dataset" through both the candidate and baseline versions. Using an LLM-as-a-judge or task-specific metrics (e.g., faithfulness, answer relevance) allows the pipeline to objectively compare scores and block deployment if the candidate version's quality drops. Distractor Analysis: - B is Incorrect: Because generative AI responses naturally vary between runs, exact string assertions will cause false negatives and flaky pipeline builds. - C is Incorrect: Manual review introduces significant operational overhead and eliminates the automated CI/CD gating required by the scenario. - D is Incorrect: Monitoring infrastructure metrics like CPU utilization and latency checks throughput and responsiveness, but does not detect drops in semantic response accuracy or generation quality. > Exam Tip / AWS Best Practice: For non-deterministic GenAI models, use automated evaluation frameworks (like LLM-as-a-judge or ROUGE/BERTScore) on a golden dataset to gate deployments in CI/CD pipelines.
### Question 2 - Amazon Bedrock Prompt Management Versioning A company is using Amazon Bedrock Prompt Management to iterate on prompts for a customer service chatbot. The prompt engineering team frequently updates prompts to improve response relevance. The engineering team wants to automate the release process for these prompts so that application code doesn't hardcode prompt text, and prompt updates are version-controlled. The solution must ensure that a new prompt variant is automatically tested against a battery of test cases before it is updated in the production application. Which ONE approach satisfies these requirements? (Select ONE) - A) Store prompt text in an Amazon S3 bucket and use AWS Lambda to fetch and test the prompt before updating the application configuration. - B) Hardcode the prompt text in the application code and use AWS CodeBuild to run unit tests on the text formatting. - C) Use Amazon Bedrock Prompt Management to create and test prompt variants, then use the Bedrock API to retrieve the approved prompt version dynamically in the application. - D) Save the prompt in an AWS Systems Manager Parameter Store and update it manually when tests pass.	Correct Answer: C Correct Answer Analysis: - C is Correct: Amazon Bedrock Prompt Management stores prompts as versioned resources with variables and comparable variants. This allows prompt changes to be tested and released through a pipeline rather than edited directly in application code. The application retrieves the specific version dynamically, ensuring updates are tracked and tested without code deployments. Distractor Analysis: - A is Incorrect: Storing prompts in S3 and using Lambda introduces custom, unnecessary operational overhead compared to the native Bedrock Prompt Management service. - B is Incorrect: Hardcoding prompt text in application code violates the requirement to decouple prompts from application logic and makes version control and automated variant testing difficult. - D is Incorrect: Manually updating Parameter Store violates the automated testing and release requirements of the CI/CD pipeline. > Exam Tip / AWS Best Practice: Treat prompts as code. Use Amazon Bedrock Prompt Management to version, test, and dynamically fetch prompt variants, avoiding hardcoded text in application logic.
### Question 3 - Testing RAG System Updates in CI/CD An AI team is building a Retrieval-Augmented Generation (RAG) application using Amazon Bedrock Knowledge Bases. When the team updates the chunking strategy or embedding model, they need to ensure the retrieval quality hasn't degraded before updating the production vector index. The team wants to automate this testing within their AWS CodePipeline. The solution must evaluate both the retrieval accuracy and the generation quality with the least operational overhead. Which ONE approach fulfills these requirements? (Select ONE) - A) Manually query the RAG application after updating the index and visually inspect the answers for accuracy. - B) Configure an Amazon Bedrock Model Evaluation job that uses a custom dataset to evaluate retrieval and generation metrics, triggered by AWS CodePipeline. - C) Monitor the CloudWatch metrics for the OpenSearch Serverless collection to check for an increase in search latency. - D) Use AWS Lambda to run deterministic unit tests against the retrieval API, expecting exact string matches for the generated responses.	Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock Model Evaluation supports evaluating RAG systems by measuring both retrieval metrics (e.g., context recall, context precision) and generation metrics (e.g., faithfulness, answer relevance). Integrating this natively into CodePipeline automates the gating process with minimal custom code. Distractor Analysis: - A is Incorrect: Manual visual inspection introduces massive operational overhead and lacks the automated CI/CD gating required. - C is Incorrect: CloudWatch search latency metrics indicate infrastructure performance, not the semantic accuracy or quality of the retrieved context and generated responses. - D is Incorrect: Generative responses cannot be effectively tested using deterministic unit tests with exact string matches due to their non-deterministic nature. > Exam Tip / AWS Best Practice: Use Amazon Bedrock Model Evaluation to automatically assess both retrieval (context accuracy) and generation (answer faithfulness) metrics for RAG pipelines in CI/CD.
### Question 4 - Gating Traditional ML Models in CI/CD A machine learning team is migrating a traditional classification model's CI/CD pipeline to AWS. The model produces a probability score, and the pipeline must automatically prevent a new model version from registering in the Amazon SageMaker Model Registry if its F1 score drops by more than 2% compared to the previous version. The team wants to achieve this with the least operational overhead using native AWS services. Which ONE configuration achieves this? (Select ONE) - A) Configure a SageMaker Pipeline `ConditionStep` that evaluates the model, compares the F1 score against the previous version's metadata in the Model Registry, and uses a `Fail` step if the condition is not met. - B) Deploy the model to a SageMaker endpoint, route 10% of traffic via a canary deployment, and trigger an AWS Lambda function to calculate the F1 score from CloudWatch logs. - C) Export the training metrics to Amazon S3 and manually review them before approving the model package in the SageMaker Model Registry. - D) Use an AWS Lambda function to trigger the SageMaker training job and stop the pipeline if the F1 score in the CloudWatch logs is lower than the threshold.	Correct Answer: A Correct Answer Analysis: - A is Correct: Amazon SageMaker Pipelines natively support `ConditionStep` to gate progression based on evaluation metrics. By comparing the current evaluation metrics to the metadata stored in the Model Registry, the pipeline can automatically route to a `Fail` step if the F1 score drops, preventing bad models from registering. Distractor Analysis: - B is Incorrect: Canary deployments are for validating live inference performance, not for evaluating batch model metrics like F1 score. Computing F1 from live CloudWatch logs is complex and operationally heavy. - C is Incorrect: Manual review contradicts the requirement for an automated CI/CD pipeline. - D is Incorrect: Using Lambda to monitor CloudWatch logs for training metrics introduces high custom operational overhead compared to native SageMaker Pipeline steps. > Exam Tip / AWS Best Practice: Use SageMaker Pipelines `ConditionStep` to compare evaluation metrics against thresholds or previous versions, automatically gating model registration.
### Question 5 - Automated Agent Testing An enterprise is developing an autonomous agent using Amazon Bedrock that utilizes multiple tools to fetch data. The engineering team needs to implement a testing framework to ensure that updates to the agent's configuration or underlying model do not cause it to call the wrong tools or enter infinite loops. The pipeline must be automated and evaluate the agent's multi-turn decision-making process. Which ONE testing approach fits this requirement? (Select ONE) - A) Deploy the updated agent and rely on user feedback in production to identify if it enters an infinite loop. - B) Run a curated set of multi-turn conversation scenarios in a staging environment using an automated evaluation script, checking if the correct tool was called and ensuring the conversation terminates properly. - C) Check the agent's configuration file in the GitHub repository for syntax errors using AWS CodeBuild. - D) Monitor the agent's API invocation latency in CloudWatch and roll back if the latency exceeds 5 seconds.	Correct Answer: B Correct Answer Analysis: - B is Correct: Agent behavior depends on tool selection and multi-turn logic. An automated test framework must simulate multi-turn conversations, verify that the agent selects the correct tools based on the user's intent, and ensure the agent successfully terminates rather than looping indefinitely. Distractor Analysis: - A is Incorrect: Relying on production user feedback for infinite loops is a poor testing practice and exposes users to severe application failures. - C is Incorrect: Syntax validation in CodeBuild only checks code structure, not the runtime decision-making behavior of the agent. - D is Incorrect: Latency monitoring might catch an infinite loop eventually, but it does not evaluate whether the agent called the correct tools or made the right decisions. > Exam Tip / AWS Best Practice: Test autonomous agents by running automated multi-turn conversational scenarios that validate tool selection logic and ensure proper conversation termination.
### Question 6 - Testing Toxicity and Prompt Adherence A company is fine-tuning a foundation model using Amazon SageMaker JumpStart. They want to implement a CI/CD pipeline that automatically tests the fine-tuned model for toxicity and prompt adherence before deploying it to an Amazon Bedrock provisioned throughput endpoint. The testing framework must require minimal custom code and use AWS native evaluation services. Which ONE approach satisfies these requirements? (Select ONE) - A) Write a custom Python script using AWS Lambda to call the model and check for specific keywords in the responses. - B) Use Amazon Bedrock Model Evaluation to automatically evaluate the fine-tuned model for toxicity and prompt adherence using built-in metrics and datasets. - C) Deploy the model to an endpoint and use Amazon Macie to scan the output text for toxic content. - D) Manually execute a test prompt suite in the SageMaker Studio notebook and record the results in a spreadsheet.	Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon Bedrock Model Evaluation provides built-in evaluation capabilities for toxicity, prompt adherence, and other safety metrics. It requires minimal custom code and integrates seamlessly into CI/CD pipelines to gate deployments based on safety and adherence thresholds. Distractor Analysis: - A is Incorrect: Writing custom Lambda scripts with keyword matching is brittle, operationally heavy, and inferior to native LLM evaluation mechanisms. - C is Incorrect: Amazon Macie is a data security service for discovering and protecting sensitive data in S3; it does not evaluate generative AI text for toxicity. - D is Incorrect: Manual testing in SageMaker Studio violates the automated CI/CD requirement. > Exam Tip / AWS Best Practice: Leverage Amazon Bedrock Model Evaluation's built-in metrics for automated, low-overhead toxicity and adherence testing in CI/CD pipelines.
### Question 7 - Comparing Prompt Variants An AI engineering team is using Amazon Bedrock Prompt Management to optimize prompts. They want to test two prompt variants against each other to see which one generates more concise and accurate responses. The testing must be automated and trigger a deployment of the winning prompt version if it outperforms the current one. The constraint is to use AWS native services to compare the prompt variants. Which ONE approach fulfills these requirements? (Select ONE) - A) Manually test both variants in the Bedrock playground and update the application's configuration file with the winning prompt text. - B) Hardcode both variants in the application, route 50% of traffic to each, and use CloudWatch to monitor which one has lower latency. - C) Use Amazon Bedrock Prompt Management to create the variants, run an automated evaluation using LLM-as-a-judge, and update the application to point to the winning version's ARN. - D) Export the prompt variants to an S3 bucket and use AWS Glue to analyze the response lengths.	Correct Answer: C Correct Answer Analysis: - C is Correct: Amazon Bedrock Prompt Management allows the creation of comparable variants. By running an automated evaluation (e.g., using LLM-as-a-judge) against these variants, the pipeline can objectively identify the winner and dynamically update the application to reference the new prompt version's ARN. Distractor Analysis: - A is Incorrect: Manual testing in the playground and manual config updates violate the automation requirement. - B is Incorrect: Hardcoding prompts in the application is an anti-pattern, and latency is not a measure of response conciseness or accuracy. - D is Incorrect: AWS Glue is an ETL service for data processing, not an LLM evaluation tool for prompt variant comparison. > Exam Tip / AWS Best Practice: Automate prompt variant testing using Amazon Bedrock Prompt Management and LLM-as-a-judge, dynamically pointing applications to the winning prompt ARN.
### Question 8 - Testing Generated SQL Queries A GenAI application relies on an LLM that generates SQL queries based on user natural language input. The engineering team needs an automated testing framework within AWS CodePipeline to ensure that updates to the LLM or prompt do not break the SQL syntax or result in incorrect query logic. The pipeline must execute the generated SQL against a synthetic database and verify the results. Which ONE testing strategy meets these requirements? (Select ONE) - A) Use AWS CodeBuild to run a test harness that generates SQL from a dataset of natural language prompts, executes the SQL against a test database, and verifies the returned data matches expected results. - B) Evaluate the generated SQL by checking if it contains the correct table names using a regular expression in a Lambda function. - C) Deploy the model to production and monitor CloudWatch for SQL syntax errors returned by the application database. - D) Use an LLM-as-a-judge to evaluate whether the generated SQL "looks correct" without executing it against a database.	Correct Answer: A Correct Answer Analysis: - A is Correct: For code-generating models (like Text-to-SQL), the most reliable testing framework is functional execution. Using AWS CodeBuild to run a test harness that generates the SQL, executes it against a synthetic database, and validates the output data ensures both syntax correctness and logical accuracy. Distractor Analysis: - B is Incorrect: Regular expressions checking for table names do not validate SQL syntax or logical query correctness. - C is Incorrect: Relying on production database errors exposes end-users to failures and does not verify logical correctness, only syntax errors that reach the DB. - D is Incorrect: LLM-as-a-judge cannot reliably verify if complex SQL will execute correctly or return the precise dataset expected without actually running it. > Exam Tip / AWS Best Practice: For code-generating AI models, use functional testing (executing the generated code) in CodeBuild rather than semantic evaluation to ensure syntax and logical correctness.
### Question 9 - Testing Guardrails in CI/CD A team is building a CI/CD pipeline for an Amazon Bedrock application that must adhere to strict safety guidelines. The pipeline must block the deployment of a new prompt or model version if the responses contain Personally Identifiable Information (PII) or harmful content. The solution requires the least operational overhead and must use AWS native features to test the safety mechanisms. Which ONE approach fulfills these requirements? (Select ONE) - A) Deploy Amazon Bedrock Guardrails with automatic blocking enabled, but skip testing in the pipeline as Guardrails will handle it in production. - B) Create an AWS Lambda function that uses regular expressions to scan outputs for PII patterns and blocks deployment if found. - C) Integrate Amazon Bedrock Guardrails into the CI/CD pipeline testing phase, run a set of adversarial prompts, and verify that the Guardrails successfully block or redact the responses before allowing deployment. - D) Rely on AWS Artifact reports to ensure the model complies with data protection regulations.	Correct Answer: C Correct Answer Analysis: - C is Correct: Bedrock Guardrails should be tested adversarially in the CI/CD pipeline. By running a curated set of harmful or PII-leaking prompts through the model with the Guardrail applied, the pipeline can verify that the Guardrail correctly intercepts them. If the Guardrail fails to block/redact the test cases, the pipeline blocks deployment. Distractor Analysis: - A is Incorrect: Skipping pipeline testing and relying purely on production behavior risks deploying misconfigured Guardrails or models that bypass the safety filters. - B is Incorrect: Regex patterns in Lambda are brittle and cannot effectively catch all nuanced forms of PII or harmful content compared to native ML-based Guardrails. - D is Incorrect: AWS Artifact provides compliance reports for AWS services, but it does not test your specific prompt/model configurations for safety. > Exam Tip / AWS Best Practice: Test Amazon Bedrock Guardrails adversarially within your CI/CD pipeline using known malicious prompts to verify they block/redact correctly before production deployment.
### Question 10 - Gating SageMaker Pipelines on GenAI Metrics An organization is deploying an automated MLOps pipeline using Amazon SageMaker Pipelines for a generative AI model. The pipeline currently evaluates the model using ROUGE and BERTScore metrics. The team wants to ensure that the pipeline execution fails and stops automatically if the new model's scores are lower than the current production model. Which approach satisfies this with the least operational overhead? Which ONE configuration achieves this? (Select ONE) - A) Add an Amazon SageMaker Pipelines `ConditionStep` that compares the evaluation metrics from the current run with the metrics of the registered production model, followed by a `Fail` step if the condition is not met. - B) Export the evaluation metrics to Amazon CloudWatch, create a CloudWatch Alarm, and use AWS Lambda to stop the pipeline if the alarm triggers. - C) Run the evaluation step in AWS CodeBuild and fail the build if the scores are lower, preventing the SageMaker Pipelines execution from proceeding. - D) Manually review the ROUGE and BERTScore metrics in Amazon SageMaker Studio and approve the pipeline execution manually.	Correct Answer: A Correct Answer Analysis: - A is Correct: SageMaker Pipelines natively support `ConditionStep` to evaluate logic. By comparing the current run's ROUGE/BERTScore metrics to the baseline production model's metrics stored in the Model Registry, the pipeline can route to a `Fail` step to halt execution automatically if the quality regresses, ensuring zero operational overhead. Distractor Analysis: - B is Incorrect: Using CloudWatch Alarms and Lambda to stop a pipeline introduces significant custom operational overhead and is not a native SageMaker Pipelines feature for gating steps. - C is Incorrect: Running the evaluation in CodeBuild breaks the native SageMaker Pipeline lineage and makes tracking pipeline execution artifacts more complex. - D is Incorrect: Manual review in SageMaker Studio defeats the purpose of an automated CI/CD pipeline. > Exam Tip / AWS Best Practice: Use SageMaker Pipelines `ConditionStep` to natively gate pipeline progression on evaluation metrics, halting the workflow with a `Fail` step if quality thresholds are not met.