Skip to content

Skill 2.3.1: Perform reproducible experiments (MLflow on SageMaker AI, Bedrock evaluations, Bedrock Prompt Management)

Question 1 - Experiment Tracking for Reproducibility

A data science team at a financial services company is training multiple fraud detection models using Amazon SageMaker AI. The team frequently runs hyperparameter tuning experiments with varying learning rates, batch sizes, and feature engineering pipelines. Several team members work on different experiments simultaneously, and they often struggle to reproduce results from previous runs because parameters, code versions, and data lineage are inconsistently documented. The team needs a solution that automatically tracks all run metadata while minimizing operational overhead.

Which approach should the team implement to ensure reproducible experiments?

  • A) Configure SageMaker Training jobs to write hyperparameters to Amazon CloudWatch Logs and manually document code versions in a shared wiki.
  • B) Enable MLflow on Amazon SageMaker AI to track each experiment run, including parameters, metrics, artifacts, and code versions automatically.
  • C) Store all training scripts in an S3 bucket and use S3 Versioning to track code changes, while logging hyperparameters to a DynamoDB table via a custom Lambda function.
  • D) Use AWS CloudTrail to record API calls made during training jobs and derive experiment metadata from event history.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: MLflow on SageMaker AI provides a fully managed experiment tracking service that automatically records parameters, metrics, artifacts, model lineage, and source code version for every training run. It integrates natively with SageMaker training jobs, eliminating manual documentation and enabling team-wide reproducibility without building custom infrastructure.

Distractor Analysis

  • A is Incorrect: Manually documenting code versions in a wiki and relying on CloudWatch Logs introduces human error, does not provide structured querying, and fails to capture complete experiment lineage (artifacts, data versions, environment dependencies) automatically.
  • C is Incorrect: Building a custom tracking solution with S3 Versioning and DynamoDB requires significant operational overhead to maintain, synchronize, and query — violating the "minimizing operational overhead" constraint. It also does not capture all MLflow-native metadata like model artifacts and evaluation traces.
  • D is Incorrect: CloudTrail records API call events (who called what, when), but it does not capture training-specific metadata such as hyperparameters, loss metrics, model artifacts, or dataset versions needed for experiment reproducibility.

Key Takeaway: MLflow on SageMaker AI is the purpose-built, fully managed service for experiment tracking and reproducibility, automatically capturing parameters, metrics, artifacts, and code lineage without custom infrastructure.

Question 2 - Tracing GenAI Application Inputs and Outputs

A machine learning engineer is debugging an Amazon Bedrock-based generative AI application that occasionally returns incorrect persona reflections in a customer support chatbot. The team needs to trace the full execution path — including prompt templates, retrieved context chunks, model parameters, and generated responses — across multiple orchestration steps to identify which component introduced the error. The solution must integrate with their existing SageMaker AI workflow and provide step-by-step visibility.

Which service capability should the engineer enable?

  • A) Enable AWS X-Ray tracing on the Lambda functions that invoke Amazon Bedrock to capture segment-level input and output metadata.
  • B) Configure MLflow tracing on Amazon SageMaker AI to record inputs, outputs, and metadata at each step of the generative AI application execution.
  • C) Enable Amazon Bedrock invocation logging to Amazon S3 and analyze raw request/response payloads using Amazon Athena.
  • D) Use Amazon Bedrock Guardrails to intercept and log each inference step before the model generates a response.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: MLflow tracing on SageMaker AI is specifically designed to instrument generative AI applications, capturing the inputs, outputs, prompts, retrieved context, model parameters, and latency at each step of the pipeline. This provides end-to-end observability and allows engineers to pinpoint exactly which component (e.g., retrieval, prompt assembly, or model generation) introduced the error.

Distractor Analysis

  • A is Incorrect: AWS X-Ray traces service-to-service requests (e.g., Lambda → Bedrock API calls) and captures latency and error metadata, but it does not natively record the internal GenAI execution details such as prompt text, embedding vectors, retrieved chunks, or token-level generation traces required for debugging persona leaks.
  • C is Incorrect: Bedrock invocation logging to S3 captures raw request/response payloads, but analyzing them with Athena requires manual query setup, does not provide step-by-step execution tracing across orchestration layers, and misses intermediate state transitions within the GenAI pipeline.
  • D is Incorrect: Amazon Bedrock Guardrails are content filtering and PII redaction mechanisms — not execution tracing tools. They do not log intermediate orchestration steps or the flow between retrieval, prompting, and generation.

Key Takeaway: MLflow tracing on SageMaker AI is the purpose-built capability for instrumenting GenAI applications, capturing inputs, outputs, and metadata at every step to enable root-cause analysis of unexpected responses.

Question 3 - Prompt Variant Comparison Workflow

A product team is optimizing the system prompt for a customer-facing summarization feature built on Amazon Bedrock. The team has developed three candidate prompt variations that differ in tone instructions, output format specifications, and example inclusion. They need to systematically compare these variants against a curated evaluation dataset to determine which prompt produces the highest quality summaries, while ensuring the comparison process is versioned, repeatable, and auditable. Operational overhead must be minimal.

Which TWO AWS capabilities should the team use together? (Select TWO)

  • A) Store prompt templates in Amazon S3 with manual version tracking and invoke each variant through a custom Step Functions workflow for comparison.
  • B) Use Amazon Bedrock Prompt Management to store each prompt variant with versioning and structured comparison capabilities.
  • C) Run Amazon Bedrock Evaluations with the curated dataset to programmatically score each prompt variant using LLM-as-a-judge metrics.
  • D) Deploy all three prompt variants as SageMaker endpoints behind an Amazon Route 53 weighted routing policy to capture live traffic comparisons.
  • E) Use AWS Config rules to track prompt changes and trigger evaluation jobs using Amazon EventBridge.
Click to reveal Answer & Explanation

Correct Answers: B and C

Correct Options Analysis

  • B is Correct: Amazon Bedrock Prompt Management provides native prompt storage with variables, versions, and comparison workflows — enabling teams to track every prompt iteration and compare variants side-by-side as a repeatable, auditable experiment.
  • C is Correct: Amazon Bedrock Evaluations supports automated programmatic and LLM-as-a-judge scoring against curated evaluation datasets, allowing the team to quantify summary quality for each prompt variant systematically.

Distractor Analysis

  • A is Incorrect: Storing prompts in S3 with custom Step Functions workflows introduces unnecessary operational complexity. The team would need to build and maintain version tracking, evaluation orchestration, and result comparison logic — functionalities already native to Bedrock Prompt Management and Bedrock Evaluations.
  • D is Incorrect: Deploying prompt variants as separate SageMaker endpoints with Route 53 weighted routing describes an A/B testing or canary rollout infrastructure, not a systematic offline prompt comparison. This approach requires full endpoint deployment and live traffic routing rather than a versioned comparison workflow.
  • E is Incorrect: AWS Config tracks infrastructure configuration changes (e.g., S3 bucket policies, IAM roles) and is not designed to track prompt text variations, nor does it provide prompt comparison or evaluation scoring capabilities.

Key Takeaway: Amazon Bedrock Prompt Management + Amazon Bedrock Evaluations together provide a fully managed, repeatable prompt experimentation workflow — versioning prompts and scoring them against curated datasets without building custom comparison infrastructure.

Question 4 - Reproducing LLM Evaluation Results

A machine learning operations team is running automated evaluations on an LLM-based document summarization system at scale. The team runs hundreds of evaluation jobs using Amazon Bedrock Evaluations with LLM-as-a-judge metrics. After several weeks, the team observes that identical evaluation jobs run on different days produce slightly different judge scores. They suspect the judge model's configuration or evaluation prompt might have changed, but they have no systematic record of each evaluation job's configuration. The team must ensure that any past evaluation result can be reproduced and attributed to its exact configuration.

Which approach should the team implement?

  • A) Store evaluation result JSON files in Amazon S3 with SSE-KMS encryption and enable S3 Object Lock for immutability.
  • B) Enable MLflow tracking on Amazon SageMaker AI to record each evaluation run, including judge model configuration, evaluation prompts, dataset versions, and scoring outputs.
  • C) Write evaluation job metadata to an Amazon RDS for PostgreSQL table using a custom ETL pipeline triggered by CloudWatch Events.
  • D) Use AWS CloudFormation drift detection to identify stack changes between evaluation runs and correlate them with score variations.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: MLflow tracking on SageMaker AI records complete evaluation run metadata — including judge model identifiers, temperature settings, evaluation prompt templates, dataset version, and generated scores — in a structured, queryable format. This enables exact reproducibility of any past evaluation result and facilitates root-cause analysis of score variations.

Distractor Analysis

  • A is Incorrect: S3 Object Lock ensures immutability (preventing deletion/overwrite) but does not provide structured tracking of evaluation configurations. It stores the what (result files) but not the why (configurations, prompts, judge model parameters) in an easily queryable format.
  • C is Incorrect: Building a custom ETL pipeline to write metadata to RDS introduces significant operational overhead and still requires manual instrumentation of every evaluation job. It duplicates functionality already provided by MLflow tracking on SageMaker AI without adding value.
  • D is Incorrect: CloudFormation drift detection identifies infrastructure configuration changes (e.g., EC2 instance type, VPC settings), not evaluation job parameters.

Key Takeaway: MLflow tracking on SageMaker AI is the standard for recording evaluation job metadata (judge model, prompts, dataset versions, scores) to guarantee reproducibility and enable root-cause analysis of metric variations.

Question 5 - Experiment Tracker Selection for RAG Evaluation

A startup is building a Retrieval-Augmented Generation (RAG) system using Amazon Bedrock Knowledge Bases and Amazon OpenSearch Serverless. The ML team needs to continuously evaluate the system's retrieval precision separately from generation quality. They plan to run weekly evaluation jobs that compute Context Precision and Faithfulness metrics on a curated test dataset. The team also wants to track how these metrics change over time across different knowledge base versions and model configurations, ensuring complete experiment lineage. The startup operates with a small team and requires the least operational overhead solution.

Which approach should the startup implement?

  • A) Create a custom evaluation dashboard in Amazon QuickSight that queries evaluation metrics from a DynamoDB table populated by a Lambda function after each evaluation job.
  • B) Use Amazon Bedrock Evaluations to run weekly evaluation jobs and integrate with MLflow on SageMaker AI to track metrics, dataset versions, and knowledge base configurations across runs.
  • C) Schedule evaluation jobs using AWS Batch, write results to Amazon Redshift, and build a custom visualization layer in Amazon Managed Grafana for trend analysis.
  • D) Use Amazon SageMaker Model Monitor to track evaluation metrics through CloudWatch metrics and create CloudWatch dashboards for trend visualization.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: Amazon Bedrock Evaluations natively supports separate evaluation of retrieval sources (Context Precision) and generated responses (Faithfulness) with minimal setup. Integrating evaluation results with MLflow on SageMaker AI provides automatic tracking of metrics, dataset versions, knowledge base configurations, and model identifiers — enabling longitudinal trend analysis without building custom infrastructure. Both services are fully managed, satisfying the low operational overhead constraint.

Distractor Analysis

  • A is Incorrect: Building a custom pipeline with Lambda + DynamoDB + QuickSight requires significant engineering effort to design schemas, write Lambda functions, manage IAM permissions, and build dashboard visualizations — violating the least operational overhead requirement for a small team.
  • C is Incorrect: AWS Batch + Redshift + Managed Grafana introduces a heavy, multi-service custom architecture requiring job scheduling, cluster management, data warehouse optimization, and dashboard maintenance — entirely unnecessary when managed Bedrock Evaluations and MLflow tracking already exist.
  • D is Incorrect: SageMaker Model Monitor is designed for monitoring deployed endpoint drift (data, prediction, and schema drift) on real-time inference traffic, not for scoring curated evaluation datasets against RAG retrieval and generation quality metrics.

Key Takeaway: Amazon Bedrock Evaluations provides purpose-built RAG evaluations (retrieval and generation quality measured separately), and MLflow on SageMaker AI adds automatic tracking of these metrics across runs without custom infrastructure — ideal for small teams needing the least operational overhead.

Question 6 - Shadow Test Configuration for Production Comparison

A fraud detection platform serves real-time credit scoring predictions through a SageMaker AI endpoint with a production variant. The data science team has trained a new gradient-boosted tree model that reduced offline validation error by 12%. Before promoting the candidate model, the team must validate its performance on live request traffic — including actual payload size distributions, concurrency patterns, and response latency — without affecting the responses that callers currently receive. The team also needs to compare error rates and latency metrics between the two models.

Which approach should the team use?

  • A) Deploy the candidate model as a new SageMaker endpoint and route 10% of traffic to it using an Application Load Balancer for A/B testing.
  • B) Configure a SageMaker AI shadow test to deploy the candidate as a shadow variant alongside the production variant on the same endpoint, routing live request copies while production responses go to callers.
  • C) Run an offline evaluation using a synthetic replay of historical request logs through Amazon SageMaker Batch Transform to compare latency and error rates.
  • D) Deploy the candidate model to a SageMaker endpoint with a different instance type and compare CloudWatch metrics between the two endpoints.
Click to reveal Answer & Explanation

Correct Answer: B

Correct Answer Analysis

  • B is Correct: SageMaker AI shadow testing is specifically designed for comparing a candidate variant against a production variant on live traffic without changing caller responses. It routes live request copies asynchronously to the shadow variant, logs its responses, and captures operational metrics (latency, error rate, throughput) for offline comparison. The production variant continues to serve all caller responses with zero impact.

Distractor Analysis

  • A is Incorrect: An Application Load Balancer routing 10% of traffic to a separate endpoint is an A/B testing or canary rollout pattern, meaning 10% of real callers receive candidate model responses — violating the constraint that production responses must not change for callers.
  • C is Incorrect: Batch Transform runs offline inference on stored data, but it does not capture live traffic characteristics such as real-time concurrency patterns, network latency under load, or peak-hour request shapes. Offline replay cannot fully substitute for live shadow testing.
  • D is Incorrect: Deploying on a separate endpoint introduces confounding variables — different instance type, endpoint configuration, and network paths — making it impossible to isolate model performance differences from infrastructure differences. It also does not test against live production request traffic.

Key Takeaway: SageMaker AI shadow testing is the standard for validating candidate models against live production traffic patterns without changing what callers receive — enabling safe offline comparison of latency, error rates, and predictions.

Question 7 - Debugging Non-Convergent Training

A machine learning engineer is training a transformer-based text classification model using SageMaker Training. During training, the engineer observes that the training loss decreases steadily for the first few epochs, but then begins oscillating dramatically between epochs without settling into a stable minimum. The validation loss follows the same oscillation pattern. The engineer needs to identify the root cause and apply the most appropriate fix.

Which actions should the engineer take? (Select TWO)

  • A) Increase the number of training epochs to allow the model more time to converge past the oscillation phase.
  • B) Reduce the learning rate to prevent weight updates from overshooting the loss minimum.
  • C) Increase the batch size significantly until the loss curve smooths out.
  • D) Add a learning rate decay scheduler to gradually reduce the step size during training.
  • E) Increase the number of attention heads in the transformer to give the model more capacity to fit the data.
Click to reveal Answer & Explanation

Correct Answers: B and D

Correct Options Analysis

  • B is Correct: Oscillating loss without convergence is a classic symptom of a learning rate that is too high. Reducing the learning rate reduces the magnitude of weight updates, preventing the optimizer from repeatedly overshooting the loss minimum and allowing the loss to settle.
  • D is Correct: A learning rate decay scheduler (e.g., cosine annealing, step decay, or ReduceLROnPlateau) progressively reduces the step size during the course of training, allowing aggressive optimization early on and stable convergence near the minimum. This is a best practice for transformers and other deep learning models.

Distractor Analysis

  • A is Incorrect: Simply increasing the number of epochs does not address the root cause — an unstable optimization step size. The loss will continue to oscillate indefinitely without settling if the learning rate remains too high.
  • C is Incorrect: Increasing batch size changes the gradient estimate noise characteristics but does not directly address the root cause of oscillation caused by an oversized learning rate. In fact, larger batch sizes often permit slightly higher learning rates, but this is not a guaranteed fix without adjusting the learning rate itself.
  • E is Incorrect: Increasing model capacity (attention heads) addresses underfitting — where the model cannot capture signal in the data (loss barely moves) — not the oscillation problem caused by optimization instability. Adding capacity may actually worsen oscillation if the model overfits.

Key Takeaway: Loss oscillation that never settles indicates a learning rate that is too high — fix it directly by reducing the learning rate and/or adding a decay scheduler, not by extending epochs or changing model architecture.

Question 8 - Evaluating Generated Text Quality Metrics

A healthcare documentation team is evaluating a medical summarization model that condenses clinical notes into discharge summaries. They have reference summaries written by human clinicians. The generated summaries are fluent but occasionally paraphrase the clinical findings using different medical terminology while preserving meaning. The team needs to select evaluation metrics that assess both lexical overlap with reference summaries and semantic meaning preservation, as the summaries often use synonyms and reordered sentence structures.

Which TWO metrics should the team use together? (Select TWO)

  • A) ROUGE (Recall-Oriented Understudy for Gisting Evaluation) to measure n-gram overlap between generated and reference summaries.
  • B) Accuracy to measure the percentage of exactly matching summary tokens.
  • C) BERTScore to measure semantic similarity between generated and reference summaries using contextual embeddings.
  • D) Confusion Matrix to track true positive and false positive summary tokens.
  • E) Root Mean Squared Error (RMSE) to measure the difference between generated and reference summary lengths.
Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: ROUGE is the standard metric for text summarization evaluation, measuring n-gram overlap (ROUGE-1, ROUGE-2, ROUGE-L) between generated and reference summaries. It captures lexical similarity, including recall-oriented evaluation of how many reference n-grams appear in the generated summary.
  • C is Correct: BERTScore uses contextual embeddings from pre-trained transformer models (e.g., BERT) to measure semantic similarity between generated and reference text, even when the exact wording differs. This addresses the case where the generated summary uses different medical terminology (synonyms, paraphrases) while preserving meaning.

Distractor Analysis

  • B is Incorrect: Simple accuracy assumes exact-match or token-level matching, which fails for open-ended text generation where multiple correct wordings are valid. Generative text nearly never produces exact matches with reference summaries.
  • D is Incorrect: Confusion matrices are designed for classification problems (true/false positives/negatives), not for evaluating generated text quality. They do not measure semantic or lexical similarity between summaries.
  • E is Incorrect: RMSE is a regression metric used to measure numeric prediction errors, not text similarity or quality. Comparing summary lengths does not assess content correctness or meaning preservation.

Key Takeaway: For text generation evaluation, combine overlap-based metrics like ROUGE (lexical) with embedding-based metrics like BERTScore (semantic) to capture both surface-level fidelity and meaning preservation.

Question 9 - LLM-as-a-Judge Evaluation Mode Selection

A legal technology company has deployed a contract review AI system using Amazon Bedrock that extracts key clauses and flags risk provisions from legal documents. The company's compliance team requires that model outputs be evaluated on subjective quality dimensions — legal reasoning soundness, explanation clarity, and factual faithfulness to source text. The evaluation must produce both a quantitative score and a narrative explanation justifying each score. The company has a limited budget and does not want to hire external human reviewers.

Which evaluation approach should the company use?

  • A) Configure an Amazon Bedrock Evaluations job using LLM-as-a-judge with a judge model that scores each response and explains the score with reasoning.
  • B) Configure an Amazon Bedrock Evaluations programmatic job that computes ROUGE and BERTScore metrics against reference contract clauses.
  • C) Configure an Amazon Bedrock Evaluations human-based job using Amazon Mechanical Turk workforce to rate each output on the subjective quality dimensions.
  • D) Use Amazon SageMaker Clarify to generate Shapley values that explain which input features contributed most to each risk flag.
Click to reveal Answer & Explanation

Correct Answer: A

Correct Answer Analysis

  • A is Correct: Amazon Bedrock Evaluations supports LLM-as-a-judge evaluation mode, where a judge model (e.g., another LLM) rates each response on custom quality dimensions and generates natural language explanations justifying its score. This satisfies the requirements for subjective quality assessment, quantitative scoring with narrative reasoning, and no external human reviewer dependency.

Distractor Analysis

  • B is Incorrect: Programmatic evaluations using ROUGE and BERTScore compute overlap and semantic similarity, but they cannot evaluate subjective dimensions like reasoning soundness, explanation clarity, or factual faithfulness. They do not produce narrative explanations justifying scores.
  • C is Incorrect: Human-based evaluation via Mechanical Turk requires external human reviewers, increasing cost, introduces review latency, and raises data privacy concerns for sensitive legal documents. The requirement explicitly states avoiding external human reviewers due to budget.
  • D is Incorrect: SageMaker Clarify generates model explainability outputs (feature attribution/Shapley values) for traditional ML models or large language models — it explains why a model produced a particular output, but it does not generate quality scores or narrative evaluations of output quality dimensions.

Key Takeaway: Amazon Bedrock Evaluations LLM-as-a-judge mode is the right choice when you need subjective quality scoring (reasoning, clarity, faithfulness) with narrative explanations — without hiring human reviewers.

Question 10 - RAG System Monitoring Separation

An e-commerce platform has deployed a customer service RAG system using Amazon Bedrock Knowledge Bases to answer product and order-related questions. The system occasionally returns answers that are fluent and confident but omit critical information present in the retrieved product documentation. The engineering team has instrumented the system to collect metrics but is currently only tracking end-to-end answer accuracy. They need to implement monitoring that isolates whether failures originate in the retrieval step or the generation step.

Which TWO metrics or monitoring approaches should the team implement?

  • A) Track Context Recall to measure whether the retrieval step returns all relevant document chunks containing the ground-truth answer.
  • B) Track Token Latency to measure the time each answer takes to generate and correlate with quality failures.
  • C) Track Faithfulness to measure whether the generated response accurately reflects the retrieved context without hallucination or omission.
  • D) Track Request Volume to identify traffic surges correlated with answer quality degradation.
  • E) Track Endpoint Error Rate from CloudWatch metrics to identify infrastructure failures during retrieval or generation.
Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: Context Recall (also called retrieval recall) measures the proportion of relevant document chunks successfully retrieved relative to the ground-truth relevant set. Low Context Recall indicates the retrieval step is failing to fetch relevant context — a retrieval failure.
  • C is Correct: Faithfulness measures whether the generated response accurately uses the retrieved context without adding hallucinated facts or omitting key information. Low Faithfulness while Context Recall is high indicates a generation failure (the model ignored or dropped relevant context).

Distractor Analysis

  • B is Incorrect: Token latency measures generation speed but does not indicate whether the response is factually complete or whether retrieval returned the right context. A fast response can still omit critical information.
  • D is Incorrect: Request volume correlation may identify load-related issues but does not isolate retrieval vs. generation failure modes. High traffic may trigger throttling or queueing, but this is an operational concern separate from RAG quality monitoring.
  • E is Incorrect: Endpoint error rate tracks infrastructure health (5xx errors, timeouts) but does not capture semantic quality failures like missing facts in fluent responses. A system can return 200 OK responses while still failing to retrieve or generate correct information.

Key Takeaway: In RAG systems, split monitoring into retrieval quality (Context Recall/Precision) and generation quality (Faithfulness/Answer Relevance) — fluent answers that omit facts are often generation failures on top of successful retrieval, and only separate measurement reveals the true root cause.