Skip to content

Task 2.3 - Analyze and evaluate the performance of ML and AI systems

Overview

Task 2.3, Analyze and evaluate the performance of ML and AI systems, is the step that turns a trained artifact into a defensible decision.

A numeric prediction and a paragraph of generated text are not judged the same way. This task covers reproducing an experiment, recording a baseline, comparing a candidate against what is already serving traffic, and choosing metrics that suit the output.

As you review these lessons for Task 2.3, check that you understand how to do the following.

  • Perform reproducible experiments (for example, by using MLflow on SageMaker AI, Amazon Bedrock evaluations, Amazon Bedrock Prompt Management).
  • Create model performance baselines and implement drift detection.
  • Compare the performance of shadow variants to production variants.
  • Explain model outputs.
  • Debug model convergence issues.
  • Apply comprehensive model evaluation techniques for traditional ML and GenAI models.
  • Implement integrated human evaluation frameworks (for example, human-in- the-loop workflows, text generation quality assessment).
  • Apply natural language processing (NLP) evaluation metrics (for example, bilingual evaluation understudy [BLEU], Recall-Oriented Understudy for Gisting Evaluation [ROUGE], BERTScore, semantic similarity).
  • Perform AI evaluation (for example, model output assessment, content quality validation, bias detection, LLM-as-a-judge frameworks).
  • Configure RAG system monitoring, including retrieval accuracy assessment.

AWS services overview

The services here answer three different questions: what happened in an experiment, how a candidate behaves against live traffic, and how good an output actually is. Recognize which one each service answers, because reaching for the wrong one produces a number nobody can act on. Review each service to learn more.

AWS Service Description
Amazon SageMaker AI with MLflow Fully managed MLflow tracks each run with its parameters, metrics, and artifacts, and its tracing records the inputs and outputs at each step of a generative AI application.
Amazon SageMaker AI shadow tests Deploys a shadow variant beside a production variant on one endpoint and routes it a copy of live requests, while only the production responses go back to the caller.
Amazon Bedrock evaluations Runs evaluation jobs three ways, computed programmatically, scored by a judge model, or rated by human workers, against models and against retrieval sources.
Amazon Bedrock Prompt Management Stores prompts with variables and versions, and compares prompt variants so a wording change becomes a tracked, repeatable comparison.
Amazon Bedrock Knowledge Bases The retrieval source whose accuracy an evaluation measures separately from the quality of the answer generated on top of it.
Amazon CloudWatch Holds the metrics and logs that make a baseline measurable and a departure from that baseline visible over time.

Knowledge Check

A team wants to compare a candidate variant against the variant currently serving production, by using live requests, without changing what callers receive. Which option fits? (Choose 1)

  • A. A programmatic evaluation job in Amazon Bedrock
  • B. A shadow test in Amazon SageMaker AI
  • C. An MLflow tracking run in Amazon SageMaker AI
  • D. A prompt variant comparison in Amazon Bedrock Prompt Management
Answer & Explanation

Correct Answer: B. A shadow test in Amazon SageMaker AI

Explanation:
SageMaker AI shadow testing deploys a candidate model variant alongside an active production variant on the same endpoint. Live inference requests are routed to the production variant, which returns responses to the caller. Simultaneously, a copy of the live request is asynchronously sent to the shadow variant. This allows teams to log and compare latency, predictions, and operational metrics on real production traffic with zero impact or risk to the end-user experience.

Why the other options are incorrect: * A. A programmatic evaluation job in Amazon Bedrock: Programmatic evaluations run automated scoring algorithms (e.g., ROUGE, BLEU, or BERTScore) against pre-compiled evaluation datasets offline, rather than live request streams. * C. An MLflow tracking run in Amazon SageMaker AI: MLflow tracks training runs, hyperparameter logs, model artifacts, and execution traces; it does not perform live endpoint traffic mirroring. * D. A prompt variant comparison in Amazon Bedrock Prompt Management: Prompt Management provides version control and side-by-side comparison of prompt text templates, rather than handling real-time endpoint deployment traffic.

AWS skills overview

In this lesson, you review the skills behind Task 2.3 as a light refresher. The ten skill statements group into four topics: reproducible experiments and baselines, comparing variants and debugging a run that will not converge, evaluating predictions and generated text, and the human and automated judgment a generative AI system needs. Read through them and notice which ones feel shaky enough that you want to come back to them before exam day.

Reproducible experiments and performance baselines

In practice, this group of skills lets the machine learning engineer answer the question that follows every promising result, which is whether it can be reproduced.

Be comfortable doing the following:

  • Recording each run as an experiment: the parameters it used, the metrics it produced, and the artifacts it wrote, so two runs can be compared later without reconstructing them from memory.
  • Tracing a generative AI application through its steps, capturing the inputs, outputs, and metadata at each one, so an unexpected answer can be traced back to the code, data, or parameters that produced it.
  • Establishing a baseline from a period of known-good behavior, and watching for drift as a departure from that baseline in the metrics that matter, whether that is prediction quality, the distribution of inputs arriving, or the operational metrics an endpoint emits.

A baseline is only as useful as the thing it measures. Recording a single accuracy figure at launch gives you a number to compare against, and it will not tell you why the comparison moved. Recording the inputs alongside the outcome tells you whether the data changed, the world changed, or the model changed. The same discipline applies to prompts: a stored, versioned prompt with a comparison between variants makes a wording change a measurable event rather than an unexplained shift in behavior.

Knowledge Check

Two runs of the same training configuration produce different scores, and afterward nobody can identify what differed. Which practice would have made the difference visible? (Choose 1)

  • A. Recording every run with its parameters, metrics, and artifacts in an experiment tracker
  • B. Increasing the size of the training instance
  • C. Reporting only the stronger of the two scores
  • D. Deleting the earlier run before starting the later one
Answer & Explanation

Correct Answer: A. Recording every run with its parameters, metrics, and artifacts in an experiment tracker

Explanation:
Experiment tracking tools (such as Amazon SageMaker AI with MLflow or SageMaker Experiments) automatically record and version every training run, including input parameters, code commits, dataset versions, environment dependencies, random seeds, and output metrics. When two seemingly identical runs yield different results, diffing the tracked runs in an experiment tracker immediately highlights underlying variance (e.g., unseeded random initializations, environment package drifts, or data source mutations).

Why the other options are incorrect: * B. Increasing the size of the training instance: Upgrading compute resources affects hardware throughput and execution speed, but does not provide tracking or auditing capabilities for run configurations. * C. Reporting only the stronger of the two scores: Cherry-picking top results hides variance and non-determinism, failing to solve the root issue of unidentifiable run differences. * D. Deleting the earlier run before starting the later one: Destroying historical execution logs completely eliminates data lineage and renders comparison impossible.

Comparing variants and debugging convergence

In practice, this group of skills lets the machine learning engineer validate a change against reality before customers feel it, and read a failing training run instead of restarting it hopefully.

Be comfortable doing the following:

  • Comparing a shadow variant against a production variant on live requests, where the shadow variant receives a copy of the traffic, only the production variant answers the caller, and the shadow responses are logged for offline comparison or discarded.
  • Recognizing what a shadow comparison is for, which is validating a change to the model, the container, or the instance type against real request patterns, including operational metrics such as latency and error rate.
  • Debugging convergence: a loss that oscillates and never settles points at a learning rate set too high, a loss that barely moves points at a rate too low or a model without the capacity for the signal, and a loss that improves on training data while validation worsens points at overfitting rather than at a broken run.

The reason a shadow comparison is worth the setup is that offline evaluation and live traffic disagree in ways nobody predicts. A model that scored well on a stored dataset can still meet request shapes, payload sizes, and concurrency the dataset never contained. Note also that some endpoint configurations are excluded from shadow testing, so the pattern is a design consideration rather than something to assume is available.

Knowledge Check

Training loss oscillates and never settles across epochs. Which hyperparameter is the FIRST one to examine? (Choose 1)

  • A. The learning rate
  • B. The number of output classes
  • C. The instance count for the training cluster
  • D. The storage format of the training data
Answer & Explanation

Correct Answer: A. The learning rate

Explanation:
Oscillating or unstable loss curves during gradient descent typically indicate that the learning rate is set too high. When the learning rate is overly aggressive, weight updates repeatedly overshoot the local loss minimum, causing the objective function value to bounce back and forth unpredictably rather than converging smoothly. Reducing the learning rate or applying a learning rate decay scheduler stabilizes optimization.

Why the other options are incorrect: * B. The number of output classes: The number of target classes is defined by the dataset problem formulation (e.g., binary vs. multi-class classification) and is fixed rather than a tuned hyperparameter that causes loss oscillation. * C. The instance count for the training cluster: Scaling the instance count changes total compute throughput and distributed worker configuration, but does not dictate gradient descent step size unless mini-batch sizes are altered without adjusting the learning rate accordingly. * D. The storage format of the training data: Data format (e.g., RecordIO, Parquet, or CSV) impacts file I/O speed and read efficiency, but has no effect on loss function convergence behavior.

Evaluating predictions, generated text, and explaining outputs

In practice, this group of skills lets the machine learning engineer choose a metric that matches the output, and then explain the result to somebody who was not in the room.

Be comfortable doing the following:

  • Selecting evaluation techniques by problem type for traditional models, and holding out data honestly, so a reported score describes performance on data the model has not seen.
  • Applying natural language processing metrics for generated text, where overlap-based measures compare a response against reference text and embedding-based measures compare meaning rather than exact wording, and knowing that a single one of them rarely settles a question about quality.
  • Explaining an output at the level the audience needs: which inputs moved a prediction for a traditional model, and for a generated answer, which retrieved passage the response was built from, since a retrieval approach can point at its source while a customized model cannot.

Generated text breaks the habit of a single accuracy figure, because two very different wordings can both be correct. That is why overlap measures and embedding-based measures are read together, and why a judgment step often sits on top of them. It is also why explanation and evaluation are not the same activity. A score says how well the system did, and an explanation says why this particular answer came out the way it did, which is what an auditor, a regulator, or an unconvinced stakeholder is asking for.

Knowledge Check

Why is a single accuracy figure a weak summary of a text generation system? (Choose 1)

  • A. A generated answer can be correct in more than one wording, so quality needs measures that compare meaning as well as overlap
  • B. Generated text cannot be evaluated without human raters
  • C. Accuracy applies only to regression problems
  • D. Text generation offers no reference outputs to compare against
Answer & Explanation

Correct Answer: A. A generated answer can be correct in more than one wording, so quality needs measures that compare meaning as well as overlap

Explanation:
Text generation is an open-ended task where valid responses can take virtually infinite phrasing forms while remaining factually correct and contextually accurate. Standard "accuracy" (exact-match classification) fails because a generated sentence might mean the exact same thing as a reference answer while sharing few identical words. Comprehensive GenAI evaluation requires multi-dimensional evaluation metrics: * Semantic/Embedding Similarity & LLM-as-a-Judge: Assessing whether the underlying meaning matches the target. * N-gram Overlap (ROUGE / BLEU): Measuring lexical overlap against reference text. * RAG Triad Metrics: Evaluating Faithfulness, Answer Relevance, and Context Relevance separately.

Why the other options are incorrect: * B. Generated text cannot be evaluated without human raters: While human evaluation is valuable for nuanced preference rating, automated programmatic evaluation (e.g., ROUGE, BLEU, BERTScore) and automated LLM-as-a-judge patterns (Amazon Bedrock Evaluations) are standard practices. * C. Accuracy applies only to regression problems: Accuracy is a standard metric for classification tasks (not regression), but it is ill-suited for open-ended natural language generation. * D. Text generation offers no reference outputs to compare against: Standard evaluation benchmark datasets (e.g., MMLU, GSM8K, or custom domain evaluation sets) explicitly include ground-truth reference passages and human-curated target answers.

Human evaluation, AI evaluation, and retrieval monitoring

In practice, this group of skills lets the machine learning engineer assemble an evaluation that mixes computation, model judgment, and human judgment, and then keep it running after launch.

Be comfortable doing the following:

  • Choosing among evaluation modes: a programmatic job that computes scores over a prompt dataset, a judge model that scores each response and explains the score, and a job whose ratings come from human workers, such as employees or subject-matter experts.
  • Placing human review where judgment is genuinely required, including a review step inside the workflow for outputs that carry consequence, and treating the reviewers' ratings as data rather than as anecdote.
  • Monitoring a retrieval system in two parts, measuring whether retrieval returned the relevant passages and whether the generated response answered the question, and keeping an eye on content quality and bias in the outputs the application actually returns.

Splitting retrieval from generation is the single most useful habit in this group, because a fluent answer that omits a fact in the source documents is a retrieval failure wearing a generation costume. Measure the retrieval step and the generation step separately and the cause stops being a guess. One practical note on running these jobs: an automated evaluation job carries no browser-facing display requirement, while a human-based job created in the console does, because prompts and responses have to be presented to the people doing the rating.

Knowledge Check

A retrieval application returns fluent answers that omit facts present in the source documents. Which measurement isolates the cause? (Choose 1)

  • A. Measure retrieval separately from generation
  • B. Measure only the quality of the final answer
  • C. Compare the answer against a second model's answer
  • D. Count the tokens each answer consumes
Answer & Explanation

Correct Answer: A. Measure retrieval separately from generation

Explanation:
In a Retrieval-Augmented Generation (RAG) system, omissions can stem from two distinct failure points: 1. Retrieval Failure (Context Recall): The vector search failed to fetch the relevant document chunks containing the missing facts. 2. Generation Failure (Context Utilization / Faithfulness): The retriever successfully fetched the facts, but the generator (LLM) ignored or dropped them when writing the final answer.

Evaluating retrieval metrics (e.g., Context Precision and Context Recall) independently from generation metrics (e.g., Faithfulness and Answer Relevance) isolates whether the root cause lies in chunking/embedding indexing or LLM prompt instruction-following.

Why the other options are incorrect: * B. Measure only the quality of the final answer: Evaluating only the end-to-end response treats the pipeline as a single black box, making it impossible to determine whether the missing facts were caused by poor vector retrieval or poor LLM generation. * C. Compare the answer against a second model's answer: Running an LLM-as-a-judge comparison without inspecting the retrieved context chunks fails to distinguish between missing context and LLM generation hallucination/omission. * D. Count the tokens each answer consumes: Token count measures output length, which does not reflect factual accuracy, completeness, or retrieval precision.

Knowledge Check

A candidate model has to be compared against the model in production by using live requests, and callers have to keep receiving the production responses. Which approach meets both conditions? (Choose 1)

  • A. Route a copy of live requests to a shadow variant and log its responses for offline comparison
  • B. Replace the production variant with the candidate and watch the error rate
  • C. Score both models on a stored prompt dataset before deploying either one
  • D. Send half of the live requests to the candidate and half to production
Answer & Explanation

Correct Answer: A. Route a copy of live requests to a shadow variant and log its responses for offline comparison

Explanation:
SageMaker AI shadow testing (shadow deployment) deploys a candidate model variant alongside an existing production variant on the same endpoint. It asynchronously forks or routes a copy of real-time production traffic to the candidate (shadow) variant while continuing to serve production model responses directly back to the callers. This satisfies both requirements: testing against live requests without impacting the end-user experience or exposure risk.

Why the other options are incorrect: * B. Replace the production variant with the candidate and watch the error rate: In-place replacement exposes live users directly to the candidate model's predictions, violating the constraint that callers must keep receiving production responses. * C. Score both models on a stored prompt dataset before deploying either one: Offline evaluation uses pre-recorded static datasets rather than testing against live request traffic. * D. Send half of the live requests to the candidate and half to production: This describes A/B testing (or canary traffic splitting). In A/B testing, 50% of calling applications directly receive candidate responses rather than production responses.