Task 4.1 - Monitor ML and AI model inference and performance
Overview
Task 4.1, Monitor ML and AI model inference and performance, is about knowing how a deployed model behaves without waiting for someone to complain.
A model that scored well at training time degrades quietly as the data it sees changes, and an AI application adds failure modes of its own. What happens when a tool call fails, a response stops partway, or an agent keeps coordinating instead of finishing? Each of those needs its own signal.
As you review these lessons for Task 4.1, check that you understand how to do the following.
Monitor model performance in production by using Amazon CloudWatch generative AI observability, Amazon Bedrock Model Evaluation, and drift detection pipelines. Monitor workflows to detect anomalies or errors in data processing or model inference. Detect changes in data distribution that can affect model performance. Monitor model performance in production by using A/B testing. Monitor and automate the management of agent performance and coordination (for example, coordination failure detection, truncated streaming, tool failures). Configure AI-specific performance monitoring for foundation models (FMs), such as Amazon Bedrock evaluations.
AWS services overview
The services here divide into three jobs: collecting the measures a workload emits, judging whether the output is still good enough, and tracing one request through the components that produced it. Recognize which job each service does, because a monitoring gap is usually a missing job rather than a missing dashboard. Review each service to learn more.
| AWS Service | Description |
|---|---|
| Amazon CloudWatch | Collects metrics and logs from the resources behind a workload, holds the dashboards that display them, and raises an alarm when a measure crosses a threshold that was set for it. |
| Amazon CloudWatch generative AI observability | Pre-built views over generative AI workloads, covering model invocations and Amazon Bedrock AgentCore agents, with token usage, latency, error and throttling measures, and end-to-end prompt tracing. |
| Amazon Bedrock model evaluation | Runs evaluation jobs that score model and knowledge base responses. An automatic job computes metrics, a judge model scores responses and explains each score, and a human-based job collects ratings from reviewers. |
| Amazon Bedrock AgentCore Observability | The default telemetry for agents running on AgentCore Runtime. Its OpenTelemetry-compatible instrumentation captures model inference calls, tool invocations, and memory operations without each agent adding its own spans. |
| AWS X-Ray | Traces a request across the components that handled it, so a slow or failing step is attributed to one of them rather than guessed at from an aggregate. |
| Amazon SageMaker AI | Hosts models behind endpoints, publishes invocation and instance metrics for them, and can place more than one production variant behind a single endpoint. |
Knowledge Check
A team needs one place that shows token usage, latency, and error rates for model invocations, plus end-to-end prompt traces for an agent that calls tools. Which option provides that view? (Choose 1)
- A. Amazon CloudWatch generative AI observability
- B. AWS CloudTrail
- C. AWS Config
- D. AWS Cost Explorer
Answer & Explanation
Correct Answer: A. Amazon CloudWatch generative AI observability
Explanation:
Amazon CloudWatch generative AI observability provides specialized, pre-built dashboards designed specifically for LLMs and agentic workloads. It aggregates model-level metrics—including token consumption (prompt/completion tokens), request latency, error rates, and throttling events—alongside distributed X-Ray prompt traces for agent executions (capturing reasoning steps, model invocations, and tool calls) in a single consolidated interface.
Why the other options are incorrect: * B. AWS CloudTrail: AWS CloudTrail logs management and data-plane API calls across your AWS account for audit, compliance, and governance purposes, but does not render request latency metrics, token consumption counts, or prompt execution traces. * C. AWS Config: AWS Config tracks resource deployment configurations, relationship histories, and compliance rules; it has no visibility into operational inference metrics or agent execution telemetry. * D. AWS Cost Explorer: AWS Cost Explorer provides high-level financial reporting and cost visualization over daily or monthly billing cycles, but lacks real-time operational telemetry like latency, token counts per invocation, or distributed traces.
AWS skills overview
In this lesson, you review the skills behind Task 4.1 as a light refresher. Each task groups a handful of related skills into a few topics. Read through them and notice which ones already feel familiar, and which ones feel shaky enough that you want to come back to them before exam day.
Monitoring performance and detecting drift
In practice, this group of skills lets the machine learning engineer notice a model getting worse before the business notices.
Be comfortable doing the following:
- Watching a production model against a recorded baseline, using the metrics an endpoint or a model invocation emits and the evaluation jobs that score output quality.
- Detecting a change in the distribution of the data arriving at the model, and building that detection as a pipeline rather than looking for it by hand.
- Separating a shift in the inputs from a shift in what a correct answer looks like, because the two are found different ways.
A drift detection pipeline has four moving parts and it is worth being able to recite them: capture a baseline from a period the team agrees was stable, capture the same measure from current traffic, compare the two distributions with a statistic suited to the data, and alert when the distance passes a threshold that was agreed in advance. For a generative AI application the measure is usually the distribution of input prompt embeddings, and the comparison needs a statistic that behaves well in many dimensions. That pipeline finds data drift. It does not find concept drift, where the prompts look much as they always did but the answer users now expect has changed, which surfaces in downstream business measures and user feedback instead.
Knowledge Check
Prompts arriving at a production assistant remain statistically close to the recorded baseline, and the answers users accept have changed. Which condition does that describe? (Choose 1)
- A. Concept drift
- B. Data drift
- C. A throttling limit on the model invocation call
- D. A failure in one of the tools the assistant calls
Answer & Explanation
Correct Answer: A. Concept drift
Explanation:
Concept drift occurs when the statistical relationship between the input features and target outputs changes over time (\(P(Y|X)\) shifts), even if the distribution of the input data (\(P(X)\)) stays constant. In this scenario, the incoming prompts match the baseline input distribution, but what users consider an acceptable answer (the output target) has shifted.
Why the other options are incorrect: * B. Data drift: Data drift (or covariate shift) occurs when the statistical distribution of the input data (\(P(X)\)) changes over time—such as users suddenly submitting completely different types or topics of prompts—while the underlying input-to-output mapping remains unchanged. * C. A throttling limit on the model invocation call: Throttling triggers HTTP 429 error codes or request failures due to concurrency/rate limits; it does not alter user acceptance behavior on completed responses. * D. A failure in one of the tools the assistant calls: Tool failures lead to explicit runtime exceptions, empty tool outputs, or fallback error messages rather than a subtle shift where validly generated responses no longer meet user expectations.
Detecting anomalies and errors across the pipeline
In practice, this group of skills lets the machine learning engineer find the step that broke when the symptom shows up somewhere else.
Be comfortable doing the following:
- Monitoring the data processing steps that feed a model, not only the inference call that a user experiences.
- Alarming on the measures that indicate an anomaly, including error rates, throttling, invocation counts that fall to nothing, and durations that grow run over run.
- Reading a completed job as a claim rather than a guarantee, and checking the output it produced.
Most production surprises in this task are handoff failures, and they share a shape: one component reports success while handing the next component something it cannot use. A feature job that finishes with an empty partition, a schema change that silently nulls a column, a retry that writes a record twice. The inference call is where the symptom appears, so the monitoring that catches the cause has to sit upstream of it and measure output, not only completion. That is also why an alarm on a flat line matters as much as an alarm on a spike: an invocation count that drops to zero is a broken pipeline, and a threshold written only for high values never fires on it.
Knowledge check
A scheduled data processing job reports success, and model invocation errors climb shortly afterward. Which monitoring gap does that pattern reveal?
A scheduled data processing job reports success, and model invocation errors climb shortly afterward. Which monitoring gap does that pattern reveal? (Choose 1)
- A. The endpoint hosts too few production variants for the traffic
- B. The pipeline monitors job completion without checking the output the job wrote
- C. The account has no cost allocation tags applied to the workload
- D. The model was scored by a judge model rather than by human reviewers
Answer & Explanation
Correct Answer: B. The pipeline monitors job completion without checking the output the job wrote
Explanation:
A data processing or ETL job can finish with a successful exit code (0) while still producing corrupt, truncated, schema-mismatched, or empty output files. If pipeline monitoring relies solely on job completion status without performing data quality validations (e.g., verifying record counts, non-null constraints, schema integrity, or anomaly detection), downstream model endpoints will encounter runtime errors as soon as they attempt to read or infer on the malformed output.
Why the other options are incorrect: * A. The endpoint hosts too few production variants for the traffic: The number of production variants affects traffic splitting and A/B testing capacity, not data-driven invocation failures following a batch processing job. * C. The account has no cost allocation tags applied to the workload: Cost allocation tags are used strictly for financial reporting and resource tagging; they have no impact on runtime functionality or system error rates. * D. The model was scored by a judge model rather than by human reviewers: LLM-as-a-judge scoring relates to model evaluation and response quality benchmarks, not runtime invocation exceptions triggered by bad data inputs.
Comparing model versions with production traffic
In practice, this group of skills lets the machine learning engineer prove a new model version is better on real traffic before it carries all of it.
Be comfortable doing the following:
- Placing more than one production variant behind a single endpoint and setting the weights that decide how live traffic is distributed between them.
- Measuring both variants on the same metrics over the same window, so the comparison is a comparison and not two separate observations.
- Recognizing when live traffic should reach only one variant, and validating the other on copied traffic instead.
The mechanism is simpler than it sounds. A variant is a model plus the serving resources behind it, several variants can sit on one endpoint, and traffic is distributed according to the ratio of each variant's weight to the sum of the weights. That is what makes a small exposure possible: give the candidate a small weight, watch it on the same measures as the incumbent, and shift weight as evidence accumulates. When even a small share of real responses is too much risk, a shadow variant receives a copy of the traffic and its responses are recorded rather than returned, which tests the serving stack without any user seeing its output.
Knowledge check
An engineer needs a candidate model version to serve a small share of live requests while the current version serves the rest, and both versions must be measured together. Which configuration supports that?