Skill 4.2.2: Configure and use tools to troubleshoot and analyze resources (for example, Amazon CloudWatch, Amazon Bedrock AgentCore Observability, AWS X-Ray)

Question 1 - Selecting the Correct Observability Tool for Agent Tracing

A financial services company has deployed a customer support agent using Amazon Bedrock Agents that orchestrates multiple tool calls to internal APIs (loan calculators, transaction history) and a knowledge base for documentation retrieval. Users are reporting intermittent slow responses (5–8 seconds) that cannot be reproduced consistently.

The ML engineering team needs to determine whether the latency originates from foundation model inference, knowledge base retrieval, tool API invocations, or agent orchestration overhead. They need a per-invocation trace that shows the sequence and timing of each step inside the agent workflow.

Which AWS service or feature should the team enable to gain this level of visibility? (Select ONE)

A) Configure Amazon CloudWatch Alarms on the Bedrock Agent's InvocationCount and TokenUsage metrics.
B) Enable AWS X-Ray tracing on the agent's IAM execution role and inspect the service map.
C) Enable Amazon Bedrock AgentCore Observability and inspect the per-invocation trace for inference, retrieval, and tool call timing.
D) Stream Bedrock Agent logs to AWS CloudTrail and filter for events with eventName: InvokeAgent.
E) Deploy a custom sidecar container that intercepts and logs HTTP requests between the agent and downstream services.

Click to reveal Answer & Explanation

Correct Answer: C

Correct Options Analysis

C is Correct: Amazon Bedrock AgentCore Observability provides dedicated per-invocation traces that distinguish model inference latency, knowledge base retrieval latency, individual tool API call durations, and orchestration overhead. This allows teams to pinpoint exactly which component within an agent invocation is causing the delay.

Distractor Analysis

A is Incorrect: CloudWatch metrics like InvocationCount and TokenUsage are aggregate counters and totals. They can tell you that agent usage is high or token spend is increasing, but they cannot show the per-invocation breakdown of which step (retrieval, tool call, inference) took the longest.
B is Incorrect: While AWS X-Ray is excellent for distributed tracing across independent microservices (e.g., Lambda, API Gateway, EC2), it does not natively instrument the internal steps of a Bedrock Agent's orchestration loop. The multi-step AgentCore trace is specifically designed for this purpose.
D is Incorrect: AWS CloudTrail records API activity (who called what, when, from which IP). It does not provide internal execution timing data for agent steps like tool API calls or retrieval latency.
E is Incorrect: This introduces significant operational overhead, requires custom interception logic, and may not capture the internal orchestration timings between Bedrock services accurately. It violates the principle of using native tools for agent-specific observability.

Key Takeaway: For Bedrock Agent troubleshooting, use AgentCore Observability for per-invocation execution traces; use CloudWatch for aggregate metrics; use X-Ray for tracing across external microservices.

Question 2 - Tracing a Latency Problem Across a Multi-Service ML Pipeline

An e-commerce company runs an ML inference pipeline that is invoked via Amazon API Gateway, processed by an AWS Lambda function, and served by an Amazon SageMaker AI real-time endpoint housing a product recommendation model. The Lambda function preprocesses the input and calls the SageMaker endpoint synchronously.

Users report that checkout-page recommendations are sometimes delayed by more than 3 seconds. The team has added CloudWatch logs to the Lambda function, but logs alone do not provide clear evidence of where the delay is occurring—whether in API Gateway, Lambda cold starts, or the SageMaker endpoint.

What is the MOST efficient approach to determine the source of latency? (Select ONE)

A) Enable AWS X-Ray tracing on both API Gateway and Lambda, and inspect the X-Ray service map and trace timelines to measure the upstream and downstream call latencies.
B) Increase CloudWatch log verbosity in the Lambda function to include timestamps before and after the SageMaker invocation.
C) Monitor the InvocationsPerInstance CloudWatch metric of the SageMaker endpoint and set a high-resolution alarm.
D) Deploy a third-party application performance monitoring (APM) tool using a custom sidecar.
E) Use AWS Cost Explorer to identify if the cost of the endpoint increased during reported latency periods.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

A is Correct: AWS X-Ray is the native AWS service for distributed tracing. Its service map displays the full request path (API Gateway → Lambda → SageMaker) and shows the latency of each hop by analyzing trace segments. This avoids guessing and pinpoints whether the delay is an upstream cold start, a network transfer time, or model inference time consuming the request.

Distractor Analysis

B is Incorrect: Although adding timestamps to logs improves local visibility inside the Lambda function, it lacks visibility into the time spent in API Gateway before the Lambda is invoked or network latency to the endpoint. It confirms Lambda execution is slow but not necessarily why (e.g., is it waiting on a slow model?).
C is Incorrect: InvocationsPerInstance in CloudWatch tells you the throughput of requests hitting the endpoint. It does not show the time between the Lambda call being initiated and the response being received from the network's perspective. It's a utilization metric, not an external latency metric.
D is Incorrect: This introduces significant operational cost and custom instrumentation (sidecar, agents, etc.) plus potential data egress costs, whereas X-Ray is a fully managed service with a native SDK for these AWS services.
E is Incorrect: AWS Cost Explorer tells you "how much money was spent" on inference, not "why requests are slow." There is no causal relationship between a temporary billing spike and a latency regression.

Key Takeaway: Use X-Ray to attribute latency to a specific component in a chain of services (API Gateway → Lambda → SageMaker). Using logs or aggregate CloudWatch metrics alone often only proves that latency exists but not where it resides.

Question 3 - Dashboard Construction for Token and Utilization Correlation

A machine learning team manages an AI assistant on Amazon Bedrock that has been seeing rising costs. The team wants to create a dashboard that allows leadership to instantly differentiate between cost increases driven by higher traffic volume versus cost increases driven by heavier/inefficient prompts (more input tokens per request).

Which combination of CloudWatch metrics should be plotted together on the dashboard? (Select TWO)

A) Invocations (request count)
B) InputTokenCount (total input tokens)
C) InvocationClientErrors
D) OutputTokenCount (total output tokens)
E) ModelInvocationLatency

Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

A is Correct: Invocations tracks the number of requests made to the model. If both Invocations and InputTokenCount rise together proportionally, the rising bill is driven by more traffic.
B is Correct: InputTokenCount tracks the total number of tokens processed from prompts. If InputTokenCount rises significantly against a flat Invocations count, it indicates that prompts are getting longer/heavier, leading to higher token bills.

Distractor Analysis

C is Incorrect: InvocationClientErrors is a reliability metric that tracks errors (e.g., 4XX semantics). While critical for an operations dashboard, it does not help distinguish cost drivers related to traffic volume vs. prompt size.
D is Incorrect: While OutputTokenCount is a major cost driver, tracking output tokens on its own does not directly answer the "traffic vs. heaviness" question. The correct pairing requires observing Invocations (traffic) against InputTokenCount (heaviness of prompts). High outputs could look identical for many small requests or a few long summaries.
E is Incorrect: ModelInvocationLatency is a performance indicator but does not directly explain why billing is rising.

Key Takeaway: A cost dashboard must correlate volume metrics (like Invocations) with consumption metrics (like InputTokenCount) to determine if you have a traffic problem or a prompt-engineering problem.

Question 4 - Troubleshooting a Bedrock Agent's Resource Consumption Pattern

A logistics company has deployed a Bedrock Agent to handle customer shipment queries. The agent invokes an external weather API and a database API. The development team notices that a single user query results in multiple API calls and agent turns, generating a large number of tokens.

The team needs to understand how many tool calls and which types of calls the agent makes during a specific user session to validate if the agent is stuck in a retry loop or making unnecessary calls.

Which tool provides the necessary detailed breakdown per invocation? (Select ONE)

A) Amazon CloudWatch Logs Insights filtering on the agent's CloudTrail ID
B) AWS Config timeline for the agent resources
C) Amazon Bedrock AgentCore Observability trace filtering on the sessionId to inspect the orchestration trace
D) Amazon SageMaker Model Monitor
E) AWS Billing Dashboard

Click to reveal Answer & Explanation

Correct Answer: C

Correct Options Analysis

C is Correct: Bedrock AgentCore Observability captures the full trace of an agent invocation. It allows developers to filter by sessionId and see every scaffold/planning step, tool call (API request and response), and retrieval query. This reveals if the agent is looping or duplicating calls.

Distractor Analysis

A is Incorrect: CloudTrail logs the API calls to the Bedrock control plane and data plane (e.g., InvokeAgent) but does not log the internal execution steps of the agent (e.g., what internal tool APIs were called by the model and how many times).
B is Incorrect: AWS Config tracks configuration changes and compliance state of resources over time. It does not track functional execution traces.
D is Incorrect: SageMaker Model Monitor is used to monitor model quality (data drift, outliers) on SageMaker hosted endpoints. It cannot monitor Bedrock Agents or their LLM orchestration behavior.
E is Incorrect: The Billing Dashboard might show that cost went up, but it doesn't expose the underlying trace data required to fix the agent's logic.

Key Takeaway: AgentCore Observability is the only native AWS service that maps the internal reasoning and tool usage of a Bedrock Agent back to a specific user session (sessionId).

Question 5 - Interpreting a Trace vs. Aggregate Metrics

An ML team is supporting a high-traffic Bedrock application. The Mean Time to Recovery (MTTR) for an incident is poor because when the ModelInvocationLatency average in CloudWatch spikes, engineers debate whether it was an upstream network issue or a backend model outage.

A senior engineer suggests using AWS X-Ray. How does AWS X-Ray fundamentally improve the team's ability to troubleshoot this compared to using only CloudWatch averages? (Select ONE)

A) X-Ray automatically eliminates cold starts in downstream services, reducing the average latency measured.
B) X-Ray provides a service map and trace segments, allowing engineers to pinpoint the exact component (e.g., an overloaded OpenSearch cluster) that caused a specific request's latency spike.
C) X-Ray reduces the cost of log ingestion by replacing CloudWatch Logs.
D) X-Ray analyzes token usage patterns to suggest cheaper models.
E) X-Ray enforces IAM least-privilege policies on Bedrock agents.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

B is Correct: Unlike aggregate metrics (which are averages or percentiles over time), X-Ray traces individual requests end-to-end. If an average latency goes up, looking at traces allows engineers to sort by latency and see exactly where a specific transaction spent its 5 seconds (e.g., it left Lambda at 10ms, but the downstream DynamoDB call took 4.8s).

Distractor Analysis

A is Incorrect: X-Ray is an observability tool; it does not modify application behavior. It observes cold starts but cannot prevent them.
C is Incorrect: X-Ray traces and CloudWatch Logs serve different purposes. X-Ray provides structured tracing data, not general application log storage.
D is Incorrect: Token usage optimization is handled by prompt engineering, caching, and model selection, under Bedrock features. X-Ray handles distributed latency tracing.
E is Incorrect: IAM controls security and access. X-Ray traces requests.

Key Takeaway: "Where did the time go?" is answered by traces (X-Ray), while "Is the system healthy right now?" is answered by metrics (CloudWatch).

Question 6 - Troubleshooting SageMaker Endpoint Auto-Scaling Failures

A media company has a SageMaker real-time endpoint for a video classification model. They configured target tracking auto-scaling to keep CPU utilization at 60%. However, during a viral traffic spike, requests timed out, and the team noticed the endpoint did not scale out.

Upon investigation, the engineer suspects the CloudWatch alarm is not firing. What is the MOST likely cause of this issue? (Select ONE)

A) The SageMakerVariantInvocationsPerInstance metric was used instead of CPU utilization.
B) The auto-scaling policy was created but the CloudWatch alarm is in "Insufficient data" state because the endpoint was not emitting CPU metrics for the specified instance type.
C) Auto-scaling is only supported for asynchronous endpoints, not real-time endpoints.
D) The fully trained model is too large to replicate to multiple instances dynamically.
E) The endpoint's instance count was manually raised earlier, which permanently disables auto-scaling.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

B is Correct: Taget tracking in SageMaker depends on CloudWatch alarms. If the alarm transitions to Insufficient Data, the scaling policy cannot trigger. SageMaker endpoints emit metrics like CPUUtilization automatically, but if there's a misconfiguration (e.g., the policy looks at GPUUtilization on a CPU instance) or if permissions are missing, the alarm never breaches.

Distractor Analysis

A is Incorrect: SageMakerVariantInvocationsPerInstance is the recommended metric for scaling language and image models to ensure balance across instances. CPU is also valid. The problem is not which metric, but the state of the alarm.
C is Incorrect: Auto-scaling is fully supported for real-time endpoints. Asynchronous endpoints scale down to zero; real-time historically requires at least one instance, but scaling out is standard.
D is Incorrect: SageMaker endpoints can scale out to multiple instances regardless of model size (as long as the instance memory fits the model). Even large models are replicated to multiple instances behind the load balancer.
E is Incorrect: Manual scaling does not permanently disable auto-scaling. Once you attach a scaling policy, it governs the desired instance count, overriding the original manual config.

Key Takeaway: When auto-scaling doesn't work during a spike, check the CloudWatch alarm state first. "Insufficient data" is the silent killer of target tracking policies.

Question 7 - Selecting a Purchasing Option for SageMaker vs. EC2

A biotech firm runs two distinct ML workloads:

A steady, 24/7 pharmacology model inference endpoint on Amazon SageMaker AI.
A batch preprocessing job run on Amazon EC2 spot instances.

The CFO requests that the team commit to savings plans to reduce the rate charged for the steady SageMaker inference baseline.

Which purchasing option covers Amazon SageMaker AI usage? (Select ONE)

A) Compute Savings Plans
B) EC2 Instance Savings Plans
C) SageMaker AI Savings Plans
D) SAAS Subscription Plans

Click to reveal Answer & Explanation

Correct Answer: C

Correct Options Analysis

C is Correct: SageMaker AI Savings Plans apply specifically to SageMaker ML instance usage (training, notebooks, inference). Committing to a hourly spend over 1 or 3 years yields significant discounts.

Distractor Analysis

A is Incorrect: Compute Savings Plans apply specifically to Amazon EC2, AWS Fargate, and AWS Lambda. They do not cover SageMaker ml.* instance types.
B is Incorrect: EC2 Instance Savings Plans apply to a specific family of EC2 instances (e.g., c5.large). SageMaker instances are separate service instance families and not covered.
D is Incorrect: "SAAS Subscription Plans" is a non-existent AWS pricing feature.

Key Takeaway: The word in the title is the easiest way to pass the exam. "SageMaker Savings Plans" covers SageMaker. "Compute Savings Plans" covers Compute (EC2, Lambda, Fargate).

Question 8 - Optimizing a Diurnal Traffic Pattern

An accounting software firm runs a SageMaker endpoint for an invoice-parsing model. Traffic peaks during 9 AM – 5 PM business hours and falls to near zero overnight. The model runs on a GPU instance that is heavily underutilized at night.

The firm wants to reduce costs while maintaining the current peak performance. Which solution best meets these requirements? (Select ONE)

A) Switch the model to Amazon Bedrock Pay-per-token pricing.
B) Configure an auto-scaling policy that scales the endpoint instance count down to zero during off-peak hours.
C) Right-size the instance to a smaller GPU type to handle the average workload.
D) Purchase a 1-year SageMaker Savings Plan for the existing instance type.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

B is Correct: The workload is highly cyclical. Real-time endpoints support scaling policies that add or remove instances based on demand. By scaling down (potentially to zero if using Serverless inference or asynchronous configurations, or simply fewer instances depending on config), the firm stops paying for idle GPU hours overnight while retaining full capacity during the day.

Distractor Analysis

A is Incorrect: A SageMaker hosted model (invoice parser) is a specific custom model. Simply moving it to Bedrock may not be possible if the exact architecture isn't available as a supported Bedrock FM. Furthermore, this changes the billing entirely.
C is Incorrect: Right-sizing to a smaller instance would save money 24/7, but it would fail to handle the "daytime peak." The constraint requires maintaining current peak performance.
D is Incorrect: A Savings Plan reduces the rate. If the instance is running idle overnight, you are still paying the (discounted) rate for idle compute. Savings plans cannot reduce usage; only auto-scaling can.

Key Takeaway: If traffic has a distinct day/night cycle, "right-sizing" alone only treats the symptom. You must configure auto-scaling to follow the load, releasing idle capacity.

Question 9 - Managing Foundation Model Token Costs (Context Caching)

A legal tech startup uses Amazon Bedrock to analyze long contracts. The system uses a 2,000-token system prompt with legal definitions, followed by 10,000 tokens of contract text. The team is running thousands of these daily and wants to reduce costs.

They decide to implement prompt caching. What must they do to ensure the cache is effective? (Select ONE)

A) Append a timestamp to the system prompt so the cache understands the version.
B) Place the static legal definitions prompt at the beginning of the input and keep dynamic contract text at the end.
C) Move all dynamic contract text to the system prompt to ensure the entire prompt is cached.
D) Increase the temperature parameter so the model uses cached tokens more frequently.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

B is Correct: Prompt caching (e.g., on Bedrock/Claude) works by matching the exact prefix of a prompt. If the static legal definitions are first, they are written to the cache on the first execution and subsequent executions read from the cache, saving time and cost. The dynamic contract text after it doesn't invalidate the prefix.

Distractor Analysis

A is Incorrect: Appending a timestamp inside the cached prefix means the prefix is different on every single request. This guarantees a cache miss every time, maximizing cost.
C is Incorrect: Dynamic content must remain in the non-cached portion. Putting it in the system prompt would again invalidate the cache constantly.
D is Incorrect: Temperature affects the randomness of the output, not the KV-cache lookup of the input.

Key Takeaway: Caching is String matching. Keep the string identical for the first N tokens. Any variance (timestamps, IDs) at the top completely voids the cache.

Question 10 - Translating Endpoint Best Practices to Bedrock FMs

A startup moves from self-hosting a classification model on SageMaker to using a large general-purpose foundation model on Bedrock for summarization. The engineering lead wants to optimize costs.

They initially try to use "Compute Savings Plans" to cover the Bedrock spend and attempt to "right-size" the instances. Neither works.

What is the correct translation of these IaaS/Endpoint cost optimization principles to Foundation Model inference in Bedrock? (Select TWO)

A) Replace "Right-sizing instances" with "Selecting a smaller foundation model that meets the accuracy requirement."
B) Replace "Purchasing SageMaker Savings Plans" with "Purchasing Bedrock Provisioned Throughput."
C) Replace "Auto-scaling instances" with "Increasing the number of Bedrock API keys to support more traffic."
D) Replace "Monitoring CPU Utilization" with "Monitoring SageMaker Model Monitor for neural network drift."
E) Replace "Multi-model endpoints" with "Reducing the prompt context length."

Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

A is Correct: If a Llama-3-70B model is doing work that a Llama-3-8B can handle, using the smaller model cuts the price per token significantly. This is the "right-sizing" of GenAI.
B is Correct: Bedrock Provisioned Throughput reserves a static block of model units (compute) for a fixed hourly rate (e.g. committing to 1 model unit for 1 year). This mirrors the concept of buying a Savings Plan for a SageMaker instance.

Distractor Analysis

C is Incorrect: Increasing API keys merely increases the number of authentication credentials. It does not scale compute capacity or reduce per-token costs.
D is Incorrect: "Monitoring CPU Utilization" (checking if the server is busy or idle) translates to "Monitoring Token Usage" (checking how many tokens are coming in/out). Model Monitor checks data drift, not compute load or token billing.
E is Incorrect: Multi-model endpoints solve the problem of dozens of different custom models sitting on separate idle servers. In generative AI, this translates to architectural decisions like caching or RAG design, not just shortening context.

Key Takeaway: Never apply endpoint infrastructure logic (Savings Plans for compute) to Token-based services (Bedrock On-Demand). Translate the goal (predictability) to the right mechanism (Provisioned Throughput).