Skill 4.2.8: Monitor agent resource consumption patterns

Question 1 - Bedrock AgentCore Observability Setup

A fintech company has deployed a customer service agent using Amazon Bedrock Agents integrated with CRM tool calls and knowledge base retrieval. The agent invokes multiple large language models per user request—planning, tool execution, and synthesis—creating a complex internal workflow. The ML engineering team wants to make the agent's resource consumption visible per invocation, including every internal inference call, tool invocation, and retry loop that occurs within a single user request, so they can attribute cost and latency accurately.

Which AWS service or feature is specifically designed to capture and analyze this per-invocation agent resource consumption? (Select ONE)

A) AWS CloudWatch natural metrics—Invocations, Latency, and Error Codes—for the Bedrock agent alias.
B) Amazon Bedrock AgentCore Observability with CloudWatch integration.
C) AWS X-Ray distributed tracing on the agent's Lambda functions.
D) AWS Cost Explorer filtered by the Amazon Bedrock service-level usage type.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

B is Correct: Amazon Bedrock AgentCore Observability captures detailed telemetry internal to an agent invocation—including each model inference call, tool execution, memory/retrieval step, and inter-agent coordination—and publishes it to CloudWatch, making per-invocation resource consumption visible.

Distractor Analysis

A is Incorrect: CloudWatch's default natural metrics for Bedrock provide account/alias-level aggregate totals (total invocations, latency, etc.) but do not explose the breakdown of multiple model calls, tool invocations, and retries that occur inside a single agent invocation.
C is Incorrect: X-Ray traces application-level requests across distributed components (e.g., Lambda functions calling Bedrock). It does not natively instrument the internal Bedrock agent orchestration loop—the planning call, tool call, and synthesis call remain opaque to X-Ray unless custom subsegments are written.
D is Incorrect: Cost Explorer shows service-level billing totals or tag-based aggregation, not per-invocation operational resource consumption (inference calls, tool calls, retries).

Key Takeaway: For Bedrock Agent per-invocation resource visibility, use Bedrock AgentCore Observability; for aggregates use CloudWatch natural metrics, and for cross-service bottlenecks use X-Ray.

Question 2 - Retry Loop Cost Anomaly Detection

A software-as-a-service (SaaS) provider operates an AI agent on Amazon Bedrock that calls an external weather API through a tool invocation. The operations team notices that Bedrock token usage has been climbing steadily over recent weeks, while the number of daily user requests has remained relatively flat. The team suspects the agent is entering retry loops on tool failures that are invisible to the application's top-level metrics. They want to implement monitoring for this AI-specific cost pattern with the least operational overhead.

Which TWO actions should the team take? (Select TWO)

A) Enable Amazon Bedrock AgentCore Observability and create a CloudWatch dashboard panel showing total model invocations per agent invocation over time, along with tool error counts.
B) Use AWS Cost Explorer to set a daily budget for Bedrock tokens and manually inspect the cost anomaly daily.
C) Configure AWS X-Ray tracing and write custom subsegments inside the agent's Lambda function to manually track tool call retries.
D) Add CloudWatch alarms on AgentCore Observability metrics such as modelInvocationCount or toolErrorCount exceeding a threshold to receive alerts when retry loops occur.
E) Rely on CloudWatch's built-in natural metric Invocations for the Bedrock agent alias to detect retry loops automatically.

Click to reveal Answer & Explanation

Correct Answers: A and D

Correct Options Analysis

A is Correct: AgentCore Observability exposes metrics like model invocation count, tool execution count, and tool errors. By building a dashboard that shows these per agent invocation, the team can visually verify whether the rising token bill correlates with increased internal model or tool calls (retry loops), rather than more user requests.
D is Correct: Setting alarms on AgentCore Observability metrics (especially modelInvocationCount or toolErrorCount) detects anomalous retry loops and alerts the team in real-time—precisely the "least operational overhead" way to catch the issue without manually watching dashboards.

Distractor Analysis

B is Incorrect: Cost Explorer is a historical/forecast cost reporting tool, not a real-time operational monitor. Manually checking budgets daily introduces high operational overhead and does not provide per-invocation resource consumption patterns.
C is Incorrect: X-Ray with custom subsegments requires significant engineering effort and modifications to agent code, violating the "least operational overhead" constraint. AgentCore Observability provides the visibility natively.
E is Incorrect: The agent alias-level Invocations natural metric reports only the number of top-level user requests. Retry loops occur internally and are hidden from this metric; it cannot differentiate between a request that executed cleanly and one that looped through 10 tool calls.

Key Takeaway: Agent cost discrepancies (flat traffic, rising tokens) are best diagnosed using per-invocation observability metrics—not account top-level cost or invocation totals.

Question 3 - Agent Cost Attribution Across Teams

A large enterprise runs multiple AI agent workloads on Amazon Bedrock across different business units (customer support, document processing, and code review). The finance team must attribute agent infrastructure costs to each team monthly, but they have found that agent invocations cannot be split by business unit using AWS Cost Categories because Bedrock charges are service-level and not per-agent. The team wants to implement cost attribution with minimal custom engineering.

Which approach will provide the most accurate per-agent or per-team resource consumption attribution? (Select ONE)

A) Apply distinct cost allocation tags to each Bedrock agent or agent alias and activate them in AWS Cost Explorer.
B) Use AgentCore Observability metrics manually exported from CloudWatch into spreadsheets per team.
C) Create separate AWS accounts for each business unit and configure AWS Organizations consolidated billing.
D) Use a single AWS Budget for all Bedrock token spend and divide by the number of requests per team.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Options Analysis

A is Correct: Amazon Bedrock supports AWS cost allocation tags (at the agent, alias, or session level, depending on resource type). By applying unique tags per agent or business unit and activating them in the Billing and Cost Management console, Cost Explorer can group and filter Bedrock spend per tagged resource—a fully native, low-overhead solution.

Distractor Analysis

B is Incorrect: Manually exporting metrics into spreadsheets is operationally heavy, error-prone, and does not tie directly to actual billed cost. It conflates operational telemetry with financial attribution.
C is Incorrect: Multi-account separation via AWS Organizations provides clean cost boundaries but introduces significant architectural overhead (separate IAM, networking, governance, and management requirements). Not "minimal engineering."
D is Incorrect: Splitting a total by request count gives an approximate average, not an accurate per-team figure. It ignores token usage differences (e.g., a team using long prompts pays more), and does not provide native cost reporting.

Key Takeaway: For AI cost attribution, apply cost allocation tags to Bedrock resources and use Cost Explorer for per-agent/team spend analysis—native, accurate, and low-overhead.

Question 4 - Agent Token Usage Dashboard Design

An e-commerce platform runs a generative AI shopping assistant on Amazon Bedrock. The assistant follows an intricate prompt flow, and recent changes to the system prompt increased input token size per request. The ML team needs to build a CloudWatch dashboard that helps them determine whether the rising Bedrock token bill is due to more user requests or heavier prompts per request. They want the dashboard panels to answer this question at a glance.

Which TWO CloudWatch metric combinations should the team graph together? (Select TWO)

A) Invocations for the Bedrock agent alias.
B) InputTokens per invocation aggregated over time.
C) Training job loss curve from a SageMaker experiment.
D) Average endpoint latency from the legacy SageMaker real-time endpoint.
E) Agent memory consumption in Java heap space.

Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

A is Correct: The Invocations natural metric from Bedrock tells the team how many user requests hit the agent over time—the "more traffic" dimension.
B is Correct: InputTokens per invocation reveals the average prompt size per request. When graphed alongside Invocations, the team can distinguish a traffic-driven bill increase (both metrics rise proportionally) from a prompt-heaviness increase (input tokens per invocation rises while invocations stay flat).

Distractor Analysis

C is Incorrect: Training loss curves are relevant to model quality and convergence, not inference cost patterns once the model is deployed.
D is Incorrect: Legacy endpoint latency is a performance metric from a different workload and has no direct relationship to Bedrock token billing.
E is Incorrect: Agent memory consumption in Java is a compute-level metric, not applicable to a serverless Bedrock agent that bills by token.

Key Takeaway: To separate traffic-driven from prompt-driven token cost growth, graph Invocations next to InputTokens (or InputTokens per invocation) on a single CloudWatch dashboard.

Question 5 - Token Usage Optimization via Prompt Caching

A media company uses Amazon Bedrock with Anthropic Claude to generate article summaries. The prompt includes a fixed system instruction (2,000 tokens describing style and output format) followed by the article text which changes per request. The team wants to reduce token costs while maintaining the same output quality, and they are considering prompt caching.

What is the MOST cost-effective way to structure the prompt for caching? (Select ONE)

A) Place the dynamic article text at the beginning of the prompt so it is processed first, and the static system instruction at the end.
B) Keep the static system instruction at the beginning as a stable prefix and place the dynamic article text at the end.
C) Interleave the static system instruction and article text throughout the prompt to minimize token usage.
D) Use a smaller model instead of prompt caching because caching does not reduce token costs.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

B is Correct: Prompt caching (e.g., Claude on Bedrock) reuses pre-computed KV attention states for matching token prefixes from the start of the prompt. By placing the static system instruction at the beginning and dynamic content at the end, the stable prefix can be cached and reused across requests, reducing per-request input token processing at a discounted cached rate.

Distractor Analysis

A is Incorrect: Putting dynamic content first invalidates the cache since the beginning of the sequence changes per request, causing the entire prefix to be re-processed at full uncached token rates.
C is Incorrect: Interleaving dynamic content throughout the prefix disrupts exact prefix matching and completely defeats caching; the cache will miss every time.
D is Incorrect: Prompt caching does reduce input token costs (cached tokens are billed at a lower rate than standard input tokens). While selecting a smaller model is also a valid lever, the question specifically asks for the best way to structure the prompt for caching.

Key Takeaway: Prompt caching saves input token costs only when a stable prefix appears at the start of the prompt and dynamic content is placed after it.

Question 6 - Endpoint Auto-Scaling Evaluation for Diurnal Pattern

A B2B analytics platform serves real-time recommendation model predictions from a SageMaker real-time endpoint to business customers. CloudWatch shows the endpoint instances are sized adequately for peak hours (9 AM–5 PM), but utilization drops below 10% overnight and on weekends. The ML engineering team wants to reduce monthly infrastructure costs while maintaining current daytime performance and avoiding any changes to client-side code.

Which solution is the most cost-effective? (Select ONE)

A) Migrate the workload to Amazon Bedrock on-demand token-based inference.
B) Configure SageMaker automatic scaling with a target-tracking policy on CPU utilization or invocations per instance to scale in at night and scale out before the daytime peak.
C) Replace the current instance family with a smaller one to handle overnight traffic and accept daytime performance degradation.
D) Convert the real-time endpoint to a multi-model endpoint and load the model only when flagged as active.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

B is Correct: SageMaker target-tracking auto scaling automatically adjusts instance count based on real-time metrics. It adds instances for peak demand and releases them during quiet periods, directly reducing overnight/off-peak compute costs while preserving daytime latency and performance.

Distractor Analysis

A is Incorrect: Bedrock serves foundation models, not custom recommendation models. Migrating a custom trained model to Bedrock token-based inference is architecturally infeasible and changes the business model significantly.
C is Incorrect: Downsizing the instance family to fit the low overnight traffic would lead to severe under-provisioning during peak daytime hours—causing latency spikes, throttling, and a poor customer experience.
D is Incorrect: Multi-model endpoints are useful when many different models share intermittent traffic; here there is a single model with a diurnal pattern. MMEs do not scale instances in/out based on demand automatically to the same degree, and introducing cold-start latency is unnecessary for this workload.

Key Takeaway: Diurnal utilization patterns with correctly sized peak instances call for auto scaling (scale capacity with demand), not right-sizing or architectural changes.

Question 7 - Savings Plan vs Provisioned Throughput Decision

A large pharmaceutical company runs two AI workloads on AWS:

Workload A: A custom drug-interaction classification model served on SageMaker real-time endpoints with stable, year-round traffic.
Workload B: A document summarization service using Anthropic Claude on Amazon Bedrock with high, predictable daily volume and identical prompt/response patterns.

The team wants to reduce unit costs for both workloads through commitment-based pricing.

Which TWO purchasing/capacity options should the team use? (Select TWO)

A) For Workload A, purchase a SageMaker AI Savings Plan.
B) For Workload A, purchase a Compute Savings Plan (covering EC2, Lambda, and Fargate).
C) For Workload B, switch from On-Demand Bedrock inference to Provisioned Throughput.
D) For Workload B, purchase a SageMaker AI Savings Plan since the workload uses AWS compute.
E) For both workloads, use AWS Reserved Instances for all underlying infrastructure.

Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

A is Correct: SageMaker AI Savings Plans apply specifically to SageMaker ML instance usage (real-time endpoints, training, batch, etc.) and reduce the hourly rate for a committed spend. This matches Workload A's steady baseline perfectly.
C is Correct: Amazon Bedrock Provisioned Throughput allows reserving model units for predictable, high-volume foundation model inference at a fixed hourly rate (lower effective per-token cost) rather than on-demand token pricing. This applies to Workload B.

Distractor Analysis

B is Incorrect: Compute Savings Plans cover EC2, AWS Fargate, and AWS Lambda—but not SageMaker ML instance types. So purchasing this for Workload A provides no discount.
D is Incorrect: SageMaker AI Savings Plans only cover SageMaker ML instance usage, not Amazon Bedrock token-based inference. Bedrock usage is billed by tokens, not instance hours.
E is Incorrect: AWS Reserved Instances apply to specific EC2 instances and do not cover SageMaker ML endpoint instances or Bedrock model inference. This is a common but incorrect conflation.

Key Takeaway: Match the commitment to the billing unit: SageMaker AI Savings Plans for SageMaker compute hours; Provisioned Throughput for Bedrock token workloads.

Question 8 - Agent Cost Amplification Monitoring

An AI agent built on Bedrock handles complex multi-step tasks. A single user request triggers:

A planning inference call (model A).
Three tool invocations (each requires a synthetic reasoning call model B).
A final synthesis call (model A).

The agent currently logs only the top-level request latency. The team wants to monitor the cost per agent invocation and detect unexpected amplification (e.g., a retry loop causing the tool call count to double or triple).

Which operational metric best captures this amplification? (Select ONE)

A) Total number of user requests per hour.
B) Total number of model inference calls per agent invocation (modelInvocationCount or equivalent).
C) Total tokens of all prompts across the account per month.
D) Average response time of the web application frontend.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

B is Correct: AgentCore Observability exposes the number of internal model inference calls (modelInvocationCount) per agent invocation. If a retry loop causes extra tool calls, this metric spikes even though the top-level user request count does not—revealing cost amplification.

Distractor Analysis

A is Incorrect: User request count is a leading indicator of demand, but does not capture internal amplification (e.g., one request generating 10 internal inference calls).
C is Incorrect: Account-wide token totals are an aggregate bill metric that cannot isolate per-invocation behavior or attribute amplification to a specific agent or request pattern.
D is Incorrect: Frontend response time is a performance latency measure and may be affected by many factors unrelated to agent internal cost (network, rendering, third-party APIs).

Key Takeaway: For agent cost amplification patterns, monitor the number of internal model inference calls per agent invocation—not the top-level request count or account-level token total.

Question 9 - Agent Resource Consumption Alarm Setup

An autonomous document processing agent on Bedrock uses a combination of embedding computation, vector database queries (OpenSearch Serverless), and foundation model inference. The team wants to be alerted whenever the number of internal tool calls per agent invocation exceeds 10 in a 5-minute period—indicating a likely retry loop or runaway agent execution.

Which alerting approach is most appropriate? (Select ONE)

A) Configure a CloudWatch alarm on the agent's Duration natural metric with a threshold of 10 seconds.
B) Configure a CloudWatch alarm on the AgentCore Observability metric (e.g., toolInvocationCount or modelInvocationCount) with a threshold and period matching the requirement.
C) Use AWS Budgets with an alert on monthly token spend exceeding $10,000.
D) Use AWS X-Ray service map alerts for agent retry detection.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

B is Correct: AgentCore Observability metrics such as tool or model invocation counts are published to CloudWatch, allowing the team to create a CloudWatch alarm on the specific metric with a threshold (e.g., >10 tool calls) and evaluation period (5 minutes). This satisfies the alerting requirement.

Distractor Analysis

A is Incorrect: The agent's Duration natural metric measures overall invocation time, not the number of internal tool calls. A long duration could be caused by model response latency, network delays, or other factors rather than retry loops.
C is Incorrect: AWS Budgets is a financial tracking tool that sends alerts on spend, not operational metrics such as tool invocation counts. It cannot detect retry loops in real time.
D is Incorrect: X-Ray service maps show distributed application topology and latency between AWS services. They do not natively expose agent internal tool call counts, and X-Ray does not provide built-in alerts for such fine-grained operational thresholds.

Key Takeaway: Use CloudWatch alarms on AgentCore Observability metrics for real-time detection of agent resource consumption anomalies (retry loops, tool call storms).

Question 10 - SageMaker Right-Sizing Evidence-Based Decision

A retail chain serves product recommendation predictions from a SageMaker real-time endpoint. Monthly CloudWatch data shows average CPU utilization of 28%, ever-peak utilization of 74%, and memory utilization averaging 40%. The ML team has not changed the instance type since initial deployment 12 months ago and wants to optimize cost proactively while maintaining acceptable peak performance and avoiding latency spikes.

Which initial step should the ML engineering team take? (Select ONE)

A) Immediately migrate the endpoint to Amazon Bedrock for token-based pricing.
B) Evaluate AWS Compute Optimizer recommendations for the SageMaker endpoint and right-size to a smaller instance family based on historical utilization.
C) Shut down the endpoint during non-business hours since the utilization is so low.
D) Purchase an EC2 Reserved Instance for the current oversized instance type to reduce cost without changing capacity.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

B is Correct: AWS Compute Optimizer analyzes utilization history (CPU, memory, network, etc.) and recommends smaller instance families that match observed workload patterns. Right-sizing based on measured evidence (28% average CPU, 74% peak) allows cost reduction while maintaining headroom for peak loads—a standard evidence-based optimization approach.

Distractor Analysis

A is Incorrect: Bedrock token-based inference is for foundation models, not custom-trained recommendation models. This migration is architecturally infeasible and unrelated to the cost issue.
C is Incorrect: A retail chain's real-time recommendation traffic may vary, but shutting down entire endpoints during business hours would cause service disruption for active users. The peak utilization of 74% indicates active use that must be served.
D is Incorrect: Purchasing an EC2 Reserved Instance does not apply to SageMaker ML instance types (which require SageMaker AI Savings Plans). Additionally, committing to an oversized instance preserves waste rather than optimizing capacity.

Key Takeaway: Before changing capacity, analyze measured utilization (CloudWatch, Compute Optimizer) to ensure the chosen instance family matches real workload demand, preserving peak headroom.