Domain 4: Operating, Monitoring, and Securing ML and AI Solutions

Question: SageMaker Multi-Tenant Cost & Budget Alerts

Category: MLA – Operating, Monitoring, and Securing ML and AI Solutions

Review Bucket:

[ ] Bucket A (Term Mapping)
[ ] Bucket B (Missed Constraint)
[X] Bucket C (Knowledge Gap)

A media company runs machine learning workloads in Amazon SageMaker AI for several internal business units. Each unit receives a predetermined AWS spending allowance every month, and resources are labeled with department-specific tags so that costs can be tracked separately.

Finance wants an automated way to notify each business unit before its projected monthly charges surpass its assigned spending limit, with minimal administrative effort.

What solution should be implemented?

A) Create an AWS Cost Anomaly Detection monitor for each business unit. Scope each monitor by the cost allocation tags associated with the corresponding unit. Configure alert subscriptions to notify the unit when unusual spending patterns are detected during the month.
B) Configure a fixed budget in AWS Budgets for each business unit. Filter the budget by user-defined cost allocation tags assigned to each unit. Send an alert when forecasted monthly charges are expected to exceed the configured spending threshold.
C) Configure an auto-adjusting budget in AWS Budgets for each business unit. Filter the budget by AWS-generated cost allocation tags associated with each unit. Configure daily budget reports to provide updates on departmental spending.
D) Create a fixed budget in AWS Budgets for each business unit and filter the budget by user-defined cost allocation tags. Configure the budget to publish a notification to an Amazon SNS topic when actual monthly spending reaches the assigned limit. Subscribe an AWS Lambda function to the SNS topic to process the event and notify the corresponding business unit.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: AWS Budgets allows setting custom budget thresholds filtered by user-defined cost allocation tags (custom tags added by the team, like Department: Marketing). It natively supports alerts based on forecasted costs (notifying users before the limit is hit) without writing custom code or serverless infrastructure.

Distractor Analysis

A is Incorrect: AWS Cost Anomaly Detection identifies unexpected spending spikes based on historical trends; it does not track spending against a fixed monthly budget cap or send alerts based on forecasted monthly totals.
C is Incorrect: Auto-adjusting budgets dynamically change based on spending patterns over time, which violates the requirement for a predetermined monthly allowance. Furthermore, custom tags applied by business units are user-defined, not AWS-generated.
D is Incorrect: This option uses actual spending (which alerts after the limit is reached instead of before) and introduces unnecessary operational overhead by building a custom Lambda + SNS processing pipeline when AWS Budgets natively supports email/SNS notifications.

Key Takeaway: For cost alerts before exceeding a limit, choose AWS Budgets with Forecasted alerts. Avoid custom Lambda pipelines when native AWS service alerts exist ("minimal operational effort").

self-notes

Auto-adjust = change over-time, cannot predetermine

Practice Question: Amazon Bedrock Knowledge Base RAG Evaluation

Category: MLA – Deployment and Orchestration of ML and AI Workflows
Review Bucket:

[x] Bucket A (Term Mapping)
[ ] Bucket B (Missed Constraint)
[ ] Bucket C (Knowledge Gap)

A financial services company is assessing the retrieval quality of an Amazon Bedrock Knowledge Base by configuring a retrieve-only RAG evaluation job. Bedrock will invoke the Knowledge Base during the evaluation, and the company plans to use the Context coverage metric to determine whether retrieved content contains the information represented in the expected answers. An AI engineer must create the input prompt dataset in the format and structure supported by Bedrock.

Which of the following should be implemented?

A) Create a JSONL dataset in which each line contains a conversationTurns object with a prompt containing the evaluation query and referenceContexts containing the expected retrieved passages.
B) Create a JSONL dataset in which each line contains a conversationTurns object with a referenceResponses field containing the evaluation query and a prompt field containing the expected retrieved passages.
C) Create a JSONL dataset in which each line contains a conversationTurns object with a prompt containing the evaluation query and referenceResponses containing the expected ground-truth response.
D) Create a CSV dataset with a prompt column containing the evaluation query and a referenceResponses column containing the expected ground-truth response.

Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

C is Correct: Amazon Bedrock Model Evaluation jobs for Knowledge Bases require a .jsonl (JSON Lines) dataset format stored in Amazon S3. Each line must contain a conversationTurns array structured with a prompt object (holding the evaluation query) and a referenceResponses object (holding the ground-truth benchmark answer required to compute metrics like Context coverage).

Distractor Analysis

A is Incorrect: Bedrock RAG evaluation dataset schema uses referenceResponses to supply the ground-truth reference answer as the benchmark, rather than using a referenceContexts key for raw passages.
B is Incorrect: This option reverses the key fields—placing the query in referenceResponses and passages in prompt, which breaks the required Bedrock JSON schema validation.
D is Incorrect: Bedrock evaluation jobs do not support CSV file formats; datasets must be formatted as .jsonl (JSON Lines).

Key Decision Rules: * Bedrock Evaluation Dataset Format: .jsonl (JSON Lines) stored in Amazon S3. * RAG Evaluation Schema Keys: conversationTurns \(\rightarrow\) prompt (query) + referenceResponses (ground-truth reference answer). * Context Coverage Prerequisite: Ground-truth reference answer supplied in referenceResponses.

self-notes

referenceContexts doesn't exist

Practice Question: Least Privilege Access Control for ML Projects

Category: MLA – Operating, Monitoring, and Securing ML and AI Solutions
Review Bucket: [ ] Bucket A (Term Mapping) | [ ] Bucket B (Missed Constraint) | [x] Bucket C (Knowledge Gap)

An ML engineer must set up a secure environment for a project involving Amazon SageMaker, Amazon S3, and AWS Glue. The engineer needs to ensure that access to these resources is restricted based on the principle of least privilege.

Which IAM components are MOST appropriate for managing access control in this scenario?

A) AWS Resource Access Manager (RAM)
B) Amazon SageMaker Endpoint Policies
C) Service Control Policies (SCPs)
D) IAM Roles with Identity-Based Policies

Click to reveal Answer & Explanation

Correct Answer: D

Correct Option Analysis

D is Correct: To enforce the principle of least privilege across multiple AWS services (SageMaker, S3, AWS Glue) within a project, you use IAM Roles attached with Identity-Based Policies.
IAM Roles allow services (like a SageMaker Notebook or Glue ETL Job) or human users to temporarily assume permissions without using static, hardcoded credentials.
Identity-Based Policies (JSON documents) define granular actions (s3:GetObject, glue:GetTable, sagemaker:CreateTrainingJob) restricted strictly to the exact S3 buckets, Glue databases, or SageMaker resources required for the project.

Distractor Analysis

A is Incorrect: AWS Resource Access Manager (RAM) is used to share specific AWS resources (like subnets, Transit Gateways, or Glue Data Catalog tables) across different AWS accounts. It is not the core IAM mechanism for configuring least-privilege role permissions within an account/project environment.
B is Incorrect: SageMaker does not have native "Endpoint Policies" in the same way S3 has Bucket Policies. Controlling access to SageMaker endpoints or resources is done via IAM identity-based policies or VPC Endpoint Policies (which control network traffic going through a VPC endpoint).
C is Incorrect: Service Control Policies (SCPs) are organization-wide guardrails managed in AWS Organizations. SCPs specify the maximum allowed permissions for accounts or Organizational Units (OUs), but SCPs do not grant permissions on their own. Even with an SCP, you still need IAM Roles and identity-based policies to grant specific access.

Key Decision Rule: * Grant fine-grained, least-privilege permissions to users or services: Use IAM Roles with Identity-Based Policies. * Set maximum permission ceilings across an AWS Organization: Use Service Control Policies (SCPs). * Share resources across separate AWS Accounts: Use AWS RAM.

self-notes

┌────────────────────────────────────────────────────────────────────────────────────────┐ │ IAM ACCESS CONTROL ARCHITECTURE │ ├────────────────────────────────────────────────────────────────────────────────────────┤ │ 1. SCPs (Service Control Policies) ──► The Ceiling (Max permissions for the Account) │ │ 2. IAM Roles + Identity Policies ──► The Grantor (Gives exact actions to identities)│ │ 3. Resource-Based Policies ──► The Guard (Controls who can touch a resource) │ └────────────────────────────────────────────────────────────────────────────────────────┘

AWS Practice Question: Amazon Bedrock Knowledge Base RAG Evaluation for Hallucinations

Category: MLA – Operating, Monitoring, and Securing ML and AI Solutions

Review Bucket: [ ] Bucket A (Term Mapping) | [ ] Bucket B (Missed Constraint) | [x] Bucket C (Knowledge Gap)

A pharmaceutical company uses Amazon Bedrock Knowledge Bases to power a Retrieval Augmented Generation (RAG) based assistant. The assistant answers questions about drug interactions. The company verified that the assistant retrieves the correct documents from the knowledge base. Before releasing the assistant to a larger audience, the company wants to evaluate for hallucinations in the responses. The company will run an Amazon Bedrock knowledge base evaluation job.

Which evaluation job configuration will meet these requirements?

A) Select the retrieval only evaluation type. Configure the context relevance and context coverage metrics.
B) Select the retrieval and response generation evaluation type. Configure the citation coverage and completeness metrics.
C) Select the retrieval and response generation evaluation type. Configure the helpfulness and correctness metrics.
D) Select the retrieval and response generation evaluation type. Configure the faithfulness metric.

Click to reveal Answer & Explanation

Correct Answer: D

#### Correct Option Analysis

* **D is Correct:** To evaluate whether a model's generated response contains **hallucinations**, you must select the **Retrieval and Response Generation** evaluation type with the **`faithfulness`** metric. The `faithfulness` metric specifically measures whether the generated response is grounded strictly in the retrieved context passages, flagging any information introduced that is absent from the source documents.

#### Distractor Analysis

* **A is Incorrect:** Retrieval-only evaluation measures search quality (whether the right documents were retrieved), not whether the downstream LLM generation hallucinates.
* **B is Incorrect:** `Citation coverage` and `completeness` measure citation presence and answer thoroughness, not hallucination/groundedness.
* **C is Incorrect:** `Helpfulness` and `correctness` evaluate general output utility against user queries, but do not explicitly verify if the facts are hallucinated beyond the retrieved context chunks.

> **Key Decision Rules:**
> * **Detect GenAI / RAG Hallucinations:** Use the **`faithfulness` metric** under **Retrieval and Response Generation**.
> * **Evaluate Document Retrieval Quality:** Use **`context relevance`** or **`context coverage`** under **Retrieval-only**.

self-notes

Faithfulness = Groundedness (detects hallucinations). Helpfulness/Correctness = Answer quality.