Skill 4.3.5: Troubleshoot and debug security issues in ML and AI systems

Question 1 - Diagnosing Model Endpoint Access Denial

A financial services company has deployed a fraud detection model on Amazon SageMaker with a real-time inference endpoint. The endpoint is configured inside a VPC with a security group that allows inbound traffic only from the company's application tier on port 443.

The ML engineering team receives reports that the application can no longer invoke the endpoint after a recent IAM policy change. The application logs show repeated AccessDeniedException errors when calling sagemaker:InvokeEndpoint. CloudTrail confirms the calls are reaching the SageMaker service.

What is the MOST LIKELY root cause and the appropriate remediation?

A) The VPC security group is blocking inbound traffic from the application tier, so the security group rules must be expanded to allow all inbound traffic from the VPC CIDR range.

B) The IAM role attached to the application has lost the sagemaker:InvokeEndpoint permission for that specific endpoint resource, and the role's policy should be updated to restore least-privilege access to that endpoint ARN.

C) The SageMaker endpoint has exceeded its maximum concurrent invocation limit, so the invocation rate should be reduced by implementing exponential backoff in the application.

D) The model artifact in Amazon S3 has been encrypted with a different KMS key, preventing the endpoint from loading the model, so the endpoint's KMS key policy must be updated.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: The CloudTrail logs confirm that the API calls are successfully reaching SageMaker (ruling out network issues), and the explicit AccessDeniedException error precisely indicates an IAM authorization failure. Since the IAM policy was recently changed, the sagemaker:InvokeEndpoint permission for that specific resource ARN was likely removed or narrowed unexpectedly. The remediation is to restore the exact permission on the endpoint ARN.

Distractor Analysis

A is Incorrect: CloudTrail confirms the calls reach SageMaker, which means the network path through the security group is functioning. Security group issues would manifest as timeouts or connection failures, not AccessDeniedException from the service API.
C is Incorrect: Throttling errors from exceeding invocation limits typically surface as ThrottlingException or HTTP 429 responses, not AccessDeniedException. The error type explicitly points to authorization, not rate limiting.
D is Incorrect: If the KMS key prevented the model from loading, the endpoint would fail at deployment or update time and would not be in service. An endpoint that is actively rejecting calls with access-denied errors indicates the endpoint itself is healthy.

Key Takeaway: Read the specific error code (AccessDeniedException vs. ThrottlingException vs. timeout) and cross-reference CloudTrail to isolate IAM authorization issues from network or quota problems before changing security controls.

Question 2 - CloudTrail Data Events for Model Invocation Audit

A healthcare organization must comply with HIPAA audit requirements that mandate capturing every model invocation request, including the identity of the caller and the timestamp, for all foundation model calls made through Amazon Bedrock in their production account.

The security team has already enabled a multi-region CloudTrail trail that records management events. During a recent audit, however, they discovered that no model invocation records were present in the trail.

Which configuration change is required to capture the missing records?

A) Enable CloudTrail data events for Amazon Bedrock and select the specific Bedrock model resources or all Bedrock model invocations in the account.

B) Add an Amazon EventBridge rule that listens for Bedrock InvokeModel events and writes them to the existing CloudTrail trail manually.

C) Configure AWS Config with a custom rule that monitors Bedrock model invocations and records them to an S3 bucket in the CloudTrail trail's prefix.

D) Move the existing CloudTrail trail into the same region where the Bedrock models are deployed, since multi-region trails only capture management events.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

A is Correct: Amazon Bedrock InvokeModel and Converse API calls are classified as data events in CloudTrail. By default, trails record only management events. To capture Bedrock model invocations, you must explicitly configure CloudTrail data events for Amazon Bedrock and select either specific resources or all current and future Bedrock resources in the account.

Distractor Analysis

B is Incorrect: EventBridge rules do not write events into CloudTrail; CloudTrail is AWS's native API activity recording service. While you can use EventBridge for real-time event routing, it cannot populate CloudTrail trails. Additionally, manually forwarding events creates operational overhead and does not integrate with CloudTrail's native event format.
C is Incorrect: AWS Config records resource configurations and evaluates them against compliance rules, not runtime API call activity. It cannot be used to capture model invocation events, which are data-plane operations.
D is Incorrect: Multi-region CloudTrail trails already capture events from all AWS regions. The issue is not trail location but rather the event type (management vs. data events) selected in the trail configuration.

Key Takeaway: Model invocation calls (like Bedrock InvokeModel or SageMaker InvokeEndpoint) are data events, and require explicit data event configuration on the CloudTrail trail—management event logging alone will not capture them.

Question 3 - Troubleshooting Private Subnet Training Job Failure to Access S3

An ML engineering team runs a SageMaker training job in a private subnet with no internet access. The training job's IAM role has full s3:GetObject and s3:ListBucket permissions on the training data bucket. The job fails immediately with a connection timeout when loading data from Amazon S3.

The team has already confirmed that the IAM role permissions are correct and the bucket exists. Which TWO actions should the team take to resolve this issue? (Select TWO)

A) Add an S3 Gateway endpoint to the VPC and update the subnet's route table to route S3 traffic through the endpoint.

B) Modify the training job's security group to allow inbound traffic from the S3 service's public IP ranges.

C) Add a NAT Gateway to the private subnet so the training job can route outbound connections to the internet and reach S3.

D) Verify that the training job's security group has an outbound rule allowing traffic to port 443 and that the VPC endpoint policy allows access to the specific S3 bucket ARN.

E) Encrypt the S3 bucket with SSE-KMS and grant the training job's role permission to use the KMS key for decryption.

Click to reveal Answer & Explanation

Correct Answers: A and D

Correct Options Analysis

A is Correct: In a private subnet without internet access, the training job cannot reach the public S3 endpoint. Adding an S3 Gateway VPC endpoint creates a private network path to S3. The subnet's route table must include a route directing S3 traffic to the endpoint.
D is Correct: Security group outbound rules must allow HTTPS (port 443) traffic to the S3 service. Additionally, a VPC endpoint policy may be configured to restrict which S3 buckets can be accessed through the endpoint; if the policy does not explicitly allow access to the training data bucket ARN, the connection will fail.

Distractor Analysis

B is Incorrect: Security groups cannot be configured with public IP ranges of AWS services as inbound sources for outbound service access. Additionally, inbound rules control traffic entering the training instances, not the outbound connection to S3.
C is Incorrect: While a NAT Gateway would enable internet access, it introduces additional cost and operational overhead. An S3 Gateway endpoint is the more efficient and secure solution for accessing S3 from a private subnet, keeping traffic on the AWS network backbone.
E is Incorrect: If the data were encrypted with SSE-KMS and the role lacked KMS decryption permissions, the error would be an AccessDeniedException from S3, not a connection timeout. The timeout indicates a network connectivity problem, not a permissions issue.

Key Takeaway: A connection timeout (vs. access denied) from a private subnet points to missing network connectivity—resolve it with the appropriate VPC endpoint (Gateway for S3/DynamoDB, Interface for most other services) and verify both route tables and security group outbound rules.

Question 4 - IAM Policy Troubleshooting for Bedrock Production Workload

A production application on Amazon EKS uses Amazon Bedrock to invoke foundation models. The application previously ran with no issues, but after a security review, the IAM role attached to the application's Kubernetes service account was updated to enforce least privilege.

Now the application can list available models (bedrock:ListFoundationModels) but cannot invoke them. CloudTrail shows AccessDeniedException for bedrock:InvokeModel calls. The security team's policy includes the following statement:

{
  "Effect": "Allow",
  "Action": [
    "bedrock:ListFoundationModels",
    "bedrock:InvokeModel"
  ],
  "Resource": "*"
}

What is the likely reason for the continued denial?

A) The bedrock:InvokeModel action requires a separate Resource specification for each foundation model ARN, and wildcard resource specifications are not supported for Bedrock IAM policies.

B) The policy includes a Condition block elsewhere that restricts the bedrock:InvokeModel action, or a permissions boundary or service control policy (SCP) is blocking the action at the account or organizational level.

C) The bedrock:InvokeModel action is not possible with IAM roles; it requires a long-term API key generated from the Bedrock console.

D) The application's Kubernetes service account does not have permission to assume the IAM role, so the role's policy is not being applied to the application's requests.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: The provided IAM policy statement explicitly allows bedrock:InvokeModel with Resource: "*", which is valid for Bedrock. If the application can list models (proving the role is attached and functioning), the denial on invoke calls must come from another layer—either a Condition block in the same or another policy, a permissions boundary on the role, or an SCP at the organizational level that denies bedrock:InvokeModel. CloudTrail's error message and the recent policy changes point to a restrictive element introduced during the security review.

Distractor Analysis

A is Incorrect: IAM policies for Bedrock support wildcard resources. You can specify "Resource": "*" or scope down to specific model ARNs. The wildcard specification itself is not the problem.
C is Incorrect: IAM roles with temporary credentials are the recommended production credential type for Bedrock. Long-term API keys are documented for development and exploration, not production workloads. The role-based approach is correct; the denial stems from another restriction.
D is Incorrect: The fact that the application can successfully perform bedrock:ListFoundationModels proves the Kubernetes service account can assume the IAM role and the role's permissions are in effect. The denial is specific to the InvokeModel action, pointing to a condition or boundary restriction.

Key Takeaway: When debugging IAM denials, evaluate the full policy document including Condition blocks, permissions boundaries on the role, and SCPs at the organizational level—wildcard resources and action allowances can still be overridden by other permission layers.

Question 5 - Bedrock Guardrails Sensitive Information Filter Configuration

A customer support chatbot built on Amazon Bedrock uses a retrieval-augmented generation (RAG) architecture. During testing, the chatbot occasionally responds with customer email addresses and phone numbers retrieved from the knowledge base, violating the company's data protection policy.

The ML engineering team needs to prevent these sensitive values from ever appearing in model responses while maintaining response quality. Which configuration addresses this requirement with minimal operational overhead?

A) Implement a post-processing AWS Lambda function that scans all model responses for PII patterns and redacts them before sending to users.

B) Configure a sensitive information filter in Amazon Bedrock Guardrails that detects and masks email addresses and phone numbers in model output.

C) Add a denied topic in Amazon Bedrock Guardrails that blocks all responses containing contact information.

D) Manually review and redact all customer data from the knowledge base documents before uploading them to the S3 data source.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: Amazon Bedrock Guardrails provides native sensitive information filters that detect and mask PII (including email addresses, phone numbers, and custom regex patterns) in both model inputs and outputs. This requires no custom code, operates at low latency, and applies the filter consistently before responses reach end users.

Distractor Analysis

A is Incorrect: Building a custom Lambda post-processing layer introduces significant operational overhead: maintaining PII detection logic, handling edge cases, and managing additional compute. Bedrock Guardrails provides this natively without custom infrastructure.
C is Incorrect: Denied topics block entire topics of conversation, not specific sensitive information patterns. Using a denied topic for contact information would likely block legitimate responses that happen to mention customer information in a non-sensitive context (like "please contact support"), degrading response quality.
D is Incorrect: Manually redacting all customer data from source documents is operationally unscalable, especially with a growing knowledge base. It also fails to address cases where the model generates synthetic PII-like patterns. The guardrail operates at inference time, catching both retrieved and generated sensitive values.

Key Takeaway: Sensitive information filters in Bedrock Guardrails are purpose-built for PII detection and masking at inference time—use them for data protection requirements rather than building custom redaction logic or using denied topics, which are meant for content moderation.

Question 6 - Troubleshooting Model Endpoint Security Group Configuration

A company has deployed a SageMaker real-time inference endpoint for a recommendation model. The endpoint is configured with a security group that has the following rules:

Inbound: Allow TCP 443 from the application load balancer's security group only.
Outbound: No rules specified (all outbound traffic denied).

The application successfully invokes the endpoint, but the endpoint's model container cannot retrieve updated recommendation data from Amazon DynamoDB during inference, causing stale results. CloudWatch logs show the container timing out when connecting to DynamoDB.

What is the root cause and appropriate fix?

A) The inbound rule is too restrictive; DynamoDB needs to be explicitly allowed as an inbound source in the security group.

B) The security group's outbound rules must be updated to allow traffic to the DynamoDB endpoint (HTTPS port 443) or a VPC Gateway endpoint for DynamoDB must be added to the subnet's route table.

C) The IAM role attached to the endpoint must be granted dynamodb:GetItem and dynamodb:Query permissions to access the DynamoDB table.

D) The endpoint must be redeployed with a new security group that allows all outbound traffic to the internet.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: When a security group is created, the default behavior is to allow all outbound traffic unless explicitly restricted. However, if outbound rules have been modified to deny traffic (or if the default outbound rule was removed), the endpoint cannot reach DynamoDB. The fix is either adding an outbound HTTPS rule for the DynamoDB service or using a VPC Gateway endpoint for DynamoDB, which routes traffic through the private AWS network and does not require outbound internet access.

Distractor Analysis

A is Incorrect: Inbound rules control traffic entering the endpoint's network interface from external sources. DynamoDB does not initiate inbound connections to the endpoint; the endpoint initiates outbound connections to DynamoDB. The inbound rule is correctly scoped and not the cause.
C is Incorrect: If the IAM role lacked DynamoDB permissions, the container would receive an AccessDeniedException from the DynamoDB API, not a connection timeout. Timeouts indicate network connectivity issues.
D is Incorrect: Opening all outbound traffic to the internet is a security anti-pattern that violates least-privilege network access. The appropriate fix is a scoped outbound rule or VPC endpoint, not unrestricted internet access.

Key Takeaway: When a model endpoint cannot reach AWS services (timeout, not access denied), inspect security group outbound rules and VPC route tables—and prefer VPC endpoints for private connectivity without internet exposure.

Question 7 - CodeGuru Security for Hardcoded Credentials in ML Pipeline

A DevOps team manages an ML pipeline using AWS CodePipeline that builds and deploys a SageMaker training job. During a recent security review, the team discovers that the source code repository contains an API key for an external data provider hardcoded in a Python script used by the training job.

The team must implement a prevention mechanism that automatically detects hardcoded credentials and secrets in the source code before deployment, with minimal changes to the existing pipeline.

Which AWS service should the team add to the pipeline?

A) Amazon GuardDuty to scan the source code repository for exposed credentials.

B) Amazon Inspector to scan the container images built by the pipeline for the hardcoded API key.

C) Amazon CodeGuru Security to perform static analysis on the source code and detect hardcoded credentials and other security vulnerabilities.

D) AWS Secrets Manager to automatically rotate the hardcoded API key in the source code repository.

Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

C is Correct: Amazon CodeGuru Security performs static code analysis that detects security policy violations, including hardcoded credentials in application code. It integrates directly with CodePipeline as an action or can be invoked through the API. Adding CodeGuru Security to the pipeline provides automated credential detection at the source code level, which is exactly where the hardcoded API key exists.

Distractor Analysis

A is Incorrect: Amazon GuardDuty monitors AWS account activity and network traffic for threats like unusual API call patterns, compromised instances, or data exfiltration. It does not scan source code for hardcoded credentials.
B is Incorrect: Amazon Inspector scans container images and EC2 instances for known software vulnerabilities (CVEs) and network exposure. While it might flag a container image containing a secret in some cases, it is designed for package-level vulnerability scanning, not source code static analysis. The API key in a Python script is best detected at the source code level before image building.
D is Incorrect: AWS Secrets Manager stores and rotates secrets, but it does not detect hardcoded credentials in source code. The team would need to first identify the credential, then migrate it to Secrets Manager—CodeGuru Security is the detection mechanism.

Key Takeaway: Use CodeGuru Security for static code analysis (hardcoded credentials, security patterns) and Amazon Inspector for container image vulnerability scanning (CVEs)—both are needed in a pipeline, but each answers a different question.

Question 8 - Debugging SageMaker Endpoint KMS Encryption Failure

A data science team is deploying a new SageMaker endpoint with a model artifact stored in Amazon S3. The model artifact is encrypted with a customer-managed KMS key. The team's SageMaker execution role has s3:GetObject permission on the model artifact bucket, but the endpoint creation fails with an AccessDeniedException.

CloudTrail shows the sagemaker:CreateEndpoint call failed during the model download phase. Which of the following is the likely root cause?

A) The VPC security group attached to the endpoint does not allow inbound traffic from the S3 service, preventing model download.

B) The SageMaker execution role is missing the KMS decryption permission (kms:Decrypt) for the customer-managed key used to encrypt the model artifact.

C) The S3 bucket policy explicitly denies access to the SageMaker service principal, blocking model artifact downloads.

D) The SageMaker endpoint quota has been reached, causing the endpoint creation to fail with an access denied error.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

B is Correct: When an S3 object is encrypted with a customer-managed KMS key, the principal attempting to read the object must have both s3:GetObject on the bucket/object AND kms:Decrypt on the KMS key. The SageMaker execution role has the S3 permission but lacks the KMS decryption permission, which causes an AccessDeniedException during the model download phase.

Distractor Analysis

A is Incorrect: If the endpoint had VPC network connectivity issues reaching S3, the failure would manifest as a timeout or network error, not an AccessDeniedException. The access denied error indicates an authorization failure, not a network problem.
C is Incorrect: While an S3 bucket policy denying the SageMaker service principal could cause failure, the scenario states the role has s3:GetObject permission. The most likely and common gap in KMS-encrypted artifacts is the missing kms:Decrypt permission—a frequent troubleshooting issue in SageMaker deployments.
D is Incorrect: Quota exhaustion produces ResourceLimitExceeded or throttling errors, not AccessDeniedException. The error type clearly indicates an IAM or KMS authorization failure.

Key Takeaway: When model artifacts are encrypted with a customer-managed KMS key, the execution role needs both S3 read permissions AND kms:Decrypt on the KMS key—forgetting the KMS permission is a common cause of endpoint creation failures.

Question 9 - Bedrock Guardrails Contextual Grounding vs. Automated Reasoning

A legal document assistant uses Amazon Bedrock with a RAG architecture. The assistant must satisfy two requirements:

Responses must be grounded in the retrieved legal passages and not hallucinate.
The assistant must only provide answers that are consistent with the company's formal compliance rules, which are expressed as structured logical statements.

Which combination of Bedrock Guardrails configurations addresses both requirements? (Select TWO)

A) Configure a contextual grounding check that validates whether responses are supported by the reference passages retrieved for the query.

B) Configure a denied topics filter for legal subjects that the model should not discuss.

C) Configure an Automated Reasoning check that validates responses against the formal compliance rules expressed as structured logical statements.

D) Configure a sensitive information filter to block personal data (e.g., names, addresses) from appearing in legal document responses.

E) Configure a word filter with a list of legal terms that must be blocked from all responses.

Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

A is Correct: Contextual grounding checks in Bedrock Guardrails evaluate whether model responses are factually supported by the reference passages provided in the prompt context. This directly addresses the hallucination requirement by flagging or blocking responses that are not grounded in the retrieved passages.
C is Correct: Automated Reasoning checks validate whether a response is logically consistent with formal rules expressed as structured logical statements. This matches the second requirement about compliance rules, ensuring the assistant's conclusions follow from the stated policies.

Distractor Analysis

B is Incorrect: Denied topics are used to block entire categories of conversation (e.g., "medical advice," "investment recommendations"). They do not address hallucination or logical consistency with formal rules.
D is Incorrect: Sensitive information filters detect and mask PII, which is unrelated to either grounding or logical consistency with compliance rules. The requirements are about factual accuracy and policy alignment, not data protection.
E is Incorrect: Word filters block specific terms and profanity, not semantic or logical consistency. They cannot evaluate whether a response is grounded or consistent with formal rules.

Key Takeaway: Contextual grounding checks address hallucination (is the response supported by source passages?), while Automated Reasoning checks address policy consistency (is the response logical given formal rules?)—recognize the difference for RAG and policy-bound applications.

Question 10 - Troubleshooting Endpoint Exposure Through Overly Permissive Security Group

A security audit finds that a SageMaker endpoint hosting a customer churn prediction model is accessible from the public internet. The endpoint was supposed to be private and accessible only from the internal application tier.

The audit report shows the endpoint's security group has the following inbound rule:

Inbound: Allow TCP 443 from source 0.0.0.0/0

The ML engineering team must restrict access to only the internal application tier, which runs in a different subnet within the same VPC. What is the appropriate remediation?

A) Replace the source 0.0.0.0/0 with the security group ID of the application tier, so only instances assigned that security group can invoke the endpoint.

B) Move the endpoint to a different VPC and create a peering connection to the application tier.

C) Disable all inbound rules and configure the application tier to use an AWS PrivateLink interface endpoint to reach the SageMaker endpoint.

D) Add a deny rule for source 0.0.0.0/0 and allow the application tier's IP address range specifically.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

A is Correct: The security group rule source 0.0.0.0/0 allows traffic from any IP address (the public internet). Replacing the source with the security group ID of the application tier restricts access to only those EC2 instances or containers that share that security group. This is the standard AWS best practice for restricting access within a VPC, using security group chaining.

Distractor Analysis

B is Incorrect: Moving the endpoint to a different VPC adds significant operational complexity and requires a VPC peering connection between the application tier and the endpoint's new VPC. The same restriction can be achieved with a simple security group rule change.
C is Incorrect: This solution is incorrect on multiple levels. Security groups do not support explicit deny rules; they are allow-lists only. Additionally, AWS PrivateLink interface endpoints are used to privately reach AWS services from within a VPC, not to control access to your own SageMaker endpoint. SageMaker endpoints are accessed via SageMaker's service endpoint, and network access is controlled by the endpoint's security group.
D is Incorrect: Security groups support only allow rules, not explicit deny rules. You cannot add a deny rule for 0.0.0.0/0 to override an existing allow rule. The fix is to modify the existing allow rule's source.

Key Takeaway: Use security group chaining (referencing other security group IDs as the source) to restrict inbound traffic between application tiers and model endpoints—this avoids IP- range management and follows least-privilege network access.