Skip to content

Skill 4.3.5: Troubleshoot and debug security issues in ML and AI systems

Task 4.3: Operating, Monitoring, and Securing ML and AI Solutions - Secure ML and AI workloads and model endpoints

Debug rule: ML/AI security failure = missing control in identity/access, network path, encryption/key, or audit/safeguard. Read the exact denial/finding; never widen permissions until the error disappears.

Security Triage Flow

  1. Read exact error: AccessDenied vs timeout vs KMS vs Guardrail intervention.
  2. Identify: principal, action, resource, Region, network path.
  3. Evidence: CloudTrail, IAM Access Analyzer, VPC Flow Logs/Reachability Analyzer, KMS key policy, Guardrail logs.
  4. Check all policy layers: IAM, SCP, permissions boundary, resource policy, VPC endpoint policy, KMS key policy, session policy.
  5. Fix least privilege: exact action/resource; add endpoint/key policy/guardrail; enable data events.
  6. Verify: re-test and confirm audit record.

Security Triage Matrix

Symptom First Evidence Likely Root Cause Correct Fix Direction
AccessDenied on model invoke/training/endpoint CloudTrail error, IAM Policy Simulator, Access Analyzer Missing IAM action/resource; SCP/boundary; resource policy; KMS key policy; VPC endpoint policy Grant exact action/resource; check every policy layer
Timeout/connection refused from private subnet VPC Flow Logs, Reachability Analyzer No NAT/IGW and no VPC endpoint; SG/NACL egress; DNS Add Gateway/Interface endpoint; allow 443 egress; enable DNS
KMS decrypt/encrypt denied CloudTrail KMS events, key policy Role missing kms:Decrypt/GenerateDataKey; wrong key/Region; disabled key Update key policy + IAM; enable key; same Region
CloudTrail missing InvokeModel/InvokeEndpoint Trail config Data events not enabled Enable data events; use model invocation logging/data capture for payload
Guardrail not blocking/masking Guardrail trace/logs, invocation logs Guardrail not attached, wrong version, action NONE, input/output mismatch Attach correct guardrail/version; set BLOCK/MASK
Unexpected principal/activity CloudTrail, GuardDuty, IAM Access Analyzer Over-permissive role, leaked long-term key, broad trust Scope down, rotate, use temporary credentials
ECR pull failure in training/endpoint CloudTrail, ECR policy, VPC logs Missing ECR/STS/S3 endpoint or IAM Add endpoints and repo policy

Four Control Planes

Control Plane Question Evidence Common ML/AI Failure
Identity/access Who can call? CloudTrail, IAM Access Analyzer, policy simulator Missing PassRole, KMS, S3, ECR, Bedrock actions; SCP/boundary
Network Can traffic reach? VPC Flow Logs, Reachability Analyzer, SG/NACL, endpoint policies Private subnet no endpoint/NAT; SG egress; endpoint policy deny
Encryption/key Can data be read/written? CloudTrail KMS events, key policy/grants Key policy missing service role; cross-Region; disabled key
Audit/safeguard What happened? Was content allowed? CloudTrail management vs data events, Config, Guardrails logs Data events off; guardrail not attached; logs unprotected

Access Denied Debugging Checklist

  • Principal: Human or workload? Check role trust, session policy, permissions boundary, SCP.
  • Action: Exact IAM action: bedrock:InvokeModel, sagemaker:InvokeEndpoint, s3:GetObject, ecr:BatchGetImage, kms:Decrypt, iam:PassRole.
  • Resource: ARN/Region/model/endpoint/bucket/key. Cross-account needs resource policy + key policy + endpoint policy.
  • KMS: IAM allow alone is insufficient; key policy must allow the role/service. Check encryption context.
  • VPC endpoint policy: Can deny even if IAM allows.
  • Bedrock: Model access must be enabled in account/Region. Wrong Region = AccessDenied/ValidationException.
  • SageMaker: Execution role needs ECR, S3, CloudWatch, KMS; user needs iam:PassRole.
  • Secrets Manager: Secret resource policy, KMS, rotation role.

Network Isolation & Private Access

Need Correct Mechanism Gotcha
S3/DynamoDB from VPC Gateway VPC endpoint Route table association; endpoint policy
Bedrock, SageMaker, ECR, KMS, STS, Secrets Manager, CloudWatch Logs Interface VPC endpoint (PrivateLink) Creates ENI; SG must allow 443; enable private DNS
On-prem/hybrid to AWS services VPN/Direct Connect + PrivateLink PrivateLink secures path, not content
Public model endpoint Avoid; use VPC-only endpoint, IAM auth, SG least privilege SG 0.0.0.0/0 is a finding
Cross-VPC VPC peering/TGW/PrivateLink NACL/SG/route tables

Gotcha: Removing internet access from a subnet also removes the path to AWS APIs. Add VPC endpoints or a managed NAT path, or the workload fails with timeouts—not AccessDenied.

Auditing & Monitoring Debugging

Record Answers Captures Enable/Check
CloudTrail management events Who changed control plane? CreateModel, CreateEndpoint, PutBucketPolicy On by default for trails
CloudTrail data events Who invoked model/read object? InvokeModel, InvokeEndpoint, S3 GetObject/PutObject Must be configured; high volume
AWS Config What did resource look like? Config history, rules Drift/compliance
GuardDuty Is behavior anomalous? Findings Threat detection
Model invocation logging / data capture What prompt/response? Payloads Separate; protect with KMS/S3
Guardrails logs Why blocked/masked? Policy match Attach guardrail/version

Gotcha: CloudTrail does not record prompt/response payloads by default. Model invocation logging is separate. A trail without data events will not show InvokeModel/InvokeEndpoint calls.

Guardrails / Safeguard Troubleshooting

Policy Purpose Use When Debug Check
Content filters Harmful categories General safety Threshold/action; input/output
Denied topics Refuse subject areas Policy-bound app Topic definition; action
Word filters Block specific terms Profanity/custom Case/regex; custom words
Sensitive info filters PII/custom regex Account numbers, PII Mask vs block; input/output; logs
Contextual grounding Response supported by source RAG hallucination Grounding threshold; reference text
Automated Reasoning Statement consistent with formal rules Policy compliance Rules expressed formally; output

Gotcha: Guardrails filter content; they do not replace IAM/VPC. Masking PII in a response does not protect raw prompts/responses stored in logs.

CI/CD & Vulnerability Debugging

Tool Finds Does Not Find Gate?
CodeGuru Security Code vulnerabilities, hardcoded credentials Vulnerable OS/packages in image Fail build on high severity
Inspector EC2, ECR images, Lambda CVEs, network exposure Logic flaws, hardcoded secrets in source Block deploy on critical
ECR image scanning Container package CVEs Runtime prompt attacks Scan on push
Secrets Manager Secret storage/rotation Code vulnerabilities Use instead of hardcoded secrets

Gotcha: A pipeline needs both static code analysis and image scanning. They answer different questions.

Credential Selection

Scenario Credential Why
Production app invoking FM/model IAM role with temporary credentials Rotated, auditable, least privilege
Exploration/dev Short-term API key (inherits identity, expires) Convenient, not production
Long-term API key Dev/exploration only AWS documents not for production
Human admin IAM Identity Center/permission set No shared keys
Static secret required Secrets Manager with rotation Avoid code/image/env files

Anti-Patterns / Fast Wrong Fixes

  • Widening IAM to clear AccessDenied → masks root cause; grant exact action/resource.
  • One shared role for all ML workloads → no blast radius; separate training, endpoint, pipeline roles.
  • Public inference endpoint or SG 0.0.0.0/0 → expose model; use VPC-only/PrivateLink.
  • Assuming CloudTrail default logs model calls → data events required.
  • CMK encryption without key policy for ML roles → AccessDenied at runtime.
  • No VPC endpoint after removing internet → timeouts.
  • Guardrails as access control → wrong control; use IAM/VPC for access.
  • Ignoring Inspector/CodeGuru findings → vulnerable image/code reaches prod.
  • Long-term Bedrock API keys in prod → rotate/leak risk; use roles.

Exam Gotchas: - IAM allow is necessary but not sufficient: check SCP, permissions boundary, resource policy, VPC endpoint policy, KMS key policy, session policy. - Model invocation calls are data-plane events. Enable CloudTrail data events and/or model invocation logging for audit. - Private subnet + no NAT/IGW = VPC endpoints for every AWS service called. Missing endpoint looks like timeout, not AccessDenied. - Production FM access = IAM role temporary credentials. Long-term API keys are for exploration/development only. - Guardrails filter content; IAM/VPC control who can call. Use contextual grounding for RAG, Automated Reasoning for formal rules.