Skill 4.3.4: Configure comprehensive monitoring, auditing, compliance, and logging for ML and AI systems (for example, AWS CloudTrail, AWS Config).
Task 4.3: Operating, Monitoring, and Securing ML and AI Solutions - Secure ML and AI workloads and model endpoints.
Exam Gotcha: CloudTrail logs management events by default. Model invocations and S3 object reads are data events and must be explicitly enabled. Bedrock model invocation logging is separate and can capture full prompt/response bodies.
Core Mental Model
- Activity record → CloudTrail: who called which API, when, from where.
- Configuration record → AWS Config: what the resource looked like over time, and whether it complies.
- Invocation record → Bedrock model invocation logging / SageMaker Data Capture: what went in and what came back.
- Runtime record → CloudWatch Logs / metrics: application, container, endpoint behavior.
- Network record → VPC Flow Logs: IP/port/protocol traffic.
- Threat/vulnerability record → GuardDuty, Inspector, Security Hub, Access Analyzer.
- Compliance evidence → Audit Manager, Artifact, Config conformance packs.
Service Selection Matrix
| Audit/Compliance Question | Primary Service | What It Captures | ML/AI Relevance | Key Config |
|---|---|---|---|---|
| Who invoked a model or changed an endpoint? | CloudTrail | API activity metadata | SageMaker CreateEndpoint, InvokeEndpoint; Bedrock InvokeModel, Converse | Management default; data events opt-in; multi-region/org trail |
| What was the resource configuration? | AWS Config | Resource config history + rules | Endpoint encryption, public access, IAM roles, S3/KMS policies | Recorder, rules, conformance packs, aggregator, remediation |
| What prompt/response crossed the model? | Bedrock model invocation logging | Request/response bodies + metadata | Audit, debugging, compliance | CloudWatch Logs/S3, KMS, retention, masking |
| What inference payload hit an endpoint? | SageMaker Endpoint Data Capture | Request/response payloads | Monitoring, drift, audit | S3, KMS, bucket policy, lifecycle |
| What happened in app/container? | CloudWatch Logs | Application/runtime logs | Training, inference, custom containers | Log groups, retention, KMS, metric filters, alarms |
| What network traffic occurred? | VPC Flow Logs | IP/port/protocol | Private subnet/endpoint verification | Enable per VPC/subnet/ENI; CloudWatch Logs/S3 |
| What threats exist? | GuardDuty | Anomalous API/network behavior | Compromised credentials, crypto mining | Org-wide, findings to Security Hub |
| What external access exists? | IAM Access Analyzer | Resource policy findings | S3, KMS, IAM roles, Bedrock | Analyze external access; remediate |
| What sensitive data is in S3? | Amazon Macie | PII discovery in S3 | Training data, logs | Classification, findings to Security Hub |
| What compliance evidence? | Audit Manager / Artifact | Automated evidence, reports | ML controls, frameworks | Frameworks, assessments, reports |
CloudTrail: Audit Activity
- Default coverage: management events (control plane) only; Event history = 90 days.
- Data events: opt-in, high volume, extra cost. Enable for S3 object-level, Lambda invoke, Bedrock runtime, SageMaker endpoint invocations as needed.
- Trail design: multi-region, organization trail, log file validation, KMS CMK, S3 bucket policy, CloudWatch Logs integration.
- Insights: anomalous management API call rates. Does not analyze data events.
- CloudTrail Lake: immutable event data store for SQL queries and long-term audit retention.
- Gotcha: CloudTrail data events record API metadata, not full prompt/response bodies. Bedrock model invocation logging captures content.
- Gotcha: A trail without data events will not show model invocation calls.
AWS Config: Configuration Compliance
- Purpose: record resource config changes over time; evaluate against rules; detect drift.
- ML/AI use: SageMaker endpoint KMS/encryption, public access, instance types, IAM roles; S3 bucket policies; KMS key policies; VPC settings.
- Key features: configuration recorder, delivery channel, managed/custom rules, conformance packs, aggregators, remediation via SSM Automation.
- Gotcha: Config answers "what did the resource look like?" not "who called the API?" It is not a real-time API audit log.
- Gotcha: Config rules evaluate configuration, not runtime model behavior or prompt content.
Model & Endpoint Logging
| Source | Captures | Destination | Security/Compliance Notes |
|---|---|---|---|
| Bedrock model invocation logging | Full request/response, metadata | CloudWatch Logs, S3 | Encrypt with KMS; mask sensitive data; set retention |
| SageMaker Endpoint Data Capture | Request/response payloads | S3 | KMS, bucket policy, lifecycle; use for monitoring/audit |
| SageMaker Model Monitor | Drift, bias, quality metrics | CloudWatch, S3 | Operational monitoring, not security audit |
| CloudWatch Logs | App/container logs | Log groups | Retention, KMS, metric filters, alarms, Logs Insights |
| VPC Flow Logs | IP/port/protocol | CloudWatch Logs, S3 | Network isolation verification; no payload |
Hard rule: Logs containing prompts, responses, or PII inherit the same classification as source data. Encrypt with KMS, restrict IAM, set retention, and mask sensitive values.
Security & Compliance Controls
- IAM: least privilege roles for training, endpoints, Bedrock. Separate human vs workload identity. Use permission boundaries, SCPs, resource policies, VPC endpoint policies.
- KMS: CMK for S3, EBS, EFS, SageMaker, Bedrock logs, CloudTrail, CloudWatch Logs. Key policies + grants. Encryption at rest; TLS in transit.
- PrivateLink/VPC endpoints: private access to S3, CloudWatch, CloudTrail, Config, KMS, Bedrock, SageMaker. Endpoint policies restrict.
- Guardrails: content filters, denied topics, word filters, sensitive info filters, contextual grounding, Automated Reasoning. Applies to prompts/responses. Not access control.
- Secrets Manager: rotate API keys/secrets. Avoid hardcoded credentials. Use short-term credentials in production.
- Compliance: Audit Manager, Artifact, Security Hub, GuardDuty, Access Analyzer. Central log archive account, S3 Object Lock, cross-account access.
Decision Traps & Anti-Patterns
- CloudTrail default only management events → no model invocation audit. Enable data events or model invocation logging.
- Config for API audit → wrong; Config is config history.
- CloudWatch Logs for API audit → wrong; CloudWatch is app/runtime logs.
- VPC Flow Logs for content → wrong; no payload.
- KMS for access control → wrong; KMS encrypts, IAM authorizes.
- PrivateLink for content filtering → wrong; controls path, not content.
- Guardrails for access control → wrong; filters content, not who can call.
- Long-term API keys in production → use IAM roles/temporary credentials.
- Logging prompts/responses without protection → same sensitivity as data; encrypt, mask, retain.
- No multi-region/org trail → gaps in audit.
- No log file validation/Object Lock → tamper risk.
- Over-broad IAM to fix AccessDenied → widen symptom, not cause.
High-Yield Config Checklist
- CloudTrail: multi-region org trail, management + selected data events, Insights, KMS, validation, S3 + CloudWatch Logs.
- Config: all resources, rules for encryption/public access/IAM, conformance packs, aggregator, remediation.
- CloudWatch: log groups with retention/KMS, metric filters for AccessDenied/InvokeModel errors, alarms, Logs Insights.
- Bedrock: model invocation logging, Guardrails, sensitive info filters, KMS, VPC endpoints.
- SageMaker: endpoint data capture, Model Monitor, CloudWatch metrics, VPC config, KMS.
- Security: GuardDuty, Security Hub, Access Analyzer, Inspector, Secrets Manager, Audit Manager.
- Network: private subnets, security groups least privilege, VPC endpoints, endpoint policies, Flow Logs.