Domain 4: Operating, Monitoring, and Securing ML and AI Solutions
Task 4.1 — Monitor ML and AI model inference and performance
| Trigger Keywords & Scenario Constraints | Lock-In Architecture | Immediate Rejection / Trap to Avoid | Explanation |
|---|---|---|---|
| "Data drift / feature distribution change" + "production endpoint / batch transform" + no labels | SageMaker Model Monitor — Data Quality (Data Capture → baseline statistics.json/constraints.json → schedule) |
Model Quality monitor (needs ground truth) or CloudTrail | Compares live feature stats vs training baseline; labels are not required. |
| "Model accuracy / F1 / RMSE degraded in production" + "ground truth available (delayed)" | SageMaker Model Monitor — Model Quality (join captured predictions with ground-truth in S3) | Data Quality monitor (no accuracy metrics) | Model-quality jobs need merged labels; data-quality jobs do not. |
| "Bias increased after deploy" + "sensitive attribute / fairness metric (DPPL)" | SageMaker Clarify + Model Monitor — Bias Drift | Data Quality only, or Amazon Macie | Bias drift tracks fairness metrics on live predictions against an allowed range + confidence interval. |
| "Feature importance ranking changed" + "SHAP / NDCG < 0.90" | SageMaker Clarify — Feature Attribution Drift | Data Quality PSI or Model Quality F1 | Attribution drift uses SHAP ranking comparison (NDCG), not raw feature histograms. |
| "Capture request/response payloads" + "S3 for later monitoring" | SageMaker Data Capture (DataCaptureConfig, sampling %) |
CloudTrail data events or VPC Flow Logs | Data Capture is the prerequisite for Model Monitor / Clarify on endpoints and batch transform. |
| "Need a baseline before drift alerts" | Model Monitor baselining job on training dataset (Deequ) | Alerting on raw CloudWatch Invocations |
Baseline suggests constraints; violations vs baseline fire CloudWatch metrics. |
| "Scheduled drift job" + "notify on violation" | Model Monitor schedule + CloudWatch Alarm + Amazon SNS | Polling Studio manually or GuardDuty | Monitor jobs emit CloudWatch metrics; SNS (not GuardDuty) is the exam notification path. |
| "Single pane of all production models / failed monitors" | SageMaker Model Dashboard | CloudTrail Event history | Model Dashboard aggregates monitor statuses, endpoints, and risk across models. |
| "A/B test two models on live users" + "both return responses" | SageMaker production variants + traffic weights (UpdateEndpointWeightsAndCapacities) |
Shadow variant (client never sees shadow output) | A/B splits real traffic; each request is served by exactly one variant that responds. |
| "Invoke a specific model version for a request" | TargetVariant on InvokeEndpoint |
Changing instance count or creating a second endpoint | Direct targeting bypasses weighted routing for controlled experiments. |
| "Validate new model/container/instance with zero user impact" + "compare latency/error rate" | SageMaker Shadow Testing (shadow variant; copy of traffic; response discarded) | Production-variant A/B (users receive new model output) | Shadow copies requests; only production variant answers the caller. |
| "Promote candidate after shadow test" | Promote shadow variant to production on same endpoint | Blue/green via a brand-new account | Console/API promotion swaps shadow → production without a parallel product stack. |
| "Canary: send 5–10% traffic to new model, rest to old" | Production variants with small DesiredWeight on new variant | Shadow test (no user-visible canary) or Serverless Inference | Weighted variants are the canary/A-B mechanism; serverless does not support multiple variants. |
| "FM quality / hallucination / helpfulness / toxicity" + "Bedrock" | Amazon Bedrock Model Evaluation (automatic metrics, LLM-as-a-judge, human eval) | SageMaker Model Monitor Data Quality | FM eval is qualitative/task-based; tabular data-quality monitors do not score generations. |
| "Human-in-the-loop scoring of generations" | Bedrock evaluations — human workforce / HITL workflow | CloudWatch Invocations alarm |
Human eval is for subjective quality; CloudWatch only has operational metrics. |
| "RAG answer not grounded / irrelevant to retrieved chunks" | Bedrock Guardrails Contextual Grounding Checks + retrieval-accuracy eval | Data Quality monitor on the vector index | Grounding checks score faithfulness/relevance of the generation vs source, not feature drift. |
| "Token usage, P90/P99 latency, throttles, invocation logs for Bedrock/agents" | Amazon CloudWatch generative AI observability (model invocations dashboard) | AWS Config or Cost Explorer | GenAI observability is the FM ops dashboard: tokens, latency, errors, traces. |
| "End-to-end prompt tracing" + "KB / tools / guardrails / model" | CloudWatch GenAI observability + prompt traces (Bedrock traces; ADOT for 3rd-party) | X-Ray only on EC2 CPU | Prompt traces locate which component (KB, tool, model) failed; X-Ray alone has no prompt payload view. |
| "Agent coordination failure / truncated streaming / tool failure (401/403/400/500)" | Amazon Bedrock AgentCore Observability (metrics + OTEL traces + structured logs) | SageMaker Debugger or Model Monitor | AgentCore Observability is purpose-built for agent steps, tools, memory, and identity errors. |
| "Where did the agent choose the wrong tool?" | AgentCore traces (OpenTelemetry spans) + AgentCore Evaluations (tool-selection accuracy) | CloudTrail InvokeModel records |
CloudTrail shows the API was called; traces show the reasoning/tool sequence. |
| "Agent behavioral drift / goal not attained" | Bedrock AgentCore Evaluations + CloudWatch alarm on eval scores | Model Monitor Data Quality | Agent evals score correctness/helpfulness/safety, not tabular feature stats. |
| "Detect anomalies in inference errors without setting static thresholds" | CloudWatch Anomaly Detection on Invocation5XXErrors / ModelLatency |
AWS Config rules | Anomaly Detection learns a band; Config watches resource configuration, not latency. |
| "ML-powered operational insights across endpoint + pipeline" | Amazon DevOps Guru | Amazon Inspector | DevOps Guru correlates ops anomalies; Inspector scans vulnerabilities. |
| "Distributed trace across API Gateway → Lambda → SageMaker/Bedrock" | AWS X-Ray (+ CloudWatch Service Lens / Application Signals) | CloudTrail or VPC Flow Logs | X-Ray traces request paths and latency segments; CloudTrail is an audit log. |
| "Endpoint invocation count, ModelLatency, OverheadLatency, 4XX/5XX" | CloudWatch AWS/SageMaker invocation metrics |
Custom Model Monitor constraints | These are infrastructure/inference SLIs, not data-drift statistics. |
| "GPU/CPU/memory/disk of the model container" | CloudWatch CPUUtilization / GPUUtilization / MemoryUtilization / DiskUtilization |
Cost Explorer | Resource utilization is CloudWatch; Cost Explorer is spend. |
| Serverless endpoint cold start duration | CloudWatch OverheadLatency |
ModelLatency |
ModelLatency is inside the container; OverheadLatency includes provision time. |
| "4XX vs 5XX on endpoint" | 4XX = client/bad payload; 5XX = container/model/server failure | Treating 4XX as model drift | Drift monitors do not replace HTTP-class error triage. |
| "Workflow / pipeline step failed" + "kick off alarm or retrain" | SageMaker Pipelines failure notifications + EventBridge + SNS/Lambda | CloudTrail Insights | Pipeline/model events are operational workflow monitors, not API-audit events. |
| "FM invocation input/output logging for later eval" | Bedrock model invocation logging → CloudWatch Logs and/or S3 | SageMaker Data Capture on a Bedrock API | Data Capture is SageMaker endpoints; Bedrock uses invocation logging. |
| "Dashboards for execs / business quality KPIs on predictions" | Amazon QuickSight (on Athena/S3 monitor outputs) | CloudWatch metric math only | QuickSight is BI; CloudWatch dashboards are ops SLIs. |
| "Query raw inference logs ad hoc" | CloudWatch Logs Insights | AWS Config aggregator | Logs Insights queries log text; Config queries resource state. |
| "Need sampling to cut monitor cost" | Data Capture sampling percentage < 100% | Turning off KMS | Sampling reduces Processing-job and S3 cost; it is not a security control. |
| "Retrain automatically when drift exceeds threshold" | CloudWatch Alarm → SNS/EventBridge → Lambda → SageMaker Pipeline | Manual Studio stop/start | Exam lock-in for closed-loop monitoring is alarm-driven retrain, not a new endpoint type. |
| "Compare shadow vs production operational metrics live" | Shadow test dashboard (Invocations, ModelLatency, Invocation5XXErrors, CPU/mem) |
A/B accuracy notebook only | Shadow dashboard is operational (latency/errors); A/B can also score labels. |
| "No GPU, no Data Capture, still need health of serverless endpoint" | CloudWatch Invocations, ModelLatency, OverheadLatency on serverless |
Model Monitor schedule | Serverless Inference does not support Data Capture or Model Monitor. |
Task 4.2 — Optimize and manage ML and AI infrastructure costs and performance
| Trigger Keywords & Scenario Constraints | Lock-In Architecture | Immediate Rejection / Trap to Avoid | Explanation |
|---|---|---|---|
| "Recommend instance type/count for a model under load" + "latency SLA" | SageMaker Inference Recommender (Default or Advanced load test) | AWS Compute Optimizer alone, or guessing ml.p4d |
Inference Recommender load-tests the actual model; Compute Optimizer is generic right-sizing. |
| "CPU tabular model (XGBoost/sklearn), cost-sensitive" | ml.c5 / ml.c6i / ml.c7i compute-optimized |
ml.p5 or ml.g5 GPU |
GPUs add cost with no gain for classical CPU models. |
| "Large in-memory features / wide models, CPU" | ml.r5 / ml.r6i memory-optimized |
ml.t burstable for prod SLAs |
Memory-optimized avoids swap; T-class bursting is a prod trap. |
| "Deep learning / LLM inference needing NVIDIA CUDA" | ml.g5 / ml.g6 (cost) or ml.p4d/ml.p5 (max performance) |
ml.inf2 if the model is not Neuron-compiled |
Inferentia needs Neuron SDK compilation; CUDA models stay on NVIDIA. |
| "Highest throughput / lowest $ per inference on compiled DL model" | ml.inf2 (Inferentia2) + AWS Neuron |
ml.p5 “because GPU is faster” |
Inf2 is the cost/perf lock-in once the model compiles to Neuron. |
| "Intermittent / unpredictable traffic" + "tolerate cold start" + "scale to zero" | SageMaker Serverless Inference | Real-time endpoint MinCapacity=1, or GPU instance |
Serverless has no idle instance charge; no GPU, no VPC, 6 GB / 60 s / ~4 MB limits. |
| "Payload up to 1 GB or inference up to 15–60 min" + "queue + S3 in/out" | SageMaker Asynchronous Inference + SNS completion + scale-to-zero | Real-time (seconds/MB limits) or Serverless (60 s, no GPU) | Async is the large-payload / long-running / idle-to-zero pattern. |
| "Async auto scale on queue depth" | Application Auto Scaling on ApproximateBacklogSize (optionally HasBacklogWithoutCapacity) |
InvocationsPerInstance only, or multiple conflicting policies |
Backlog size is the async-specific scale metric. |
| "Millions of offline scores, no API, S3→S3" | SageMaker Batch Transform | Always-on real-time endpoint | Batch has no idle endpoint cost. |
| "Tens/hundreds of similar models, same framework, sporadic each" | Multi-model endpoint (MME) on one instance | One endpoint per model | MME dynamically loads models; huge idle-cost win. |
| "Different frameworks on one endpoint" | Multi-container endpoint (direct or serial pipeline) | MME (same serving container) | MME ≠ multi-container; MME shares one image. |
| "Pack many models on one GPU endpoint, independent scaling, scale-to-zero copies" | SageMaker Inference Components | Classic MME on CPU XGBoost only | Inference components independently autoscale copies on a shared endpoint. |
| "Auto scale real-time endpoint with simplest policy" | Target tracking on SageMakerVariantInvocationsPerInstance |
Step scaling on Invocations (not per instance) or scheduled scaling only |
Per-instance invocations keep load stable as capacity changes. |
| "Scale GPU endpoint on hardware saturation" | Target tracking on GPUUtilization / CPUUtilization |
Scaling on 4XX errors | Utilization indicates saturation; 4XX is usually bad input. |
| "Steady 24/7 production traffic, tight p99" | Real-time endpoint + MinCapacity ≥ 1 + target tracking | Serverless (cold starts) or Spot | Spot/serverless break latency SLAs for steady prod. |
| "Right-size EC2/SageMaker family from utilization history" | AWS Compute Optimizer | Trusted Advisor “fault tolerance” check as the primary tool | Compute Optimizer is the right-sizing recommendation service. |
| "Find idle/underused endpoints and savings" | AWS Trusted Advisor + Cost Explorer | Amazon Inspector | Inspector is CVE scanning, not spend. |
| "Analyze historical spend by model/tag/account" | AWS Cost Explorer (+ cost allocation tags) | AWS Budgets (that is alerting) | Explorer analyzes; Budgets alerts/quotas. |
| "Alert when ML account will exceed $X / notify VP" | AWS Budgets (cost/usage budget + SNS) | Cost Explorer forecast chart only | Budgets is the quota/alert lock-in. |
| "Set a hard-ish cost guardrail for Bedrock/SageMaker" | AWS Budgets actions / IAM + quota + application inference profiles | Deleting CloudTrail | Never disable audit to save money. |
| "Commit 1 or 3 years for SageMaker instance $ off" | SageMaker Savings Plans | EC2 Standard RI on a SageMaker-managed instance, or Spot for a real-time endpoint | SageMaker usage is covered by SageMaker Savings Plans, not classic EC2 RIs. |
| "Training / batch that can be interrupted" | Managed Spot Training / Spot for batch | Spot for synchronous real-time customer traffic | Interruptions are acceptable for training/batch, not for live SLAs. |
| "Variable Bedrock PoC / spiky FM traffic" | Bedrock On-Demand | Provisioned Throughput (idle MU cost) | On-Demand is pay-per-token with no commitment. |
| "Steady high Bedrock QPS, must avoid throttling" or "customized Bedrock model inference" | Bedrock Provisioned Throughput (1-month or 6-month model units) | On-Demand retries forever | Custom models require Provisioned Throughput; PT also buys dedicated capacity. |
| "Bulk / offline FM scoring, not user-facing" + "cut FM bill ~50%" | Bedrock Batch Inference (prompts in S3, results in S3) | Provisioned Throughput for a one-off job | Batch is ~50% cheaper than on-demand and is not real-time. |
| "Long repeated system prompt / same document, many turns" + "cut input-token $ and latency" | Bedrock Prompt Caching (cachePoint; watch cacheReadInputTokenCount) |
Provisioned Throughput as the first lever | Cached input tokens are billed at a steep discount and skip re-prefill. |
| "Mix cheap small FM + expensive large FM in same family automatically" | Bedrock Intelligent Prompt Routing | Always routing to the largest model, or a custom Lambda classifier as the first answer | Router sends easy prompts to Haiku/Lite and hard ones to Sonnet/Pro (~30%+ savings). |
| "Reduce Bedrock quality cost without a new model train" | Prompt optimization + smaller model + max_tokens/stop sequences | Immediately buying PT | Token/output limits and smaller FMs are cheaper than capacity reservations. |
| "Attribute Bedrock $ to app/team/customer" | Bedrock application inference profiles + request metadata tags + Cost Explorer | One shared InvokeModel with no tags |
Inference profiles are the FM cost-allocation primitive. |
| "FM bill driven by output tokens" | Cap maxTokens, tighter prompts, summarization, streaming stop |
Switching to ml.p5 |
Output tokens dominate FM spend; GPU family is SageMaker-hosting, not Bedrock tokens. |
| "Embedding recompute cost high for unchanged corpus" | Cache embeddings; incremental KB ingest; reuse vectors | Re-embed entire corpus every request | Embedding compute is a first-class GenAI cost dimension. |
| "Vector DB storage $ growing" | Right-size OpenSearch k-NN (UltraWarm/cold) or cheaper vector store (S3 / RDS pgvector) + metadata filters | Recreating a SageMaker real-time GPU endpoint | Storage tiering is independent of inference instance family. |
| "Agent token/tool-call spend exploding" | CloudWatch GenAI observability token metrics + AgentCore resource-consumption metrics + tighter tool budgets | SageMaker Inference Recommender | Recommender does not see Bedrock agent tool loops. |
| "Idle async/serverless vs idle real-time" | Async or Serverless scale-to-zero | Real-time DesiredInstanceCount=1 left running |
Real-time min capacity keeps paying 24/7. |
| "Need both cost and p99 visibility on one ops screen" | CloudWatch dashboard (latency, errors, tokens, GPU, cost custom metric) | QuickSight as the on-call tool | On-call uses CloudWatch; QuickSight is analytical. |
| "Troubleshoot which microservice added 2 s" | AWS X-Ray + CloudWatch | Cost Explorer | Spend tools do not produce span timelines. |
"Bedrock throttling (ThrottlingException)" |
PT or cross-region inference profile + backoff; or smaller model / batch | Randomly changing IAM keys | Throttling is capacity, not credentials. |
| "Choose purchasing option: unknown vs steady vs bulk" | On-Demand (unknown) / Savings Plans or PT (steady) / Batch or Spot (bulk) | RIs on Lambda or Serverless | Match commitment to utilization shape. |
Task 4.3 — Secure ML and AI workloads and model endpoints
| Trigger Keywords & Scenario Constraints | Lock-In Architecture | Immediate Rejection / Trap to Avoid | Explanation |
|---|---|---|---|
| "Least privilege to invoke one endpoint" | IAM identity policy: sagemaker:InvokeEndpoint on that endpoint ARN |
AmazonSageMakerFullAccess managed policy |
FullAccess is the classic over-privilege trap. |
| "SageMaker job needs S3/ECR/KMS/CloudWatch only" | SageMaker execution role (task role, not human user keys) | Embedding long-term access keys in the container | Jobs assume an execution role; never bake keys into images. |
| "Users vs applications accessing models" | IAM roles for apps (temporary creds); IAM Identity Center/users with scoped policies | Sharing one IAM user access key across notebooks and prod | Roles + short-term creds are the application pattern. |
| "Force every training/endpoint into private subnets" | IAM Deny if sagemaker:VpcSubnets is null (+ optional sagemaker:NetworkIsolation) |
Security group “allow 0.0.0.0/0” and hope | Condition keys are preventive; open SGs are not. |
| "No internet from training/inference container at all" | EnableNetworkIsolation = true |
VPC with a NAT Gateway | Network isolation blocks all outbound, including VPC peers; NAT still allows egress. |
| "Private access to S3/SageMaker API/Runtime/ECR/KMS/Logs, no IGW" | Interface VPC endpoints (PrivateLink) + S3 Gateway endpoint | NAT Gateway “for security” | PrivateLink keeps traffic on AWS backbone; NAT is still internet egress. |
| "Invoke endpoint from VPC without public internet" | VPC interface endpoint sagemaker.runtime (PrivateLink) |
Public endpoint + IP allowlist | Runtime PrivateLink is the exam private-invoke lock-in. |
| "Isolate ML system network" | Dedicated VPC, private subnets, security groups (stateful), NACLs if extra layer | Public subnet + Elastic IP on the endpoint ENI | Endpoints should sit on private ENIs with SG ingress only from callers. |
| "SG vs NACL" | SG = stateful instance/ENI firewall; NACL = stateless subnet firewall | Using NACLs as the only app-layer auth | SGs are the default isolation control; NACLs are coarse. |
| "Encrypt artifacts/data at rest with customer-controlled key + audit" | AWS KMS CMK on S3, EBS, SageMaker volumes, Bedrock/Guardrail resources | SSE-S3 only, or disabling encryption | CMK enables key policy, rotation, and CloudTrail on Decrypt. |
| "Encryption in transit" | TLS + S3 bucket policy aws:SecureTransport=false → Deny |
KMS CMK alone | KMS is at rest; HTTPS deny is in transit. |
"Who called InvokeEndpoint / InvokeModel / changed IAM?" |
AWS CloudTrail (management + data events as needed) | CloudWatch Invocations metric |
CloudTrail is API audit; CloudWatch is operational metrics. |
| "Is the endpoint still VPC-only / KMS-enabled tomorrow?" | AWS Config rules (continuous config compliance) | CloudTrail (point-in-time API) | Config tracks resource state drift; CloudTrail tracks calls. |
| "PII sitting in S3 training buckets" | Amazon Macie | Bedrock Guardrails (that filters prompts/responses) | Macie discovers sensitive data at rest; Guardrails filter generative I/O. |
| "Mask/block PII inside prompts or model output" | Bedrock Guardrails Sensitive Information Filters (BLOCK or ANONYMIZE/MASK) |
Amazon Macie on the chat stream | Guardrails is the runtime PII control for FMs. |
| "Block hate/sexual/violence/misconduct + jailbreak" | Bedrock Guardrails content filters (Hate, Insults, Sexual, Violence, Misconduct, Prompt Attack) with LOW/MEDIUM/HIGH | AWS WAF (out of scope / HTTP layer) | Content filters are FM-layer; WAF is not the MLA answer. |
| "Never discuss competitor / medical advice / jailbreak topic" | Guardrails Denied Topics | Word filter exact-match only | Denied topics are semantic; word filters are exact strings. |
| "Block profanity or competitor brand strings" | Guardrails Word filters (managed profanity + custom) | Contextual grounding | Word filters are literal, not hallucination checks. |
| "RAG hallucination / ungrounded answer" | Guardrails Contextual Grounding Checks | Data Quality monitor | Grounding is a Guardrails policy, not a feature-drift job. |
| "Validate FM output against logical business rules" | Guardrails Automated Reasoning checks | Regex word filter | Automated Reasoning is formal-rule validation, not string match. |
| "Jailbreak / prompt injection / prompt leakage" | Guardrails Prompt Attack filter + input tags around user text | System-prompt-only with no Guardrail | Prompt-attack evaluation requires user-input tags on InvokeModel. |
| "Apply policy without invoking the expensive FM" | ApplyGuardrail API on input first |
Always InvokeModel then inspect |
Pre-filtering saves tokens and blocks before inference. |
| "Version and roll back a safety policy" | Guardrails versioning | Editing prompts in app code only | Guardrail versions are the safety-policy rollback unit. |
| "Alarm when Guardrails blocks spike" | CloudWatch InvocationsIntervened (Input vs Output) + SNS |
Macie finding | Intervention metrics are the runtime safety KPI. |
| "Scan container images / ECR / Lambda / EC2 for CVEs before deploy" | Amazon Inspector (block CodePipeline on HIGH/CRITICAL) | Amazon CodeGuru (source-code smells) | Inspector = workload/image vulnerabilities; CodeGuru = code analysis. |
| "Scan Python/Java source or PR for security bugs / secrets in code" | Amazon CodeGuru | Inspector CVE on an AMI | CodeGuru is SAST-style; Inspector is deployed-artifact CVEs. |
| "CI/CD must fail on vulnerable image or insecure code" | CodePipeline/CodeBuild + Inspector (images) + CodeGuru (code) | Manual docker push to prod ECR |
Exam lock-in for securing ML CI/CD is Inspector + CodeGuru gates. |
| "Store DB passwords, API tokens, tool credentials with rotation" | AWS Secrets Manager | Hardcoded env vars or committing .env to CodeCommit |
Secrets Manager is the in-scope rotation store. |
| "Production access to Bedrock FMs from an AWS app" | IAM role credentials (temporary) or short-term Bedrock API key | Long-term Bedrock API key or IAM user access key in the app | Long-term API keys are exploration-only and map to an IAM user. |
| "Quick playground / 12-hour bearer token inheriting caller perms" | Short-term Bedrock API key (≤ 12 h or session) | Long-term service-specific credential | Short-term keys inherit the generator’s IAM policy. |
| "Long-lived Bedrock bearer token" | Long-term Bedrock API key (exploration only; rotate; condition bedrock:bearerTokenType) |
Using it as the prod microservice credential | Exam wants IAM/short-term for prod, long-term only for trials. |
| "All Bedrock calls auditable" | CloudTrail on Bedrock APIs (+ optional invocation logging for payloads) | Disabling CloudTrail to “hide prompts” | API keys themselves are not written as secrets in CloudTrail. |
| "Prevent public notebook root / enforce volume encryption" | IAM/Config: disable notebook root access + require KMS on volumes | Security group change | Root access and unencrypted notebooks are standard SageMaker hardening items. |
| "Studio/domain in private VPC, no public internet" | SageMaker domain VPC-only + PrivateLink + CodeArtifact for packages | NAT + open PyPI as “good enough for prod PHI” | Air-gapped pattern uses VPC endpoints and private artifact repos. |
| "Private Python packages instead of public PyPI" | AWS CodeArtifact | Random public pip install from the training container with network isolation off |
CodeArtifact is the in-scope private registry. |
| "Cross-account prod invoke of a model" | IAM role assumption into the prod account + resource policy as needed | Sharing long-term access keys | Cross-account = roles, never keys. |
| "Org-wide deny of public SageMaker or unencrypted jobs" | AWS Organizations SCPs | Per-user inline policy only | SCPs are the multi-account preventive guardrail. |
| "Approved self-service ML products" | AWS Service Catalog (governed SageMaker project templates) | Untracked CreateEndpoint in the console |
Service Catalog packages approved, pre-secured architectures. |
"Debug why endpoint invoke returns AccessDenied" |
IAM policy simulator / CloudTrail AccessDenied + missing KMS Decrypt or S3 GetObject |
Opening 0.0.0.0/0 on the SG first |
Most ML 403s are IAM/KMS/S3, not NACLs. |
| "Debug why VPC endpoint still times out" | SG on ENI, subnet route to prefix list, Private DNS, endpoint policy | Adding an IGW “just in case” | Timeouts are network path/DNS/endpoint-policy, not missing internet. |
| "Encrypt traffic between distributed training containers" | Inter-container traffic encryption | KMS on S3 only | S3 CMK does not encrypt east-west training traffic. |
| "Tag everything for cost and ABAC" | Mandatory tags (Environment, Model, Owner) + IAM tag conditions |
Untagged SageMakerFullAccess |
Tags enable Cost Explorer and ABAC least privilege. |
| "Responsible AI policy at runtime (toxicity, PII, topics)" | Bedrock Guardrails attached to agents/KBs/InvokeModel |
Amazon Macie or Model Monitor Bias Drift | Bias drift is statistical fairness on tabular models; Guardrails is generative safety. |
| "Separate prod / dev blast radius" | Multi-account: org + separate prod account + SCPs + private endpoints | One account, one FullAccess role |
Account isolation is the strongest exam blast-radius answer. |
| "S3 model artifacts only from the VPC endpoint" | S3 bucket policy Deny if aws:SourceVpce ≠ approved endpoint |
Public bucket + presigned URLs emailed around | Bucket policies bind data-plane access to the VPCe. |
| "Rotate compromised notebook credentials" | Disable keys, rotate Secrets Manager, invalidate sessions, CloudTrail review | Rebuilding the VPC | Credential incidents are IAM/Secrets/CloudTrail, not a new CIDR. |