Skip to content

Skill 3.2.4: Configure SageMaker AI endpoints within VPC network environments

Task 3.2: Deployment and Orchestration of ML and AI Workflows - Provision and configure resources for ML and AI workloads based on existing architecture and requirements.

Core Objective: Deploy SageMaker AI endpoints inside an Amazon VPC to satisfy private networking, compliance, and access-to-private-resource requirements. The endpoint’s ENIs are placed in your subnets; the model container can reach AWS services and private resources without traversing the public internet.

Key Decision: A VPC requirement eliminates serverless inference. Choose real-time, asynchronous, or batch transform.

VPC Configuration Essentials

  • VpcConfig (Endpoint Configuration): Attach to the endpoint config. Contains:
  • SecurityGroupIds: One or more security groups. Must be in the same VPC as the subnets.
  • Subnets: Up to 16 subnet IDs. Must be in the same VPC as the security groups. Use subnets across ≥2 Availability Zones for HA.
  • EnableNetworkIsolation: If true, the model container has no inbound/outbound network. Use only when the container needs no external calls (all dependencies baked into the image).
  • Serverless Inference: Does not accept VpcConfig. If VPC is mandatory, serverless is invalid.
  • Asynchronous Inference: Supports VpcConfig. Good for large payloads (≤1 GB) and long processing (≤15 min).
  • Batch Transform: Supports VpcConfig. For offline, large-scale inference. No persistent endpoint.
  • Real-time Inference: Supports VpcConfig. For low-latency, persistent endpoints, custom containers, and GPU workloads.

Exam Gotcha: SageMaker AI endpoints in VPC require subnets in at least two Availability Zones, even for a single hosting instance. Single-AZ placement is a trap.

Networking & Private Access

Requirement Configuration Notes
Model artifacts in S3 S3 Gateway VPC Endpoint Attach to private subnet route tables. Free.
Container image in ECR Interface VPC Endpoints for ecr.api, ecr.dkr + S3 Gateway for layers Interface endpoints require security groups.
CloudWatch Logs/Metrics Interface endpoints for logs, monitoring Required if no NAT/IGW.
KMS decryption Interface endpoint for kms If using CMK for encrypted artifacts/volumes.
Secrets Manager / SSM Interface endpoints for secretsmanager, ssm If container fetches secrets/parameters.
Other AWS services Interface endpoint per service Or NAT Gateway (not private) if compliance allows.
Client invocation SageMaker Runtime API If clients are in VPC, create interface endpoint for sagemaker.runtime. Endpoint SG does not need inbound from clients.

Exam Gotcha: Private subnets with no internet route cannot reach S3/ECR until you add the appropriate VPC endpoints. A missing S3 Gateway endpoint is the #1 cause of endpoint startup failure.

Exam Gotcha: NAT Gateway provides outbound internet, not private connectivity. For strict “no traffic leaves private network,” use VPC endpoints, not NAT.

Security Group Rules:

  • Endpoint SG (model container): Outbound to interface endpoint SG on port 443 (or to target services).
  • Interface Endpoint SG: Inbound from endpoint SG on port 443.
  • Gateway Endpoints (S3): Use route tables, no security groups.

IAM & Security

  • Execution Role: Attached to the model/endpoint. Must allow:
  • S3 read for model artifacts.
  • ECR pull for container images.
  • CloudWatch Logs write for logs.
  • KMS decrypt if using CMK.
  • EC2 permissions for ENIs: ec2:CreateNetworkInterface, ec2:DescribeNetworkInterfaces, ec2:DeleteNetworkInterface, ec2:DescribeVpcs, ec2:DescribeSubnets, ec2:DescribeSecurityGroups.
  • VPC Endpoint Policies: Can restrict which resources are accessible through the endpoint.
  • Network Isolation: EnableNetworkIsolation=true blocks all container network. The service still needs S3 read to pull artifacts, but the container cannot call AWS services.

Scaling & Metrics

  • Application Auto Scaling registers the endpoint variant or inference component as a scalable target.
  • Scaling Policies:
  • Target Tracking: Predefined metric SageMakerVariantInvocationsPerInstance (standard 1-min resolution). General purpose.
  • High-Resolution Concurrency: ConcurrentRequestsPerModel / ConcurrentRequestsPerCopy (10-sec resolution). Use for streaming or long-lived requests.
  • Custom Metric: e.g., GPU utilization, CPU utilization. For GPU workloads, scale on GPU utilization or use inference components. Test before production.
  • Cooldowns: Prevent flapping. Scale-out cooldown should be short; scale-in cooldown longer.
  • Inference Components: For multi-model endpoints, scale individual models. VPC config remains at the endpoint level.
  • GPU Workloads: Use GPU-backed instances (e.g., ml.g5, ml.p4d). Scale on GPU utilization or concurrency. Consider inference components for model-level scaling.

Decision Matrix: Endpoint Type vs VPC

Endpoint Type VPC Support Best For VPC Gotcha
Real-time Yes Low-latency, persistent, custom containers, GPU Must configure VpcConfig; multi-AZ subnets.
Serverless No Intermittent, spiky, small models, no VPC Cannot meet VPC/compliance requirements.
Asynchronous Yes Large payloads (≤1 GB), long processing (≤15 min), queued Requires S3 output; VpcConfig supported.
Batch Transform Yes Offline, large datasets, no persistent endpoint VpcConfig supported at job level.

Anti-Patterns & Exam Traps

  • Using serverless for VPC-required workloads: Serverless has no VpcConfig. Invalid.
  • Forgetting S3 Gateway Endpoint: Private subnets cannot pull model artifacts from S3 without it.
  • Forgetting ECR Interface Endpoints: Container image pull fails in private subnets.
  • Missing EC2 ENI permissions in execution role: Endpoint fails to create network interfaces.
  • Single subnet/AZ: No HA; use ≥2 AZs.
  • Public subnets for compliance: Traffic can traverse IGW. Use private subnets + VPC endpoints.
  • NAT Gateway for strict private compliance: NAT allows internet egress. Use VPC endpoints.
  • Overlooking interface endpoint security groups: Endpoint SG must allow outbound to interface endpoint SG; interface endpoint SG must allow inbound from endpoint SG.
  • Assuming network isolation still allows AWS calls: EnableNetworkIsolation=true blocks all container network. Only use if no external calls.
  • Ignoring high-resolution concurrency for streaming: Standard 1-min invocation metric scales too slowly for streaming/long-lived requests.

Quick Recall

  • VPC config: SecurityGroupIds + Subnets (≥2 AZs).
  • Private access: S3 Gateway + ECR/CloudWatch/KMS Interface Endpoints.
  • IAM: S3 read, ECR pull, CloudWatch Logs, KMS decrypt, EC2 ENI permissions.
  • Serverless: No VPC. Real-time/Async/Batch: VPC supported.
  • Scaling: Application Auto Scaling; target tracking on invocations, concurrency, or custom GPU metric.
  • Network isolation: Blocks all container network; use only when no external calls.