Skip to content

Skill 3.2.3: Build and maintain containers for ML and AI workloads

Task 3.2: Deployment and Orchestration of ML and AI Workflows - Provision and configure resources for ML and AI workloads based on existing architecture and requirements

Core Objective: Package ML/AI code, dependencies, and runtime into portable, versioned containers that SageMaker (and other AWS compute) can execute for training, inference, and processing—while meeting security, scalability, and maintenance requirements.

Container Types & Selection

Type Description Use When Trade-offs
Prebuilt Framework AWS-maintained Deep Learning Containers (TensorFlow, PyTorch, Hugging Face, Scikit-learn, XGBoost) Standard frameworks, quick start Less control; may lack custom libraries
Extended Prebuilt base + additional packages (e.g., pip install) Need extra libraries but want AWS-maintained base Larger image; must rebuild on base updates
Custom (BYO) Built from scratch or custom base Unique dependencies, proprietary code, strict compliance Full control; highest maintenance burden
Multi-Model / Multi-Container Host multiple models or containers on one endpoint Many small models; cost efficiency Complexity; resource contention

Exam Gotcha: SageMaker inference containers must run a web server on port 8080 and implement /ping (health) and /invocations (predict). Training containers must read from /opt/ml/input/data and write model artifacts to /opt/ml/model.

SageMaker Container Contract (Key Paths)

Workload Input Path Output Path Key Requirement
Training /opt/ml/input/data/{channel} /opt/ml/model Write final model; hyperparameters as JSON
Inference N/A N/A Web server on 8080; /ping and /invocations
Processing /opt/ml/processing/input /opt/ml/processing/output Run script; no model artifact
Batch Transform S3 input S3 output Container must handle /invocations

Amazon ECR: Build, Store, Maintain

  • Registry: Private ECR repo per model/team; enable cross-region replication for multi-region deployments.
  • Tagging: Use immutable tags or image digest for reproducible deployments. Avoid latest.
  • Scanning: Enable scan on push (basic) or Amazon Inspector for enhanced vulnerability scanning.
  • Lifecycle: Set lifecycle policies to expire untagged/old images; reduces cost and clutter.
  • Security: Encrypt images with KMS; restrict pull/push with IAM policies; use VPC endpoints for private pulls.
  • CI/CD: Automate builds with CodeBuild/CodePipeline; run tests; push to ECR; update SageMaker model/endpoint.

Exam Gotcha: SageMaker execution role must have ecr:GetDownloadUrlForLayer, ecr:BatchGetImage, ecr:BatchCheckLayerAvailability, and ecr:GetAuthorizationToken to pull images.

Networking & Security for Private Containers

  • VPC Configuration: Real-time endpoints and training jobs can run in private subnets with security groups. Serverless Inference does not support VPC and does not support GPU.
  • Private Pull: To pull ECR images without internet, create VPC endpoints for:
  • com.amazonaws.{region}.ecr.api
  • com.amazonaws.{region}.ecr.dkr
  • com.amazonaws.{region}.s3 (Gateway endpoint) for layers and model artifacts.
  • Outbound Access: For other AWS services, use interface VPC endpoints or NAT Gateway if required.
  • Encryption: ECR images encrypted at rest (KMS). Data in transit via TLS. SageMaker inter-container traffic encrypted with KMS if enabled.

GPU & Accelerated Workloads

  • Base Images: Use NVIDIA CUDA-enabled base images (e.g., nvidia/cuda or AWS DLC GPU variants).
  • Instance Types: Match container to GPU instance (P3, P4, G4, G5). Container must include CUDA toolkit and cuDNN.
  • SageMaker Host: Manages NVIDIA drivers on host; container must not install drivers.
  • Scaling: GPU utilization is a custom scaling metric; use Application Auto Scaling with target tracking on GPUUtilization or custom CloudWatch metric.
  • HyperPod: For resilient training clusters, use SageMaker HyperPod with containerized training jobs; it auto-replaces faulty hardware.

Maintenance & Anti-Patterns

  • Version Everything: Base image, dependencies, model code. Rebuild on base image CVEs.
  • Keep Images Small: Use multi-stage builds, slim base images, remove caches. Large images slow cold start and scale-out.
  • Secrets: Never bake secrets into images. Use IAM roles, Secrets Manager, or environment variables at runtime.
  • Testing: Validate /ping and /invocations locally before pushing. Test with sample payloads.
  • Anti-Patterns:
  • Using latest tag in production.
  • Ignoring base image security updates.
  • Mismatched CUDA/cuDNN versions between container and instance.
  • Forgetting VPC endpoints in private subnets (causes timeout pulling image).
  • Storing model artifacts inside container instead of /opt/ml/model (training) or S3 (inference).

Decision Matrix: Hosting Options & Container Support

Option Custom Container GPU VPC Scaling Signal
Real-time Endpoint Yes Yes Yes Invocations, concurrency, custom
Serverless Inference Yes (limited) No No Concurrency (high-res)
Batch Transform Yes Yes Yes N/A (job-based)
Asynchronous Inference Yes Yes Yes Queue length, custom
HyperPod Yes (training) Yes Yes N/A (cluster)

Exam Gotcha: Serverless Inference is best for intermittent, bursty traffic with no VPC/GPU needs. For private networking or GPU, use Real-time or Asynchronous Inference.

Key AWS Services for Container Workflows

Service Role
Amazon ECR Store, version, scan, and replicate container images.
SageMaker AI Run containers for training, inference, processing, batch transform.
SageMaker HyperPod Resilient GPU clusters for training with containerized jobs.
AWS CodeBuild/CodePipeline CI/CD for building and pushing container images.
Amazon Inspector Vulnerability scanning for ECR images.
AWS KMS Encrypt ECR images and SageMaker inter-container traffic.
Amazon VPC Private networking, subnets, security groups, VPC endpoints.

Bottom Line: Build once, run anywhere—but on AWS, the container must satisfy SageMaker’s contract, be stored securely in ECR, and be reachable without internet in private VPCs. Choose prebuilt for speed, custom for control, and always version, scan, and keep images lean.