Skill 3.3.3: Configure training and inference jobs

Task 3.3: Deployment and Orchestration of ML and AI Workflows - Implement automated orchestration and continuous integration and continuous delivery (CI/CD) pipelines for MLOps and AI workloads

Core objective: Make ML training and inference repeatable, testable, and reversible inside CI/CD. Separate ML orchestration (SageMaker Pipelines) from release orchestration (CodePipeline); hand off through SageMaker Model Registry approval.

Training Job Configuration (SageMaker)

Component	Key Configuration	Exam Focus
Execution role	IAM role assumed by training job	Least privilege; S3, ECR, CloudWatch, KMS
Algorithm/Container	Training image from ECR or built-in algorithm	Framework version, custom container
InputDataConfig	S3 channels: train/validation/test	File mode copies to EBS; Pipe mode streams—better for large data
ResourceConfig	Instance type/count, volume size, volume KMS	GPU vs CPU; distributed training
StoppingCondition	Max runtime, max wait for spot	Managed Spot Training requires checkpointing
HyperParameters	Static values	Tuning via SageMaker Automatic Model Tuning
OutputDataConfig	S3 model artifact path, KMS	Encrypted output
VpcConfig	Subnets, security groups, network isolation	No internet; use VPC endpoints
CheckpointConfig	S3 checkpoint path	Resume spot training after interruption
Debugger/Profiler	Rules, metrics	Detect vanishing gradients, bottlenecks

Distributed training: Data parallel for large batches; model parallel for large models; SageMaker Distributed libraries.
Retraining triggers:

Schedule: fixed cadence; simple, wastes compute if no change.
Event-driven: S3 ObjectCreated, EventBridge rule → start pipeline; efficient for irregular data.
Manual: during pipeline trust-building.

Inference Job Configuration

Hosting Option	Use When	Trade-offs
Real-time endpoint	Low-latency, persistent traffic	Always-on cost; autoscaling; A/B testing
Serverless inference	Intermittent, spiky traffic	Scale to zero; cold start
Asynchronous inference	Large payloads, long processing	Queue; near-real-time
Batch Transform	Offline scoring on large datasets	No endpoint; job-based
Multi-model endpoint	Many models, shared resources	Dynamically loads models; cold start per model
Multi-container endpoint	Different frameworks in one endpoint	Sequential inference; complex
Inference components	Multiple models on same endpoint	Resource isolation; cost optimization

Deployment strategies: All-at-once (downtime risk), Canary (small % first), Linear (even steps), Rolling (batch replace).
Deployment Guardrails: bring up new fleet, shift traffic, bake, watch CloudWatch alarms; if alarm trips, auto-rollback to old fleet.

Gotcha: A managed rollout with no CloudWatch alarm configured has no rollback signal—it will complete a bad deployment as “successful.”

Endpoint autoscaling: target tracking on InvocationsPerInstance, CPUUtilization, ModelLatency.
Monitoring: Data capture + Model Monitor for drift.

CI/CD Pipeline Integration

Stage	AWS Service	Purpose
Source	CodeCommit, CodeConnections	Git repo inside AWS or external without stored credentials
Build/Test	CodeBuild	Unit tests, container builds, evaluation scripts
ML Orchestration	SageMaker Pipelines	Processing → Training → Evaluation → Condition → RegisterModel
Approval	SageMaker Model Registry	Version, metadata, lineage, approval status
Deploy	CodeDeploy, SageMaker Endpoints	Blue/green, canary, rollback
Trigger	EventBridge	S3 events, pipeline status, model approval, schedule

Handoff: SageMaker Pipelines ends at Model Registry registration, not deployment. CodePipeline reads approval status and deploys. This separates model building from release.
Automated testing:

Code tests: deterministic; fail build on unit/integration test failure.
Traditional ML: gate on metric threshold over held-out dataset.
Generative AI: non-deterministic; run fixed prompt set, score against criteria/reference, compare aggregate to previous version. Version prompts alongside model.

Versioning artifacts: Model Registry (model versions), MLflow on SageMaker (experiment tracking), Bedrock Prompt Management (prompt versions), agent versioning (config + tools).

Security & Compliance

IAM: execution roles for training, inference, pipelines; least privilege.
KMS: encrypt S3 inputs/outputs, EBS volumes, model artifacts, endpoints, ECR.
TLS: in transit for all API calls and data movement.
VPC endpoints (PrivateLink): S3, ECR, CloudWatch, SageMaker API; avoid public internet.
Network isolation: training containers cannot make outbound calls.
Inter-container encryption: distributed training traffic.
Audit: Model Registry lineage, CloudTrail, CloudWatch Logs.

Decision Matrix & Anti-Patterns

Decision	Choose	Avoid
Large dataset training	Pipe mode, distributed	File mode copying all data
Cost-sensitive training	Managed Spot + checkpointing	Spot without checkpoints
Intermittent inference	Serverless	Always-on real-time endpoint
Large batch scoring	Batch Transform	Real-time endpoint
Safe model rollout	Canary/linear + alarms	All-at-once without rollback
Retraining trigger	Event-driven for irregular data	Fixed schedule when data irregular
RAG refresh	Incremental resync	Full reindex unless embedding/chunking changed
Prompt change	Versioned prompt in Bedrock Prompt Management	Editing prompt in app code

Anti-pattern: Deploying a model directly from a training pipeline without a registry approval gate. This removes auditability and rollback control. Anti-pattern: Treating a vector index copy as a refresh. Copying does not regenerate embeddings; stale vectors remain.

Key exam traps:

SageMaker Pipelines ≠ CodePipeline: Pipelines orchestrate ML steps; CodePipeline orchestrates release stages. They integrate, not replace.
CodeConnections for external Git; CodeCommit for AWS-hosted Git.
Model Registry approval status is the gate a deployment step reads.
Rollback only works if alarm is configured during baking period.
Managed rollout shapes not supported on all hosting options.
Prompts and agents are versioned artifacts—not just model weights.