Task 3.3: Deployment and Orchestration of ML and AI Workflows - Implement automated orchestration and continuous integration and continuous delivery (CI/CD) pipelines for MLOps and AI workloads.
Service Map & MLOps Handoff
| Service |
Primary Job |
MLOps Use |
Key Config |
| CodeCommit |
Managed Git repos inside AWS |
Source for pipeline |
IAM auth, KMS, triggers |
| CodeBuild |
Managed build/test |
Unit tests, container builds, eval scripts |
buildspec.yml, service role, VPC, cache |
| CodeDeploy |
Deploy to EC2/ECS/Lambda |
Deploy inference services; traffic shifting + rollback |
AppSpec, deployment group, alarms |
| CodePipeline |
Release orchestration |
Stages, approvals, artifact flow |
Stages, actions, artifact S3 + KMS, service role |
| CodeConnections |
External Git link |
Source from GitHub/GitLab/Bitbucket without stored credentials |
Connection ARN, OAuth/GitHub App |
| SageMaker Pipelines |
ML step orchestration |
Processing, training, evaluation, registration |
Pipeline definition, steps |
| SageMaker Model Registry |
Model versioning |
Approval status, lineage, audit |
Model group, model package |
| EventBridge |
Event routing |
Trigger pipeline on S3 upload, approval, schedule |
Rules, targets |
Exam Gotcha: CodePipeline orchestrates the release; SageMaker Pipelines orchestrates ML steps. They meet at a handoff—not one tool doing both.
CodeCommit
- Core: Managed Git repos in AWS. IAM users/roles for auth. Encrypted at rest with KMS.
- Triggers: SNS/Lambda on push, branch, tag.
- Limits: Repo/file size quotas. No longer available to new customers—existing only.
- Troubleshooting: Access denied → IAM policy; clone fails → credential helper; trigger not firing → event bus.
- Anti-pattern: Using CodeCommit as source for external GitHub repo. Use CodeConnections.
CodeBuild
- Core: Managed build service. Runs commands in
buildspec.yml. Provisioned build environments (Docker images).
- Key phases:
install, pre_build, build, post_build; plus artifacts, cache, env.
- Compute: EC2, Lambda, ARM/Graviton, GPU for ML builds.
- Networking: VPC config for private resources (RDS, ECR, SageMaker). Needs NAT/endpoints.
- Security: Service role, Secrets Manager/Parameter Store, no secrets in
buildspec.
- Troubleshooting: Timeout → increase timeout; VPC → subnets/SG; Docker → privileged mode; cache → S3/local; permissions → service role.
- Anti-pattern: Hardcoding credentials; no cache; building every commit without tests.
CodeDeploy
- Core: Automates deployments to EC2/on-prem, Lambda, ECS. Uses
AppSpec. Deployment groups, configs, rollback.
- Deployment configs:
AllAtOnce, HalfAtOnce, OneAtATime, Canary (10% 5min), Linear (10% every minute).
- Rollback: Automatic on alarm or deployment failure. Requires CloudWatch alarms.
- Hooks:
BeforeInstall, AfterInstall, ApplicationStart, ValidateService, etc.
- MLOps: Deploy Lambda/ECS inference services. For SageMaker endpoints, use SageMaker Deployment Guardrails (canary/linear, auto-rollback on alarms).
- Troubleshooting: Deployment failed → check hooks, permissions, target health. Rollback not triggered → no alarm configured.
- Anti-pattern: No alarm → bad deployment completes. Using CodeDeploy for SageMaker endpoints.
CodePipeline
- Core: Orchestrates stages (Source, Build, Test, Deploy, Approval, Invoke). Artifacts stored in S3 (KMS encrypted). Service role.
- Actions: Source (CodeCommit, S3, ECR, CodeConnections), Build (CodeBuild), Test (CodeBuild, Lambda), Deploy (CodeDeploy, ECS, CloudFormation, SageMaker), Approval (manual), Invoke (Lambda).
- Triggers: Event-driven (EventBridge, webhooks), scheduled, manual.
- Cross-account: Supports cross-account deployments via roles and artifact buckets.
- Troubleshooting: Artifact access denied → S3/KMS policy; stage transition blocked → approval; webhook not firing → CodeConnections status.
- Anti-pattern: Storing secrets in pipeline; no approval gate before prod; no artifact encryption.
CodeConnections
- Core: Managed connection to external Git (GitHub, GitHub Enterprise, GitLab, Bitbucket). Formerly CodeStar Connections.
- Auth: OAuth or GitHub App. No credentials stored in pipeline.
- Use: Source action in CodePipeline for external repos.
- Troubleshooting: Connection status
PENDING → complete handshake; access denied → app permissions; webhook not firing → connection ARN.
- Anti-pattern: Using CodeCommit for external repos; storing PATs in pipeline.
Security & Networking
- IAM: Least-privilege service roles for CodeBuild, CodePipeline, CodeDeploy. Cross-account roles.
- Encryption: KMS for CodeCommit repos, CodePipeline artifacts, CodeBuild artifacts, S3 buckets.
- Network: VPC endpoints for CodeBuild, CodePipeline, CodeCommit, S3, KMS, Secrets Manager. CodeBuild VPC for private resources.
- Secrets: Secrets Manager / Parameter Store—never in
buildspec.
- Audit: CloudTrail, CloudWatch Logs, CodePipeline history, Model Registry lineage.
Decision Matrix: When to Use X vs Y
| Need |
Use |
Not |
| External Git source |
CodeConnections |
CodeCommit |
| ML step orchestration |
SageMaker Pipelines |
CodePipeline alone |
| Release orchestration |
CodePipeline |
SageMaker Pipelines |
| Build/test code |
CodeBuild |
CodeDeploy |
| Deploy to EC2/ECS/Lambda |
CodeDeploy |
SageMaker Deployment Guardrails |
| Deploy SageMaker endpoint |
SageMaker Deployment Guardrails |
CodeDeploy |
| Model versioning/approval |
SageMaker Model Registry |
CodeCommit |
| Prompt versioning |
Bedrock Prompt Management |
Application code |
| Retrain trigger |
EventBridge (event/schedule) |
Manual run |
| RAG refresh |
Bedrock Knowledge Bases ingestion resync |
Full reindex every time |
Troubleshooting Quick Reference
| Symptom |
Likely Cause |
Fix |
| CodeBuild fails at start |
Service role missing permissions |
Attach IAM policy |
| CodeBuild cannot reach RDS |
No VPC config |
Add VPC, subnets, SG |
| CodeBuild Docker fails |
Privileged mode off |
Enable privileged |
| CodePipeline artifact error |
S3/KMS policy |
Grant pipeline role access |
| CodeDeploy rollback not triggered |
No CloudWatch alarm |
Attach alarm to deployment group |
| CodeConnections source fails |
Connection not available |
Complete OAuth handshake |
| CodeCommit clone denied |
IAM/credential helper |
Configure Git credentials |
| SageMaker endpoint bad deploy |
No alarm in deployment guardrail |
Configure auto-rollback alarm |
Exam Gotchas
CodeBuild needs a service role; CodePipeline needs a service role and an artifact S3 bucket with KMS.
CodeDeploy rollback only works if alarms are configured. No alarm = no rollback signal.
CodeConnections is for external repos; CodeCommit is internal Git.
SageMaker Pipelines ≠ CodePipeline. Use both: CodePipeline for CI/CD, SageMaker Pipelines for ML steps.
Model Registry approval status is read by deployment step; don't deploy directly from training.
Prompts and agents need versioning just like models.
Ingestion resync is incremental; changing embedding model/chunking requires full reindex.
CodeCommit no longer available to new customers—use CodeConnections for external, or other Git.