Skill 4.3.1: Secure CI/CD pipelines by checking for code and image vulnerabilities
Task 4.3: Operating, Monitoring, and Securing ML and AI Solutions - Secure ML and AI workloads and model endpoints
Core objective: Embed automated security checks into ML CI/CD so vulnerable code, dependencies, secrets, and container images are blocked before deployment to training jobs or inference endpoints.
Pipeline security gates
| Gate | Threat | AWS Service / Feature | What it detects | Enforcement |
|---|---|---|---|---|
| Source / commit | Hardcoded credentials, insecure code patterns | Amazon CodeGuru Security | SAST: security policy violations, injection flaws, hardcoded secrets | Fail build / create finding |
| Build / dependencies | Vulnerable open-source packages | CodeGuru Security + dependency scanning | CVEs in libraries | Gate on severity threshold |
| Container image | Vulnerable OS packages, app dependencies | Amazon Inspector + Amazon ECR scanning | CVEs in image layers | Block push/deploy if critical/high |
| Runtime / deployment | Unintended network exposure, vulnerable Lambda/EC2 | Amazon Inspector | Software vulnerabilities, network reachability | Continuous scan; prioritize findings |
| Pipeline config | Over-permissive IAM, unencrypted artifacts | IAM Access Analyzer, AWS Config, CloudTrail | Drift, excessive permissions, missing encryption | Automated remediation / alerts |
Amazon CodeGuru vs Amazon Inspector
| Dimension | Amazon CodeGuru Security | Amazon Inspector |
|---|---|---|
| Primary purpose | Static application security testing (SAST) | Vulnerability management for compute and images |
| Scans | Source code, hardcoded secrets, insecure patterns | EC2 instances, ECR container images, Lambda functions |
| Stage | Build / commit | Build (image) and runtime (continuous) |
| ML relevance | Catches insecure training/inference code, leaked keys | Catches vulnerable base images and ML framework CVEs |
| Output | Findings with severity, remediation guidance | Findings with CVE, severity, exploitability |
| Integration | CodePipeline, CodeBuild, GitHub, Bitbucket | ECR scan on push, continuous EC2/Lambda scanning |
| Key differentiator | Looks at what was written | Looks at what was assembled and running |
ECR image scanning: basic vs enhanced
| Feature | Basic scanning | Enhanced scanning (Inspector) |
|---|---|---|
| Engine | Clair (open-source) | Amazon Inspector |
| Scan trigger | On push | On push + continuous re-scan |
| Scope | OS packages | OS + programming language packages |
| CVE freshness | Point-in-time | Continuous as new CVEs emerge |
| Use case | Basic hygiene | Production ML containers, compliance |
| Exam gotcha | Does not re-scan after new CVEs | Required for continuous vulnerability management |
Exam Gotcha: CodeGuru Security and Inspector are complementary, not substitutes. CodeGuru finds insecure code and secrets; Inspector finds vulnerable packages and images. A pipeline needs both for full coverage.
Exam Gotcha: ECR basic scanning only scans on push. If a new CVE is published later, the image remains unscanned until enhanced scanning or manual re-scan. For continuous compliance, use Inspector enhanced scanning.
Exam Gotcha: CodeGuru Reviewer suggests code quality improvements; CodeGuru Security detects security vulnerabilities and hardcoded secrets. Do not confuse them.
When to use X vs Y
| Scenario | Best choice | Why |
|---|---|---|
| Detect hardcoded AWS keys in training script | CodeGuru Security | SAST scans source for secrets |
| Detect vulnerable Python package in container | Inspector enhanced scanning | Scans language packages in ECR |
| Scan Lambda function for CVEs | Inspector | Continuous Lambda scanning |
| Block deployment if critical CVE found | CodePipeline + CodeBuild quality gate | Automated enforcement |
| Audit who changed pipeline permissions | CloudTrail + AWS Config | API activity + configuration history |
| Rotate leaked secret | AWS Secrets Manager | Central rotation, avoids code embedding |
Security & compliance controls for the pipeline
- IAM least privilege: Pipeline roles should only allow required actions on specific CodeCommit/CodeBuild/ECR/S3/KMS resources. Separate human and workload identities.
- Artifact encryption: Use KMS customer managed keys for CodePipeline artifacts, ECR images, and S3 buckets. Encrypt in transit with TLS.
- Network isolation: Use VPC endpoints (AWS PrivateLink) for CodePipeline, CodeBuild, ECR, S3, KMS so pipeline traffic stays private. Security groups must restrict inbound/outbound.
- Audit & monitoring: CloudTrail records pipeline API calls. AWS Config rules detect drift (e.g., unencrypted ECR, public S3). CloudWatch alarms on failed security gates.
- Secrets management: Never store credentials in code, images, or environment files. Use Secrets Manager or Systems Manager Parameter Store with rotation.
Anti-patterns & troubleshooting
- Widening IAM to clear AccessDenied: The error names the operation and resource. Grant exactly that, not
*. Broad permissions hide the root cause. - Treating scan findings as reports: Findings must gate the pipeline. A critical CVE should fail the build or block deployment.
- Scanning only at build: New CVEs emerge after build. Use continuous scanning (Inspector enhanced) for running images and Lambda.
- Ignoring base image updates: ML containers often use old base images. Rebuild regularly to pick up patched OS packages.
- Hardcoded secrets in notebooks or training scripts: CodeGuru Security detects them, but prevention is better. Use Secrets Manager.
- Assuming CloudTrail logs data events by default: Model invocation and S3 object reads are data events and must be explicitly enabled. This is critical for auditing ML endpoints.
Key exam takeaways
- CodeGuru Security = SAST for code and secrets. Inspector = CVE scanning for ECR, EC2, Lambda.
- ECR enhanced scanning = continuous, language-aware, Inspector-powered. Basic = on-push, OS-only.
- Pipeline gates must be automated and enforced; findings should block promotion to production.
- Least privilege, KMS encryption, VPC endpoints, CloudTrail, Config are the security foundation for CI/CD.
- ML-specific risks: over-permissive training job roles, publicly readable model artifacts, unvalidated external models, internet-reachable inference endpoints.
- Credential type for FMs: Production uses IAM roles with temporary credentials; API keys are for exploration/development, not production.