Skip to content

Task 4.2: Monitor and Optimize Infrastructure and Costs

MLA-C01 Domain 4 — High-Quality Study Notes with Low-Cost Hands-on Exercises

Contents

  1. Task Overview & Mental Model
  2. Core Concepts Cheat-Sheet
  3. Exercise 1: CloudWatch + Lambda Observability
  4. Exercise 2: CloudTrail Trail + Metric Filter for Policy / API Change Detection
  5. Exercise 3: EventBridge Rules for Automation
  6. Exercise 4: Cost Governance: Tagging Strategy, Budgets, Cost Explorer, Trusted Advisor, Cost Optimization Hub
  7. Exercise 5: Rightsizing, Purchasing Options, Inference Recommender & Compute Optimizer
  8. Exam Traps & Tips Master List
  9. Quick-Reference Tables

1. Task Overview & Mental Model

Task 4.2 sits at the intersection of the Reliability and Cost Optimization pillars of the AWS Well-Architected Framework applied to ML workloads. You must be able to:

  • Collect metrics, logs, and traces so the system is observable.
  • Detect anomalies (latency spikes, low utilization, quota exhaustion, policy changes).
  • Automatically remediate or notify.
  • Right-size compute, choose the correct purchasing model, and allocate costs via tags.

💡 Observability triad for the exam * Metrics: CloudWatch (SageMaker endpoint latency, CPU/Mem, Invocations, ModelLatency, OverheadLatency) * Logs: CloudWatch Logs + Logs Insights + Lambda Insights * Traces: AWS X-Ray (end-to-end request path, cold-start analysis)

The Five Cost-Optimization Pillars

  1. Right-size instances
  2. Increase elasticity (ASG / endpoint autoscaling)
  3. Choose pricing model (Spot / RI / SP)
  4. Match storage to usage + lifecycle
  5. Measure & monitor continuously

2. Core Concepts Cheat-Sheet

Key Performance Metrics (ML Infrastructure)

Metric Category Examples (SageMaker / general) Why it matters
Utilization CPUUtilization, MemoryUtilization, GPUUtilization, DiskUtilization Right-sizing signal; low utilization = waste
Throughput Invocations, InvocationsPerInstance, Predictions count by RequestMode Capacity planning
Latency ModelLatency, OverheadLatency, ModelLoadingTime (MME), ModelCacheHit User experience + cold-start detection
Availability / Fault 4XX/5XX errors, Invocation4XXErrors, Endpoint status SLA / auto-remediation triggers
Scalability DesiredInstanceCount vs Running, ConcurrentExecutions (Lambda) Autoscaling health

Monitoring & Observability Services (Must-Know Mapping)

Need Primary Service Notes for Exam
Metrics + Alarms + Dashboards Amazon CloudWatch Native SageMaker integration; anomaly detection; Application Insights; ServiceLens
Log search / analytics CloudWatch Logs Insights Interactive queries; no extra infra
Lambda system metrics CloudWatch Lambda Insights Aggregates cold starts, memory, CPU for serverless
Distributed tracing AWS X-Ray Visualize component interactions; latency breakdown
API / user activity audit AWS CloudTrail Every SageMaker API call; log-file integrity validation
Event-driven automation Amazon EventBridge Training job state change, endpoint status, AWS Health, Trusted Advisor findings
Cost visibility Cost Explorer, CUR, Budgets, Cost Optimization Hub Tags required for allocation
Rightsizing recs SageMaker Inference Recommender, Compute Optimizer Instance family + count + cost/performance trade-off

Instance Family Decision Tree (Exam Favorite)

  • Compute-optimized (C-family): CPU-bound training or batch transforms
  • Memory-optimized (R/X-family): Large models / in-memory datasets
  • Accelerated / Inference-optimized (Inf1/Inf2, GPU G/P, Trn): Real-time inference; Inferentia = up to 3× throughput & ~45 % lower cost vs GPU
  • General purpose (M/T): Notebooks, light workloads
  • SageMaker Neo: Compile once → up to 2× faster on target hardware

💡 Tip — Latency troubleshooting order 1. Check ModelLatency vs OverheadLatency in CloudWatch. 2. Benchmark model outside the endpoint. 3. Look for cold-start (first invocation) → pre-warm or use provisioned concurrency / serverless min-capacity. 4. For multi-model endpoints inspect ModelLoadingTime / ModelCacheHit. 5. Consider Neo compilation or Inferentia. 6. Add autoscaling; verify code-level delays.

3. Exercise 1 — CloudWatch + Lambda Observability (Metrics, Logs Insights, Lambda Insights, X-Ray, Dashboard)

💡 Cost Target: < $0.05 total if cleaned up same day. Skills Covered: CloudWatch Logs/alarms/dashboards, Lambda Insights, X-Ray, latency troubleshooting.

Objective

Deploy a tiny Lambda that simulates an ML inference function. Instrument it for full observability, create alarms, a dashboard, and use Logs Insights + X-Ray to diagnose a deliberate latency issue.

Detailed Steps

  1. Create the Lambda (Python 3.12, 128 MB, 10 s timeout): In Lambda console → Create function → Author from scratch → Name: mla-sim-inference. Paste the following code (deliberately adds jitter and an occasional slow path):
import json, time, random, os

def lambda_handler(event, context):
    start = time.time()
    # simulate model load / cold path
    if random.random() < 0.3:
        time.sleep(1.5)          # deliberate slow inference
    else:
        time.sleep(0.05)

    result = {"prediction": random.choice([0,1]), "latency_ms": int((time.time()-start)*1000)}
    print(json.dumps({"metric":"inference", **result}))  # structured log
    return {"statusCode": 200, "body": json.dumps(result)}

  1. Enable X-Ray active tracing: Configuration → Monitoring and operations tools → Edit → Active tracing = ON. Also turn on CloudWatch Lambda Insights (same panel → Enhanced monitoring).

  2. Invoke 20–30 times (console Test button or CLI loop):

for i in $(seq 1 25); do aws lambda invoke --function-name mla-sim-inference out.json &> /dev/null; done
  1. CloudWatch Logs Insights query: Logs → Log groups → /aws/lambda/mla-sim-inference → Insights. Run:
fields @timestamp, latency_ms, prediction
| filter ispresent(latency_ms)
| stats avg(latency_ms), max(latency_ms), count(*) by bin(5m)

Also try:

filter latency_ms > 1000 | sort @timestamp desc

  1. Create a metric filter + alarm: Log group → Metric filters → Create.

  2. Filter pattern: { $.latency_ms > 1000 }

  3. Metric name: SlowInferences, namespace: MLA/Demo, value: 1.

  4. Create alarm: Statistic = Sum, period 1 min, threshold ≥ 1 → SNS email (create new topic, confirm subscription).

  5. Build a CloudWatch Dashboard: Dashboards → Create MLA-4-2-Observability. Add widgets:

  6. Lambda Invocations, Duration (p50/p99), Errors

  7. Custom metric SlowInferences
  8. Lambda Insights widgets (memory, cold starts)

  9. X-Ray service map (via ServiceLens or X-Ray console link)

  10. Inspect X-Ray: X-Ray → Traces / Service map. Identify the slow segment. Note response-time distribution.

  11. Cleanup (Mandatory for cost control): Delete alarm, metric filter, dashboard, SNS topic, Lambda function, and log group.

Expected results: • Logs Insights shows avg/max latency and flags the ~30 % slow invocations. • Alarm fires and you receive an email when a slow inference occurs. • Dashboard visualizes the same metrics in one place. • X-Ray service map shows the Lambda node with elevated response time; trace detail reveals the sleep as the bottleneck. • Lambda Insights surfaces cold-start count and memory utilization.

💡 Cost control notes: Lambda: 25 invocations × 128 MB × ~200 ms = fractions of a cent. CloudWatch: 1 custom metric + 1 alarm ≈ $0 (first 10 alarms free). X-Ray: First 100k traces/month free. Delete everything same day → near-zero bill.

🎯 Exam tip: When a question mentions “troubleshoot latency on a SageMaker endpoint”, the exact analogues of the metrics you just used are ModelLatency vs OverheadLatency. For multi-model endpoints add ModelLoadingTime, ModelDownloadingTime, ModelCacheHit. The remediation path (Neo → Inferentia → autoscaling → pre-warm) is frequently tested.

⚠️ Exam trap: Students confuse CloudWatch Logs Insights (ad-hoc query) with CloudWatch Metrics Insights or with Athena on CUR. Logs Insights works directly on log groups; no S3 export needed. Also, X-Ray is for distributed traces, not for raw log search.

4. Exercise 2 — CloudTrail Trail + Metric Filter for Policy / API Change Detection

💡 Cost Target: < $0.02 (management events free). Skills Covered: Creating CloudTrail trails, CloudWatch metric filters/alarms, auditability, log-file integrity.

Objective

Recreate the pattern of detecting an S3 bucket-policy change (or any SageMaker API call) via CloudTrail → CloudWatch Logs → metric filter → alarm. Also enable log-file integrity validation.

Detailed Steps

  1. Create a dedicated S3 bucket for CloudTrail: mla-ct-logs-<accountid>-<region>. Block all public access. Apply a lifecycle rule: expire objects after 3 days.

  2. Create CloudTrail trail: CloudTrail → Create trail.

  3. Trail name: mla-management-trail

  4. Storage: Use the bucket above
  5. Enable log file validation = Yes (Exam point!)
  6. Management events: Read + Write (Data events = OFF to avoid cost)
  7. CloudWatch Logs: Enable → new log group /aws/cloudtrail/mla

  8. Leave Insights events off.

  9. Generate a detectable event: Create a throw-away S3 bucket mla-demo-data-xxx. Change its bucket policy (Permissions → Bucket policy → paste a simple deny or allow, Save). This produces PutBucketPolicy in CloudTrail within ~1–5 minutes.

  10. Metric filter on the CloudTrail log group: Filter pattern (exact exam pattern):

{ ($.eventSource = "s3.amazonaws.com") && ($.eventName = "PutBucketPolicy") }

Metric: namespace MLA/Security, name S3BucketPolicyChange, value 1.

  1. Alarm: Statistic = Sum, threshold ≥ 1, period 5 min, treat missing as not breaching. Action → SNS topic. Confirm email.

  2. Verify log-file integrity:

aws cloudtrail validate-logs --trail-arn <arn> --start-time <ISO>

You should see “Validation successful” / hash chain intact.

  1. Optional extension (SageMaker training job events): Same trail already captures CreateTrainingJob, StopTrainingJob, etc. You can add a second metric filter for those event names to trigger re-training workflows.

  2. Cleanup: Delete alarm, metric filter, trail (turn off logging first), both S3 buckets (empty first), log group, SNS if unused.

Expected results: • After policy change, CloudTrail event appears in the log group. • Metric filter increments; alarm state = ALARM; email arrives. • validate-logs confirms integrity (hashes of each log file + digest file). • You can see the exact principal, source IP, and request parameters in the event JSON — foundation for audit / re-training triggers.

💡 Cost control: Management events delivered to CloudWatch Logs = free. S3 storage of a few KB of logs for one day = <$0.01. Lifecycle rule + same-day deletion keeps it negligible. Never enable data events for this exercise.

🎯 Exam tip: Classic question: “S3 bucket policy accidentally changed and broke the CI/CD artifact download. How do you get notified?” Answer path = CloudTrail → CW Logs → metric filter on PutBucketPolicy → alarm → SNS. Bonus: CodeDeploy auto-rollback on alarm.

⚠️ Exam trap: CloudTrail log-file integrity ≠ object lock / S3 versioning. Integrity uses SHA-256 digests delivered hourly; it proves logs were not tampered with after delivery. Also, CloudTrail is regional for trails but can be multi-region; management events are free, data events are not.

5. Exercise 3 — EventBridge Rules for Automation (Health, Trusted Advisor, SageMaker state changes)

💡 Cost Target: ~$0 Skills Covered: EventBridge, SNS, Lambda remediation, monitoring infrastructure events, low-utilization termination pattern.

Objective

Build two zero-cost event-driven patterns:

(A) AWS Health / EC2 maintenance notification to a team SNS topic

(B) Trusted Advisor low-utilization finding → Lambda (simulate terminate)

Detailed Steps

  1. SNS topic for the team: Create topic mla-ops-alerts, subscribe your email, confirm.

  2. Rule A — AWS Health events (EC2): EventBridge → Rules → Create rule.

  3. Name: mla-ec2-health

  4. Event source: AWS events
  5. Event pattern:
{
  "source": ["aws.health"],
  "detail-type": ["AWS Health Event"],
  "detail": {
    "service": ["EC2"]
  }
}

  • Target: SNS topic mla-ops-alerts.

  • Rule B — Trusted Advisor low-utilization → Lambda: First create a tiny Lambda mla-ta-remediator:

import json

def lambda_handler(event, context):
    print("TA finding received:", json.dumps(event))
    # In production: parse resource IDs and call ec2.terminate_instances
    return {"status": "logged"}

Give it basic execution role only. EventBridge rule pattern:

{
  "source": ["aws.trustedadvisor"],
  "detail-type": ["Trusted Advisor Check Item Refresh Notification"],
  "detail": {
    "status": ["WARN", "ERROR"],
    "check-name": ["Low Utilization Amazon EC2 Instances"]
  }
}

Target = the Lambda function.

  1. Rule C (Optional) — SageMaker training job state change: Event pattern:
{
  "source": ["aws.sagemaker"],
  "detail-type": ["SageMaker Training Job State Change"],
  "detail": {
    "TrainingJobStatus": ["Completed", "Failed"]
  }
}

Target: SNS or a Lambda that could start a batch-transform / evaluation step.

  1. Test: Use "EventBridge → Rules → your rule → Send test event" (custom payload matching the pattern) to force the target to fire. Confirm Lambda CloudWatch log and SNS email.

  2. Cleanup: Delete rules, Lambda, SNS topic/subscriptions.

Expected results: Test events produce SNS emails and Lambda log entries. You understand the exact event patterns for Health, Trusted Advisor, and SageMaker job/endpoint state changes.

💡 Cost control: EventBridge custom events and AWS-service events = free for this volume. Lambda test invokes = fractions of a cent.

🎯 Exam tip: “Large Hadoop cluster on 100+ EC2 nodes — automate AWS Health maintenance notifications.” Solution = EventBridge rule on aws.health + EC2 + SNS topic. Second classic: “Terminate low-utilization EC2” = Trusted Advisor check → EventBridge → Lambda.

⚠️ Exam trap: Do not choose CloudWatch Events (old name) as a distractor if EventBridge is present — they are the same service, but the exam prefers “EventBridge”. Trusted Advisor low-utilization EC2 check is available in the free tier of checks.

6. Exercise 4 — Cost Governance: Tagging Strategy, Budgets, Cost Explorer, Trusted Advisor, Cost Optimization Hub

💡 Cost Target: $0 (all free services) Skills Covered: Tagging, Cost Explorer, Budgets, Trusted Advisor, cost allocation, quotas.

Objective

Implement the full cost-monitoring preparation path: enforce a tagging strategy, activate cost-allocation tags, create a budget with alerts, explore Cost Explorer reports, and review Trusted Advisor / Cost Optimization Hub recommendations.

Detailed Steps

  1. Define and apply a tagging strategy: Set mandatory tags:

  2. Project = mla-c01-study

  3. Environment = dev
  4. Owner = your-alias

  5. CostCenter = ml-platform Apply these tags to your resources. In Billing → Cost allocation tags → activate user-defined tags.

  6. Create an AWS Budget: Billing → Budgets → Create budget.

  7. Cost budget, Monthly, Budget amount: $5 (or $1).

  8. Scope: Filter by tag Project=mla-c01-study.

  9. Alerts: 50% actual, 80% actual, 100% forecasted → email.

  10. Cost Explorer walk-through:

  11. Group by: Service, then by Tag (Project).
  12. Filter last 7 days. Save report: "MLA-Study-Spend".

  13. Note difference vs CUR: Cost Explorer = interactive high-level; CUR = hourly line-item raw data delivered to S3 (queryable via Athena/QuickSight).

  14. Trusted Advisor cost checks: Open Trusted Advisor → Cost Optimization category. Review:

  15. Low Utilization Amazon EC2 Instances

  16. Underutilized EBS Volumes / Idle Load Balancers
  17. Unassociated Elastic IP addresses

  18. Amazon RDS Idle DB Instances

  19. Cost Optimization Hub: Billing → Cost Optimization Hub. Observe aggregated rightsizing, idle, and purchasing-option recommendations across accounts/regions.

  20. Optional — Billing alarm via CloudWatch: CloudWatch → Billing → Create alarm on EstimatedCharges metric (threshold $5). Requires enabling billing alerts in Account settings (us-east-1 only).

  21. Cleanup: Budgets can stay ($0); remove tags if desired.

Expected results: All demo resources carry consistent tags. Budget appears in console. Cost Explorer can filter by the tags you activated. You can articulate when to use Explorer vs CUR vs Budgets vs Cost Optimization Hub vs Trusted Advisor.

💡 Cost control: Budgets, Cost Explorer, Trusted Advisor (core checks), Cost Optimization Hub = $0.

🎯 Exam tip: “How do you allocate ML training costs back to business units?” → Mandatory tagging strategy + activate cost-allocation tags + Cost Explorer / CUR filtered by those tags. “High-level interactive report” = Cost Explorer; “hourly breakdown by resource tag for Athena” = CUR.

⚠️ Exam trap: Tags are case-sensitive. Cost-allocation tags must be explicitly activated in the Billing console; merely tagging a resource is not enough for them to show up in Cost Explorer.

7. Exercise 5 — Rightsizing, Purchasing Options, Inference Recommender & Compute Optimizer (Console + Decision Drills)

💡 Cost Target: $0–0.50 (only if you launch a brief serverless endpoint) Skills Covered: Instance families, Inference Recommender, Compute Optimizer, Spot/RI/Savings Plans, provisioned concurrency, service quotas.

Objective

Internalize the decision framework for instance selection and purchasing options without leaving expensive resources running. Perform console exploration and a lightweight optional serverless-inference load test.

Detailed Steps

  1. Instance family mapping drill: | Family | ML Sweet-Spot | | --- | --- | | c5 / c6i / c7g | CPU-bound batch transform / feature engineering | | r5 / r6i / r7g | Large feature stores / in-memory data | | g4dn / g5 / p3 / p4 | GPU training & inference | | inf1 / inf2 | High-throughput, low-cost inference (Inferentia) | | trn1 | Large model training (Trainium) | | ml.t3 / ml.m5 | Notebooks, light processing |

  2. Purchasing-option decision tree (Memorize):

  3. Interruptible batch training / HPO / batch transform → Spot (cheapest)
  4. Steady-state real-time endpoints, cannot tolerate interruption, want flexibility across EC2/Fargate/Lambda/SageMaker → Compute Savings Plans or SageMaker Savings Plans
  5. Known steady EC2 only → EC2 Instance Savings Plans or Reserved Instances
  6. Short experiment / unpredictable → On-Demand

  7. Need capacity guarantee in a specific AZ → Capacity Reservations (can be combined with SP/RI)

  8. Compute Optimizer exploration: Compute Optimizer console → view findings. Understand the three recommendation categories: Over-provisioned, Under-provisioned, Optimized.

  9. SageMaker Inference Recommender: Inference Recommender benchmarks your model on multiple instance types and produces a price-performance Pareto front (expected latency, cost per 1k inferences, max TPS).

  10. Service quotas & provisioned concurrency awareness: Service Quotas console → search "SageMaker" and "Lambda". Note soft limits: max endpoint instances, max training jobs, Lambda concurrent executions. For Lambda-based inference: Provisioned Concurrency eliminates cold starts; for SageMaker Serverless set MinCapacity.

  11. Autoscaling mental model for endpoints: Target-tracking on SageMakerVariantInvocationsPerInstance or custom metric; step scaling; scheduled scaling for known peaks.

  12. Cleanup: Delete any serverless endpoint, model, or endpoint config within minutes of creation.

Expected results: You can instantly map a workload description to an instance family and a purchasing option. You know where to find Inference Recommender and Compute Optimizer recommendations and what inputs they need.

🎯 Exam tip: • “Batch job that can handle interruptions and must minimize cost” → Spot. • “Cannot handle interruptions, want max savings and freedom to move between EC2, Fargate, Lambda” → Compute Savings Plan. • “Best instance type + config for my model at lowest cost” → SageMaker Inference Recommender. • “Right-size existing EC2/Lambda” → Compute Optimizer.

⚠️ Exam trap: Inferentia (inf1/inf2) is not a GPU; it is a custom ASIC. Neo is a compiler, not an instance type. Savings Plans are not the same as Reserved Instances (SPs are more flexible). Provisioned Concurrency is a Lambda/SageMaker Serverless concept; classic real-time endpoints use autoscaling of instance count instead.

8. Exam Traps & Tips Master List

Common Exam Traps

⚠️ Metrics Mix-Up: ModelLatency = time inside the model container. OverheadLatency = time outside (overhead of the platform). High Overhead + low Model often = cold start or network; high Model = model/code issue.

⚠️ Wrong Pricing Pick: Spot for stateful real-time endpoints = wrong (interruptions). RI when you need Lambda flexibility = wrong (use Compute Savings Plan).

⚠️ Tag Activation: Tagging resources ≠ cost allocation. You must activate user-defined cost-allocation tags in Billing console.

⚠️ CloudTrail vs CloudWatch: CloudTrail = API activity audit (who did what). CloudWatch = performance metrics/logs/alarms. Both are often used together (Trail → Logs → filter → alarm).

⚠️ Data Events Cost: Enabling CloudTrail data events on busy S3/Lambda can generate large bills. Management events are free.

⚠️ Idle SageMaker Endpoints: Real-time endpoints bill while InService even with zero traffic. Use Serverless, Async inference, or delete when idle.

High-Yield Tip Sheet

  • Always pair monitoring with automation: EventBridge/Lambda/SNS for remediation.
  • Log-file integrity validation on CloudTrail proves non-repudiation for audits.
  • Inference Recommender first, then Neo, then Inferentia for latency/cost on inference.
  • Five cost pillars order: Right-size → Elasticity → Pricing model → Storage lifecycle → Continuous measurement.
  • Dashboards: CloudWatch for ops metrics; QuickSight for business/cost CUR data.
  • Capacity issues: Check service quotas before blaming autoscaling; provisioned concurrency for cold-start SLAs.
  • Trusted Advisor + EventBridge + Lambda = Classic “auto-terminate low-utilization EC2” pattern.

9. Quick-Reference Tables

Tool → Use-Case Mapping (Memorize)

Scenario Tool(s)
Endpoint latency spike CloudWatch ModelLatency/OverheadLatency + X-Ray + Logs Insights
Detect who changed a bucket policy / IAM role CloudTrail + metric filter + alarm
Training job finished → kick off next step EventBridge (SageMaker state change) → Lambda/Step Functions
Notify team of upcoming EC2 maintenance EventBridge (AWS Health) → SNS
Kill idle EC2 automatically Trusted Advisor → EventBridge → Lambda
Choose best instance for model SageMaker Inference Recommender
Right-size already-running EC2/Lambda Compute Optimizer / Cost Optimization Hub
Allocate costs to teams Tags + Cost Explorer / CUR
Cap monthly ML spend AWS Budgets (+ optional Budget Actions)
Interactive cost dashboard for execs Cost Explorer or QuickSight on CUR
Eliminate cold starts Lambda Provisioned Concurrency / SageMaker Serverless MinCapacity / pre-warm traffic
Cheapest interruptible batch Spot Instances (+ checkpointing)
Flexible long-term discount SageMaker Savings Plans or Compute Savings Plans

Cleanup Checklist (Run after every study session)

  • [ ] Lambda functions deleted
  • [ ] CloudWatch alarms, dashboards, metric filters deleted
  • [ ] CloudTrail trail stopped & deleted; temporary S3 buckets emptied & deleted
  • [ ] EventBridge rules deleted
  • [ ] SNS topics/subscriptions deleted
  • [ ] Any SageMaker endpoints, endpoint configs, models deleted
  • [ ] Budgets may remain (free)
  • [ ] Verify Billing → Cost Explorer shows no unexpected running resources

💡 Final mental model for Task 4.2 Observe (CloudWatch + X-Ray + Logs Insights + CloudTrail) → Detect (Alarms, anomaly detection, Trusted Advisor, Health) → Automate (EventBridge → Lambda / SNS / Budgets actions) → Optimize (Right-size with Inference Recommender & Compute Optimizer, pick Spot/SP/RI, tag everything, enforce budgets) → Review continuously.

MLA-C01 Task 4.2 Study Notes · Minimize cost — delete all resources the same day · Good luck on the exam!