Skip to content

MLA_C01 Study Notes: Task 3.1 – Select Deployment Infrastructure Based on Existing Architecture and Requirements

Task Overview

This task focuses on choosing the right deployment infrastructure for ML models after training. You must evaluate existing architecture, requirements (latency, throughput, payload size, cost, traffic patterns), and constraints to select:

  • Inference mode (real-time, serverless, asynchronous, batch)
  • Compute (CPU/GPU/Inferentia/Trainium, instance size/family)
  • Containers (AWS-provided deep learning containers vs custom)
  • Hosting target (SageMaker endpoints, multi-model/multi-container, serial inference pipelines, ECS/EKS/Kubernetes, Lambda, edge via Neo)
  • Orchestration and best practices (versioning, rollback/guardrails, A/B testing via production variants, SageMaker Pipelines vs Airflow, Model Registry)

Core decision framework for the exam:

  1. Traffic pattern & latency: Persistent low-latency → real-time endpoint; spiky/idle → serverless; large payloads/long-running → asynchronous; entire dataset offline → batch transform.
  2. Cost vs performance: Right-size instances; prefer Inferentia/Inf1 + Neo for high-throughput inference; multi-model endpoints to pack models; serverless to eliminate idle costs.
  3. Architecture fit: Match containers/frameworks (TensorFlow, PyTorch, scikit-learn, MXNet, XGBoost, etc.); use serial pipelines for pre/post-processing; multi-container when frameworks differ.
  4. Ops & scale: Deployment guardrails + Model Registry for safe rollout/rollback; SageMaker Pipelines (or Airflow) for orchestration; horizontal scaling preferred when framework supports it.
  5. Edge/cloud continuum: SageMaker Neo to compile once, run anywhere (cloud or edge).

Key services: Amazon SageMaker (endpoints, Inference Recommender, Neo, Model Registry, Pipelines, multi-model/multi-container/serial pipelines), AWS Inferentia/Inf1, AWS Batch, Amazon EMR (Spark MLLib vs SageMaker Spark Library), ECS/EKS, Lambda.

Key Concepts Condensed from Exam Guide & Official Review

  • Inference types (memorize decision tree):
  • Real-time: persistent endpoint, one prediction at a time, low latency.
  • Serverless: idle-friendly, cold starts OK, automatic scaling to zero.
  • Asynchronous: up to 1 GB payload, long processing, near real-time, queues requests.
  • Batch Transform: offline, entire dataset, no endpoint.
  • A/B testing: Prefer single SageMaker endpoint with production variants + traffic weights (load-balanced, low effort) over multiple API Gateway endpoints.
  • Compute selection: Training often needs GPU/TPU + horizontal scaling (framework must support); inference can use CPU, GPU, or Inferentia. Increase mini-batch + learning rate when scaling. Vertical scaling = bigger instance; horizontal = more instances.
  • Containers: Use AWS DLCs when possible; custom only when needed. Multi-model endpoint (same framework, many models) vs multi-container (different frameworks) vs serial inference pipeline (pre/post-processing on same instance → lower latency).
  • Optimization: SageMaker Neo (compile once); Inference Recommender (load-test + instance recommendation); Inf1 + Neo for cost/performance.
  • Orchestration & MLOps: SageMaker Pipelines / Model Registry / deployment guardrails for CI/CD, versioning, blue/green or canary rollouts. EMR + Spark MLLib (in-cluster) vs SageMaker Spark Library (offload training/endpoints). AWS Batch (Fargate or EC2, Spot) for batch jobs.
  • Frameworks: Know TensorFlow, PyTorch, scikit-learn, MXNet, XGBoost; Keras/Gluon are higher-level, not core frameworks.

Exam Tips & Common Traps

Tips:

  • Always match inference option to payload size, latency tolerance, and traffic pattern first, then cost.
  • Single endpoint + production variants is the lowest-effort A/B solution.
  • Serverless eliminates idle charges but has cold-start latency.
  • Neo + Inf1 is the go-to for “optimize cost and performance for inference.”
  • Multi-model endpoints save money when you have many similar models; serial pipelines reduce latency by co-locating containers.
  • Prefer horizontal scaling when the algorithm/framework supports distribution; adjust mini-batch and learning rate.
  • Clean up endpoints and transform jobs immediately—exam scenarios often test cost awareness.

Traps:

  • Choosing real-time endpoint for spiky/idle traffic (wastes money) or serverless when cold starts are unacceptable.
  • Using multiple separate endpoints + API Gateway for A/B instead of production variants.
  • Deploying GPU instances for simple CPU-friendly models (or vice-versa).
  • Forgetting that batch transform does not create a persistent endpoint.
  • Assuming every framework scales horizontally without code changes.
  • Leaving endpoints running (cost bomb) or forgetting rollback/guardrails.
  • Confusing multi-model (many models, one container type) with multi-container (different containers).
  • Using EMR cluster sizing for training when SageMaker Spark Library lets you offload.

Hands-on Exercises (Minimized Cost)

Cost-control rules used in all exercises (assume no free tier):

  • Use only CPU instances (ml.t2.medium, ml.m5.large or smaller where possible) or serverless.
  • Create → test with 1–5 invocations → delete immediately (endpoints billed per wall-clock minute).
  • Prefer serverless and batch transform (pay only for duration).
  • Run everything from AWS CloudShell (free) or local machine with AWS CLI v2 + boto3 + sagemaker SDK. No persistent Notebook/Studio instances.
  • Use tiny scikit-learn or XGBoost models on synthetic/small public data (Iris or Boston-housing equivalent).
  • Set budgets/alerts in AWS Billing; delete S3 artifacts after.
  • Total expected cost for all three exercises if cleaned up promptly: < $1–2.

Exercise 1: Deploy the Same Model to All Four SageMaker Inference Options and Compare Trade-offs

Goal: Internalize when to choose real-time vs serverless vs asynchronous vs batch. Directly maps to “Methods to serve ML models in real time and in batches” and “Model and endpoint requirements.”

What you will do:

  1. Train a tiny scikit-learn model (or use a pre-trained one) and register it.
  2. Deploy to each inference type.
  3. Invoke once or twice, capture latency/cost characteristics, then tear down.

Detailed Steps:

  1. Open AWS CloudShell (or local terminal with credentials). Install packages if needed:
    pip install sagemaker scikit-learn boto3 pandas joblib -q

  2. Create a working directory and a simple training script (train.py) that fits a RandomForestClassifier (or LogisticRegression) on Iris data, dumps model.joblib, and uploads to S3.

  3. Use the SageMaker Python SDK to:
  4. Create an SKLearn model object pointing at your model.tar.gz in S3 (or train via SKLearn estimator on ml.m5.large for < 5 min).
  5. Real-time: model.deploy(initial_instance_count=1, instance_type="ml.t2.medium", endpoint_name="ts-realtime-ex"). Invoke with predictor.predict(...). Note start-up time and per-second billing.
  6. Immediately: predictor.delete_endpoint(); predictor.delete_model().
  7. Serverless: Deploy with serverless_inference_config={"MemorySizeInMB": 2048, "MaxConcurrency": 5}. Invoke a few times (observe possible cold start). Delete endpoint.
  8. Asynchronous: Create async endpoint with async_inference_config (output S3 path). Send a larger payload (simulate near-1 GB by repeating data). Check output in S3. Delete.
  9. Batch Transform: Create transformer with instance_type="ml.m5.large", instance_count=1, run on a small CSV dataset in S3, wait for job completion, download predictions. No endpoint remains.
  10. (Optional cheap extra) Run SageMaker Inference Recommender on the model for 1–2 instance types only, then delete.

Expected Results:

  • Real-time: sub-second latency after warm-up, continuous billing while “InService.”
  • Serverless: scales to zero (no idle cost), possible 1–5 s cold start on first invoke.
  • Async: accepts large payload, returns immediately with output location; processing happens in background.
  • Batch: job runs to completion, predictions land in S3, zero persistent infrastructure.
  • You can quantify: serverless cheapest for sporadic traffic; real-time for steady low-latency; async for large/long jobs; batch for offline.

Relation to Exam Traps/Tips:

  • Trap avoided: picking real-time for idle workloads. Tip reinforced: “workloads with idle periods → serverless”; “large payload + long processing → async”; “entire dataset → batch.”
  • You practiced immediate cleanup = cost control (exam loves this mindset).
  • Decision tree becomes muscle memory.

Exercise 2: A/B Testing with Production Variants on a Single Endpoint + Safe Rollback

Goal: Practice deployment best practices (versioning, traffic shifting, rollback) and the recommended A/B pattern. Maps to “Deployment best practices,” “Choosing model deployment strategies,” and the A/B question in the transcript.

Detailed Steps:

  1. Train two slightly different models (e.g., RandomForest with different n_estimators or one LogisticRegression vs RandomForest). Package each as model.tar.gz in S3. Create two SageMaker Model objects (model-a, model-b).

  2. Create a single endpoint configuration that registers both as production variants:
    Variant-A weight 0.9, Variant-B weight 0.1 (or 50/50). Use ml.t2.medium, 1 instance total.

  3. Deploy the endpoint (ts-ab-test-ex).

  4. Invoke 20–50 times; capture which variant answered (via custom attributes or CloudWatch).
  5. Shift traffic: update endpoint weights to 0/1 (all to winner) using update_endpoint or deployment guardrails (canary/blue-green if you want extra credit—still on the same small instance).
  6. Demonstrate rollback: update back to previous weights or previous model version.
  7. Delete endpoint and models immediately.

Expected Results:

  • Single endpoint serves both variants; SageMaker load-balances according to weights.
  • You see traffic percentages reflected in invocation counts.
  • Traffic shift and rollback succeed without creating new endpoints or API Gateway resources.
  • CloudWatch metrics show per-variant latency/error rates.

Relation to Exam Traps/Tips:

  • Directly answers the transcript question: “single SageMaker endpoint + traffic percentage” is the low-effort solution vs multiple API Gateway endpoints.
  • Reinforces versioning + guardrails for production safety.
  • Trap avoided: “publish many API endpoints and monitor them all.”
  • Cost stays tiny because only one small instance runs for a few minutes.

Exercise 3: Multi-Model Endpoint vs Serial Inference Pipeline + Container Choice + Neo Compile (Optional)

Goal: Practice selecting multi-model/multi-container deployments, containers, and edge optimization. Maps to “Selecting multi-model or multi-container,” “Choosing appropriate containers,” “Methods to optimize models on edge devices (SageMaker Neo).”

Detailed Steps:

  1. Multi-model endpoint (same framework):
  2. Upload 2–3 tiny scikit-learn models (different names) into one S3 prefix.
  3. Create a multi-model endpoint using the SKLearn container on ml.t2.medium.
  4. Invoke by specifying TargetModel="model1.tar.gz" etc.
  5. Delete immediately.
  6. Serial inference pipeline (pre/post-processing):
  7. Build two containers (or reuse SKLearn + a simple processing container). Chain them in an Inference Pipeline Model.
  8. Deploy to one small real-time endpoint. Invoke and observe that pre-processing → model → post-processing all occur on the same instance (lower latency). Delete.
  9. Custom vs provided container: Note that you used the AWS-provided SKLearn DLC. (If time, build a minimal custom Dockerfile, push to ECR, and deploy once—still delete fast.)
  10. Neo (cheap compile-only):
  11. Take one of the models, create a Neo compilation job targeting ml_c5 or imx (edge) or inf1.
  12. Wait for success (compile only—do not deploy to Inf1 unless you want the extra ~$0.50). Download the compiled artifact.
  13. Observe the Neo-optimized model is ready for cloud or edge.

Expected Results:

  • Multi-model: one endpoint hosts many models; you invoke by name; cost is shared.
  • Serial pipeline: single invocation runs the full chain with low latency (containers co-located).
  • Neo job succeeds and produces a compiled model artifact usable on Inferentia or edge devices.
  • You see the difference between “many models, one container type” (multi-model) vs “different containers” (multi-container or pipeline).

Relation to Exam Traps/Tips:

  • Tip: multi-model endpoints optimize cost when models share framework; serial pipelines optimize latency.
  • Trap avoided: deploying every model on its own endpoint.
  • Neo directly addresses “train once, run anywhere” and Inf1 cost/performance questions.
  • Container choice: provided DLCs are preferred unless custom code forces otherwise.
  • Reinforces compute selection (CPU sufficient here; you would choose Inf1 + Neo for high-QPS inference).

Additional Low-Cost Practice Ideas (Optional, still < $1)

  • Create a 2-step SageMaker Pipeline (train → register in Model Registry) and execute once.
  • Submit a simple AWS Batch job (Fargate, 0.25 vCPU) that runs a scikit-learn batch inference script.
  • Compare EMR Serverless (Spark) vs calling SageMaker from Spark via the SageMaker Spark Library (conceptual or tiny job).

Final Exam-Day Checklist for Task 3.1

  • Draw the inference-option decision tree on scrap paper.
  • For any scenario ask: latency? payload? traffic shape? cost sensitivity? framework? edge?
  • Prefer managed SageMaker features (variants, multi-model, Neo, guardrails, Pipelines) over rolling your own on ECS/EKS/Lambda unless the question forces it.
  • Always mention cleanup, right-sizing, and Spot/serverless where cost is a factor.
  • Version everything and plan rollback.

After completing the three exercises you will have muscle memory for every major decision in Task 3.1 while having spent only pocket-change on AWS. Delete all endpoints, models, S3 prefixes, and compilation jobs when finished, and set a billing alarm. Good luck!