3.2

Study Notes: MLA_C01 Task 3.2 – Create and script infrastructure based on existing architecture and requirements

Core Focus
This task emphasizes scripting and automating ML infrastructure (via IaC) so solutions remain maintainable, scalable, and cost-effective. Key areas: on-demand vs. provisioned capacity, SageMaker endpoint auto-scaling policies and metrics, IaC trade-offs (CloudFormation vs. CDK), container workflows (ECR/ECS/EKS/BYOC), VPC-secured SageMaker endpoints, and SageMaker SDK deployment patterns. Always prefer pay-per-use or short-lived resources and delete everything immediately after validation to keep costs near zero.

Key Knowledge Summary

  • On-demand vs. provisioned: Provisioned = fixed SageMaker real-time endpoint instances (you pay while they run). On-demand = Serverless Inference (SageMaker manages scaling; you pay only for inference duration + data processed) or Spot for training. Use provisioned + auto-scaling when you need consistent low latency; use Serverless when traffic is spiky/unpredictable.

  • Scaling policies comparison:
    – Target tracking: set a CloudWatch metric target (e.g., InvocationsPerInstance = 100, CPUUtilization = 70 %); SageMaker adds/removes instances automatically. Easiest and recommended starting point.
    – Step scaling: you own the CloudWatch alarms and define step adjustments (more control, more management overhead).
    – Scheduled scaling: cron-like scale-out/in for known peaks (e.g., business hours).
    AWS recommends combining them (target + scheduled) for resilience.

  • Scale-up vs. scale-out: Changing instance type (scale-up) on an existing endpoint is not supported in-place. Create a new production variant with the larger/accelerator instance (G5, Inf1, etc.), then shift traffic via UpdateEndpointWeightsAndCapacities. Accelerator instances give higher throughput per host but can still saturate under sudden surges.

  • IaC trade-offs: CloudFormation = declarative JSON/YAML, native, excellent for simple stacks and cross-stack exports/imports. AWS CDK = imperative (TypeScript/Python), higher-level constructs, better for complex logic and reusable patterns, still synthesizes to CloudFormation. SAM is ideal for pure serverless (Lambda + Step Functions). Use CodeCommit/S3/GitHub + CodePipeline for versioned, automated deployments.
  • Containers: Build once, push to Amazon ECR; consume from ECS/EKS or directly as SageMaker BYOC (bring-your-own-container) images. SageMaker manages the serving stack when you supply a compliant image.
  • VPC networking: Place endpoints/notebooks in private subnets. Create Interface VPC Endpoints (PrivateLink) for com.amazonaws.region.sagemaker.api, com.amazonaws.region.sagemaker.runtime, and S3 Gateway Endpoint so traffic never leaves the AWS network. Avoid broad NACL deny-all-public rules—they break SageMaker control-plane calls.
  • Auto-scaling metrics to choose: InvocationsPerInstance, CPUUtilization, ModelLatency, OverheadLatency, or custom (ExplanationsPerInstance for Clarify). Monitor via CloudWatch dashboards.
  • Cost & maintainability best practices: Serverless Inference or multi-model endpoints + auto-scaling; managed Spot for training; Lambda as a front-end for lightweight routing; script everything in CloudFormation/CDK so environments are disposable and identical.

Exam Traps & Tips (directly tested)

  • Trap: “Just edit the existing endpoint’s instance type.” → Wrong; new variant + traffic shift required.
  • Trap: “Use Inferentia alone for sudden traffic spikes.” → Inferentia improves cost/performance but still needs auto-scaling or Serverless; it can drop requests under extreme load.
  • Trap: “Block all public internet with NACL on the notebook subnet.” → Breaks SageMaker API calls; use VPC endpoints instead.
  • Tip: Combine target-tracking + scheduled scaling; choose InvocationsPerInstance or ModelLatency as the primary metric.
  • Tip: Serverless Inference removes the need to write scaling policies entirely.
  • Tip: CloudFormation/CDK stacks can declare VPC endpoints, endpoint configs, and auto-scaling policies together for repeatable secure deployments.
  • Tip: Multi-model endpoints also support auto-scaling of model replicas.

Hands-on Exercises (cost-minimized)
All exercises use the absolute cheapest options (Serverless Inference, t3/t2 burstable if needed, delete within minutes). Expect total spend << $1 if you clean up promptly. Prerequisites: AWS CLI v2 configured, Python 3 + boto3, IAM role with SageMaker/CloudFormation/ECR/VPC full access (or least-privilege equivalent), and a simple pre-trained model artifact already in S3 (or use a public scikit-learn model tar.gz). Work in a single region (e.g., us-east-1).

Exercise 1 – Script a Serverless SageMaker Endpoint + Target-Tracking Auto-Scaling with CloudFormation (covers IaC, on-demand resources, scaling policies, metrics, SDK-equivalent deployment)
Goal: Automate provisioning, apply a scaling policy, observe metric-driven behavior, then destroy.

Detailed steps:

  1. Create a file sagemaker-serverless-endpoint.yaml: 
AWSTemplateFormatVersion: '2010-09-09'
Resources:
  ExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal: {Service: sagemaker.amazonaws.com}
            Action: sts:AssumeRole
      ManagedPolicyArns: [arn:aws:iam::aws:policy/AmazonSageMakerFullAccess]
  Model:
    Type: AWS::SageMaker::Model
    Properties:
      ExecutionRoleArn: !GetAtt ExecutionRole.Arn
      PrimaryContainer:
        Image: 683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-scikit-learn:1.0-1-cpu-py3   # public image
        ModelDataUrl: s3://my-bucket/model.tar.gz   # replace with your tiny model
  EndpointConfig:
    Type: AWS::SageMaker::EndpointConfig
    Properties:
      ProductionVariants:
        - VariantName: AllTraffic
          ModelName: !GetAtt Model.ModelName
          ServerlessConfig:
            MemorySizeInMB: 1024
            MaxConcurrency: 5
  Endpoint:
    Type: AWS::SageMaker::Endpoint
    Properties:
      EndpointConfigName: !GetAtt EndpointConfig.EndpointConfigName
  ScalingTarget:
    Type: AWS::ApplicationAutoScaling::ScalableTarget
    Properties:
      MaxCapacity: 5
      MinCapacity: 1
      ResourceId: !Sub endpoint/${Endpoint.EndpointName}/variant/AllTraffic
      RoleARN: !Sub arn:aws:iam::${AWS::AccountId}:role/aws-service-role/sagemaker.application-autoscaling.amazonaws.com/AWSServiceRoleForApplicationAutoScaling_SageMakerEndpoint
      ScalableDimension: sagemaker:variant:DesiredInstanceCount
      ServiceNamespace: sagemaker
  ScalingPolicy:
    Type: AWS::ApplicationAutoScaling::ScalingPolicy
    Properties:
      PolicyName: InvocationsTarget
      PolicyType: TargetTrackingScaling
      ScalingTargetId: !Ref ScalingTarget
      TargetTrackingScalingPolicyConfiguration:
        TargetValue: 50.0
        PredefinedMetricSpecification:
          PredefinedMetricType: SageMakerVariantInvocationsPerInstance
  1. Deploy: aws cloudformation create-stack --stack-name mla-ts32-ex1 --template-body file://sagemaker-serverless-endpoint.yaml --capabilities CAPABILITY_NAMED_IAM.
  2. Wait until CREATE_COMPLETE (aws cloudformation wait stack-create-complete ...).
  3. Invoke a few times with boto3 or CLI to generate InvocationsPerInstance metrics: 
import boto3, json
rt = boto3.client('sagemaker-runtime')
for _ in range(20):
    rt.invoke_endpoint(EndpointName='<name>', ContentType='text/csv', Body='1,2,3,4')
  1. In CloudWatch → Metrics → SageMaker → observe InvocationsPerInstance and the scaling activity.
  2. Expected result: Endpoint stays at MinCapacity under light load; if you temporarily raise concurrency/load the scalable target increases (Serverless still caps at MaxConcurrency). Stack events show clean resource creation order.
  3. Cleanup (mandatory): aws cloudformation delete-stack --stack-name mla-ts32-ex1 and confirm deletion.

Relates to exam traps/tips: Demonstrates target-tracking on the exact metric “InvocationsPerInstance”, shows Serverless as the “no-policy-needed” alternative, and proves IaC creates repeatable, versionable infrastructure. You never manually click “create endpoint”.

Exercise 2 – VPC-Secure a SageMaker Endpoint / Notebook and Contrast with NACL Mistake (covers VPC configuration, PrivateLink, security best practices, maintainability)
Goal: Place resources in a private subnet, add Interface Endpoints, prove connectivity, then illustrate the NACL trap—all torn down quickly.

Detailed steps:

  1. Create a minimal VPC (or reuse default and add private subnet): 1 public + 1 private subnet, IGW, NAT (or omit NAT if using only endpoints), route tables. Use console “VPC wizard” → “VPC with public and private subnets” then immediately note the private subnet ID and security group.

  2. Create Interface VPC Endpoints:
    – com.amazonaws.region.sagemaker.api
    – com.amazonaws.region.sagemaker.runtime
    – S3 Gateway Endpoint.
    Attach them to the private subnet/security group that allows 443 from the SageMaker execution role SG.

  3. Create a SageMaker Notebook instance (ml.t3.medium—cheapest) or reuse the Serverless endpoint from Exercise 1, placing it in the private subnet and attaching the VPC config.

  4. From the notebook (or a private EC2) run a simple boto3.client('sagemaker').list_models() and an invoke_endpoint call.
  5. Expected result: Calls succeed with no public internet route; Traffic flows via PrivateLink ENIs. CloudTrail / VPC flow logs show private IPs only.
  6. Trap demonstration (do NOT leave in place): Add a network ACL rule on the private subnet that denies all outbound 0.0.0.0/0. Re-try the boto3 call → it fails with timeout or “unable to connect to SageMaker API”. Remove the deny rule.
  7. Cleanup: Delete notebook/endpoint, delete VPC endpoints, delete VPC/stack. Use a CloudFormation template wrapping the VPC + endpoints for repeatability next time.

Relates to exam traps/tips: Directly reproduces the “NACL blocks SageMaker API” failure mode called out in the official review. Shows the correct PrivateLink pattern that can be embedded in the same CloudFormation/CDK stack used for the endpoint, tying security back to maintainable IaC.

Exercise 3 – Lightweight Container Build + ECR + Optional BYOC Reference (covers containerization concepts, ECR, BYOC readiness)
Goal: Practice the container workflow without running an expensive training job.

Detailed steps:

  1. Locally (or in Cloud9/AWS CloudShell which is free-tier-like): write a 5-line Dockerfile that just echoes “hello” or uses the public sklearn image as base.
  2. aws ecr create-repository --repository-name mla-ts32-demo
  3. Login, build, tag, push: 
aws ecr get-login-password | docker login --username AWS --password-stdin <account>.dkr.ecr.<region>.amazonaws.com
docker build -t mla-ts32-demo .
docker tag mla-ts32-demo:latest <account>.dkr.ecr.<region>.amazonaws.com/mla-ts32-demo:latest
docker push ...
  1. (Optional low-cost validation) Create a SageMaker Model resource pointing at the ECR image URI + a tiny model.tar.gz; do NOT create an endpoint unless you use Serverless and delete in <10 min.
  2. Expected result: Image appears in ECR console; aws ecr describe-images succeeds. You now understand the BYOC contract (ENTRYPOINT must follow SageMaker serving spec).
  3. Cleanup: aws ecr delete-repository --repository-name mla-ts32-demo --force.

Relates to exam tips: Reinforces that containers are first-class for custom runtimes and that the same image can be referenced inside CloudFormation/CDK Model resources. Shows why multi-model endpoints + auto-scaling work cleanly once the image is in ECR.

Additional Quick Practice (zero extra cost)

  • Use the SageMaker Python SDK (sagemaker.ServerlessInferenceConfig + model.deploy()) to achieve the same endpoint as Exercise 1, then immediately predictor.delete_endpoint(). Compare the SDK call sequence with the CloudFormation resources—you will see the identical API surface.
  • In Application Auto Scaling console, experiment with a step-scaling policy versus target-tracking on a (short-lived) provisioned endpoint variant; note the extra alarm management.
  • Add a scheduled scaling action (cron) that scales MaxConcurrency or DesiredInstanceCount at a future minute, observe, then cancel.

Final Cost-Control Checklist (apply after every exercise)

  • aws cloudformation delete-stack / aws sagemaker delete-endpoint / delete-endpoint-config / delete-model
  • aws application-autoscaling deregister-scalable-target
  • Delete any VPC endpoints and the VPC itself
  • Empty and delete any temporary S3 prefixes
  • Set billing alarms for SageMaker, EC2, VPC endpoints at $1

Mastering these three short-lived exercises gives you muscle memory for every knowledge and skill bullet in Task 3.2 and directly surfaces the exam traps around variant traffic shifting, Inferentia limitations, NACL vs. PrivateLink, metric selection, and IaC repeatability. Rehearse the cleanup commands until they are automatic.