Skill 3.2.3: Build and maintain containers for ML and AI workloads
| Question | Answer & Explanation |
|---|---|
| ### Question 1 - Custom Training Container Construction An enterprise ML team needs to train a model using a proprietary algorithm written in C++ that requires custom system-level libraries not available in any AWS-managed SageMaker Deep Learning Container (DLC). The team must package this algorithm into a custom Docker container to run on SageMaker training instances. To minimize operational overhead and ensure compatibility with SageMaker's training environment, what is the most effective approach? Which ONE solution meets these requirements? - A) Use AWS Lambda to package the C++ algorithm and trigger it via Amazon EventBridge. - B) Build a custom Docker container from a base Ubuntu image and install the sagemaker-training toolkit to interface with SageMaker APIs.- C) Launch an Amazon EC2 instance, install the libraries, create an AMI, and use the AMI for SageMaker training. - D) Extend the closest pre-built PyTorch DLC image by adding the C++ libraries via a Dockerfile. |
Correct Answer: B Correct Option Analysis: - B is Correct: When an ML algorithm requires completely custom system libraries or frameworks not supported by SageMaker DLCs, the recommended approach is to build a custom Docker container from a base OS image (like Ubuntu) and install the sagemaker-training toolkit. This toolkit handles the necessary environment variables, data channels, and communication with SageMaker training APIs.Distractor Analysis: - A is Incorrect: AWS Lambda has strict size and execution time limits and is not designed for long-running, custom C++ ML training jobs on GPU instances. - C is Incorrect: SageMaker training jobs run on containerized environments (Docker) and do not natively support running directly from custom EC2 AMIs. - D is Incorrect: While extending a DLC is a best practice for adding small dependencies, using a large PyTorch DLC as a base for a proprietary C++ algorithm introduces unnecessary overhead and potential dependency conflicts when a completely custom environment is required. > Exam Tip: Use sagemaker-training for custom training containers and sagemaker-inference for custom inference containers when building from scratch. |
| ### Question 2 - Inference Container API Endpoints A data science team is deploying a custom inference container to an Amazon SageMaker real-time endpoint. The container must expose an API for SageMaker to perform health checks and serve predictions. The team needs to ensure the endpoint passes its initial health checks without failing during the endpoint creation process. Which ONE configuration must be implemented inside the custom container? - A) Expose an HTTPS server on port 443 that responds to /ping and /invocations.- B) Expose an HTTP server on port 8080 that responds to /ping and /invocations.- C) Implement a gRPC server on port 8081 for /ping and /invocations.- D) Configure a cron job inside the container to ping the SageMaker API every 30 seconds. |
Correct Answer: B Correct Option Analysis: - B is Correct: SageMaker real-time endpoints interact with inference containers by sending HTTP requests to specific endpoints. The container must run an HTTP server listening on port 8080, responding to /ping (for health checks) and /invocations (for inference requests).Distractor Analysis: - A is Incorrect: While HTTPS is used for secure communication, the container itself must listen on port 8080 for HTTP traffic, which SageMaker manages externally. - C is Incorrect: SageMaker's standard real-time inference containers expect HTTP REST APIs on port 8080, not gRPC on 8081 (though gRPC is supported via SageMaker InvokeEndpoint with specific configurations, the foundational health checks still rely on HTTP 8080). - D is Incorrect: The SageMaker platform initiates health checks to the container; the container does not proactively ping SageMaker. > Exam Tip: Custom SageMaker inference containers must expose an HTTP server on port 8080 and implement /ping (health) and /invocations (predictions) endpoints. |
| ### Question 3 - ECR Image Versioning and Compliance A company is building a machine learning application that uses a custom Docker container. The ML engineering team needs to store, manage, and deploy this container image across multiple AWS accounts with strict compliance requirements. Security policies mandate that image tags cannot be overwritten once pushed, ensuring that a specific tag always points to the exact same image build. Which ONE AWS service and feature should be used to meet these requirements with the least operational overhead? - A) Amazon ECR with image tag mutability set to IMMUTABLE.- B) Amazon S3 with object lock enabled to store the Docker image tarball. - C) AWS CodeArtifact to manage the Docker image artifacts. - D) Amazon ECR with lifecycle policies configured to delete untagged images. |
Correct Answer: A Correct Option Analysis: - A is Correct: Amazon Elastic Container Registry (ECR) is the native AWS service for storing Docker images. By setting the image tag mutability to IMMUTABLE, the registry prevents any user from pushing an image with a tag that already exists, enforcing strict version control and compliance.Distractor Analysis: - B is Incorrect: While S3 Object Lock prevents object deletion/overwrite, ECR is the purpose-built container registry for pulling/pushing Docker images to SageMaker, offering native integration and authentication. - C is Incorrect: AWS CodeArtifact is designed for managing software development dependencies (like npm, pip, Maven), not Docker container images. - D is Incorrect: Lifecycle policies automate the deletion of old images but do not prevent the overwriting of existing tags. > Exam Tip: Enable ECR image tag immutability to prevent overwriting tags, ensuring reproducible deployments and meeting strict compliance mandates. |
| ### Question 4 - Optimizing Container Size and Model Artifacts An AI engineering team is building an inference container for a large language model (LLM). The model weights are 15 GB in size. The endpoint is experiencing slow startup times during scale-out events, and the team must optimize the container image size. The solution must reduce the image pull time and adhere to AWS best practices for ML workloads. Which ONE approach minimizes the container size and meets the requirements? - A) Bake the 15 GB model weights directly into the Docker image to ensure they are available immediately upon startup. - B) Store the model weights in an Amazon S3 bucket and download them during the container startup sequence using the entrypoint script. - C) Use Amazon EFS to mount the model weights as a file system directly into the SageMaker endpoint container. - D) Package the model weights into a Docker image using multi-stage builds to compress them into a smaller layer. |
Correct Answer: B Correct Option Analysis: - B is Correct: AWS best practice dictates keeping Docker images small by storing large model weights in Amazon S3. The container's entrypoint script should download the weights from S3 to the local instance storage upon startup. SageMaker natively handles this via model_data in the model creation API, which mounts S3 artifacts to /opt/ml/model/.Distractor Analysis: - A is Incorrect: Baking 15 GB of weights directly into the Docker image massively inflates the image size, leading to extremely slow ECR push/pull times and slower endpoint auto-scaling. - C is Incorrect: While EFS can be used, it introduces higher operational overhead and cost compared to native S3 model artifact downloading, which SageMaker handles automatically. - D is Incorrect: Multi-stage builds help remove unnecessary build tools, but they do not meaningfully compress 15 GB of pre-trained model weights. > Exam Tip: Never bake large model weights into the Docker image. Store model artifacts in S3 and let SageMaker mount them, or use an entrypoint script to fetch them. |
| ### Question 5 - Cross-Region Container Deployment A global enterprise needs to deploy a custom SageMaker inference container to multiple AWS Regions to achieve the lowest latency for global users. The container images are hosted in Amazon ECR in the primary region ( us-east-1).The deployment must minimize cross-region data transfer costs and image pull latency during endpoint deployments in remote regions. Which ONE configuration is the most operationally efficient solution? - A) Configure an AWS Transfer Family server in each region to replicate the Docker images. - B) Manually push the Docker image to a separate ECR repository in each remote region during the CI/CD pipeline. - C) Enable Amazon ECR cross-region replication to automatically copy the container images to the remote regions. - D) Store the container images in an S3 bucket with Cross-Region Replication (CRR) enabled and pull them using a custom startup script. |
Correct Answer: C Correct Option Analysis: - C is Correct: Amazon ECR supports cross-region replication, which automatically and asynchronously copies pushed images to specified destination regions. This provides local ECR registries in each region, minimizing pull latency and cross-region data transfer costs during endpoint creation. Distractor Analysis: - A is Incorrect: AWS Transfer Family is for SFTP/FTPS file transfers and is not a relevant mechanism for container registry replication. - B is Incorrect: Manually pushing to multiple registries in a CI/CD pipeline introduces high operational overhead and is prone to failure or inconsistency. - D is Incorrect: SageMaker endpoints natively pull from ECR; using S3 and custom scripts to retrieve container images is an anti-pattern and adds unneeded operational overhead. > Exam Tip: Use Amazon ECR cross-region replication to automatically copy container images to target regions, reducing latency for multi-region SageMaker deployments. |
| ### Question 6 - CI/CD Compute for Container Builds An ML engineering team has adopted a CI/CD pipeline to build and deploy custom containers for SageMaker training jobs. The pipeline currently builds the Docker image, pushes it to Amazon ECR, and then triggers a SageMaker training job using the SageMaker Python SDK. The team wants to minimize the operational overhead of managing the compute infrastructure that builds the Docker images. Which ONE AWS service is purpose-built for this task? - A) Amazon ECS with AWS Fargate. - B) AWS CodeBuild. - C) Amazon EC2 instances running Jenkins. - D) AWS Glue Docker jobs. |
Correct Answer: B Correct Option Analysis: - B is Correct: AWS CodeBuild is a fully managed continuous integration service that compiles source code, runs tests, and produces software packages (including Docker images) ready for deployment. It scales automatically and eliminates the need to provision and manage build servers. Distractor Analysis: - A is Incorrect: While you could run Docker builds on ECS/Fargate, it requires managing task definitions and orchestration, making it less operationally efficient than CodeBuild for CI/CD. - C is Incorrect: Running Jenkins on EC2 requires patching, scaling, and maintaining the underlying operating systems, violating the least operational overhead constraint. - D is Incorrect: AWS Glue is a data integration service for ETL workloads, not a general-purpose CI/CD build environment for Docker images. > Exam Tip: Use AWS CodeBuild as the serverless compute engine in your CI/CD pipeline to build and push Docker images to ECR. |
| ### Question 7 - Extending Pre-built Containers A team is using a pre-built SageMaker PyTorch Deep Learning Container (DLC) for training. However, they need to add a specific Python library and a custom system package that are not included in the DLC. The team wants to maintain the least operational overhead while leveraging the existing AWS-optimized environment. Which ONE approach is recommended? - A) Build a completely new Docker container from scratch using the sagemaker-training toolkit.- B) Use the SageMaker Python SDK's dependencies parameter to pass a requirements.txt file.- C) Create a new Dockerfile that uses the SageMaker PyTorch DLC as the base image ( FROM) and installs the required system and Python packages.- D) Attach an AWS Lambda function to the SageMaker training job to execute the installation commands at runtime. |
Correct Answer: C Correct Option Analysis: - C is Correct: When you only need to add a few packages to an existing AWS-optimized environment, the best practice is to extend the pre-built DLC using a Dockerfile. Using FROM <dlc-image> and running RUN apt-get and pip install keeps the AWS optimizations intact while adding your custom dependencies.Distractor Analysis: - A is Incorrect: Building from scratch is only necessary when you have highly custom frameworks or system requirements that fundamentally conflict with the DLCs. It introduces unnecessary overhead here. - B is Incorrect: The dependencies parameter in the SDK can handle Python packages, but it cannot install system-level packages (like custom C libraries) needed by the environment.- D is Incorrect: AWS Lambda cannot be attached to a SageMaker training job to install dependencies; the environment must be ready when the container starts. > Exam Tip: Extend pre-built SageMaker Deep Learning Containers (DLCs) using a Dockerfile when you only need to add custom Python libraries or system packages. |
| ### Question 8 - Multi-Container Inference Pipelines An enterprise ML team needs to deploy an inference pipeline where incoming requests are preprocessed by a Scikit-Learn container, passed to an XGBoost container for prediction, and then post-processed by a custom Python container. The team wants to deploy this pipeline as a single SageMaker real-time endpoint with the least operational overhead and lowest latency. How should the containers be configured? - A) Deploy three separate SageMaker endpoints and use Amazon API Gateway to chain the requests between them. - B) Use SageMaker Inference Pipelines to attach multiple containers to a single endpoint, defining the order of execution. - C) Combine the Scikit-Learn, XGBoost, and Python logic into a single massive Docker container. - D) Use AWS Step Functions to orchestrate the calls to three individual SageMaker endpoints. |
Correct Answer: B Correct Option Analysis: - B is Correct: SageMaker Inference Pipelines natively support chaining multiple containers (up to 15) into a single real-time endpoint. The output of the first container is automatically passed as the input to the second, minimizing latency and operational overhead. Distractor Analysis: - A is Incorrect: Deploying three separate endpoints introduces network latency between each hop and requires managing three distinct endpoint lifecycles. - C is Incorrect: Combining distinct frameworks (Scikit-Learn, XGBoost, custom Python) into one massive container breaks modularity, is hard to maintain, and prevents independent scaling/updating of components. - D is Incorrect: Step Functions adds orchestration latency and cost compared to native SageMaker Inference Pipelines. > Exam Tip: Use SageMaker Inference Pipelines to deploy a sequence of containers (e.g., preprocessing, prediction, postprocessing) behind a single endpoint URL. |
| ### Question 9 - ECR Lifecycle Management A company runs hundreds of data science experiments daily, resulting in numerous untagged and outdated Docker images in their Amazon ECR repositories. The DevOps team needs to automate the cleanup of these images to reduce storage costs. The solution must enforce the removal of untagged images and retain only the last 10 tagged images, with zero manual intervention. Which ONE solution requires the least operational overhead? - A) Write a Python script using Boto3 to list and delete old images, triggered by an Amazon EventBridge schedule. - B) Configure an Amazon ECR lifecycle policy to automatically remove untagged images and retain only the last 10 images. - C) Use AWS Systems Manager Run Command to execute Docker CLI commands on a build server to prune images. - D) Manually delete old images through the AWS Management Console every week. |
Correct Answer: B Correct Option Analysis: - B is Correct: Amazon ECR lifecycle policies are a built-in, native feature that automates the cleanup of images. You can create rules to remove untagged images or apply count-based rules (e.g., keep the last 10 tagged images) with zero ongoing operational overhead. Distractor Analysis: - A is Incorrect: Writing custom Boto3 scripts requires development time, maintenance, and debugging, which is more overhead than a native feature. - C is Incorrect: Systems Manager Run Command is for managing EC2 instances, not for managing images within an ECR registry. - D is Incorrect: Manual deletion does not meet the requirement for zero manual intervention and will not scale. > Exam Tip: Use Amazon ECR lifecycle policies to automatically manage image cleanup (e.g., removing untagged images or keeping only the N most recent images) to optimize storage costs. |
| ### Question 10 - Vulnerability Scanning for Containers An enterprise has a strict compliance requirement that all container images used for ML inference must be scanned for software vulnerabilities before being deployed to a SageMaker endpoint. The team uses Amazon ECR to store their images. The deployment must block if any critical vulnerabilities are found, minimizing operational overhead. Which ONE built-in AWS feature should the team utilize? - A) AWS Inspector to scan the running EC2 instances backing the SageMaker endpoint. - B) Enable Amazon ECR image scanning, configured to scan on push, and integrate the scan results into the CI/CD pipeline to block deployment. - C) Amazon GuardDuty to monitor malicious activity in the ECR registry. - D) AWS Security Hub to aggregate alerts from custom vulnerability scripts run on the build server. |
Correct Answer: B Correct Option Analysis: - B is Correct: Amazon ECR offers built-in image scanning (using either the native Clair-based scanner or Amazon Inspector Enhanced Scanning). By enabling "scan on push," every image is automatically evaluated. The CI/CD pipeline can retrieve the scan findings and halt deployment if critical vulnerabilities are detected. Distractor Analysis: - A is Incorrect: AWS Inspector can scan EC2 instances, but SageMaker endpoints are managed services where you do not have direct access to the underlying EC2 OS to install Inspector agents. ECR scanning checks the container image itself before deployment. - C is Incorrect: Amazon GuardDuty is for threat detection (e.g., unusual API calls, compromised instances), not for static vulnerability scanning of Docker images. - D is Incorrect: Writing custom vulnerability scripts violates the least operational overhead constraint, especially when ECR provides native scanning capabilities. > Exam Tip: Enable Amazon ECR "scan on push" to automatically identify vulnerabilities in container images before they are deployed to SageMaker. |