Skip to content

Skill 2.2.2: Configure SageMaker AI script mode with supported frameworks for simplicity and performance

Question 1 - SageMaker Script Mode Entry Point

A financial services company is developing a custom fraud detection model using PyTorch that requires a non-standard transformer architecture with a custom loss function. The ML engineering team has written the training script locally and verified it works on a small sample dataset.

The team must run this training job on Amazon SageMaker AI while adhering to strict security policies that prohibit installing unapproved third-party container images. The team also wants to minimize operational overhead by not managing container registries or patching frameworks manually.

Which approach should the ML engineer use to run this custom training job? (Select ONE)

A) Package the PyTorch script into a custom Docker image, push it to Amazon ECR, and configure a SageMaker Estimator to use the image URI.

B) Use the SageMaker PyTorch Estimator with the entry_point parameter pointing to the training script, and let SageMaker pull the AWS-managed Deep Learning Container.

C) Rewrite the custom architecture using the SageMaker built-in XGBoost algorithm to eliminate the need for custom code.

D) Convert the PyTorch script to ONNX format and upload it to Amazon SageMaker JumpStart for managed execution.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: SageMaker Script Mode allows you to supply a custom training script (entry_point) that runs inside an AWS-maintained Deep Learning Container (DLC) for PyTorch. The team retains full control over the model architecture, loss function, and data loading logic, while AWS handles the framework image, dependency management, and security patching—eliminating the need for custom container management.

Distractor Analysis

  • A is Incorrect: Building and maintaining a custom Docker image in Amazon ECR introduces significant operational overhead (registry management, image patching, security scanning). This violates the "minimize operational overhead" constraint and is unnecessary when Script Mode supports PyTorch natively.
  • C is Incorrect: XGBoost is a gradient-boosted tree algorithm designed for tabular data. It cannot represent a custom transformer architecture or custom loss function, so rewriting would discard the team's existing work and fail to satisfy the modeling requirements.
  • D is Incorrect: ONNX is an interchange format for model serialization, not a mechanism for running custom training scripts. SageMaker JumpStart hosts pre-trained models and fine-tuning templates; it does not execute arbitrary ONNX training pipelines.

Key Takeaway: SageMaker Script Mode is the default choice for custom training code—it provides full control over the training loop while AWS maintains the framework container.

Question 2 - Script Mode Hyperparameter Passing

A retail analytics team is migrating an on-premises TensorFlow training pipeline to Amazon SageMaker AI. The existing training script uses argparse to accept command-line hyperparameters such as --learning-rate, --batch-size, and --num-layers.

The team wants to run hyperparameter tuning jobs using SageMaker Automatic Model Tuning (AMT) without modifying the existing argument parsing logic in the script.

How should the ML engineer configure the SageMaker TensorFlow Estimator to pass hyperparameters correctly? (Select ONE)

A) Store hyperparameters in a separate JSON file in Amazon S3 and reference it using the hyperparameters parameter as an S3 URI.

B) Pass a Python dictionary to the hyperparameters parameter in the Estimator; SageMaker automatically injects them as command-line arguments to the script.

C) Set the hyperparameters as environment variables in the Estimator's environment parameter and modify the script to read them using os.environ.

D) Embed the hyperparameter values directly into the training script before each tuning job iteration using a preprocessing Lambda function.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: In SageMaker Script Mode, the hyperparameters dictionary passed to the Estimator is automatically serialized and injected as command-line arguments (e.g., --learning-rate 0.001) when the container executes the script. This is fully compatible with the argparse module, allowing the team to run AMT without modifying the script's existing argument parsing.

Distractor Analysis

  • A is Incorrect: The hyperparameters parameter expects a flat dictionary of key-value pairs, not an S3 URI. While you could load a hyperparameter JSON from S3 within the script, this adds unnecessary I/O overhead and a custom loading step.
  • C is Incorrect: Environment variables are an alternative mechanism for parameter injection, but they are not the native SageMaker Script Mode approach for hyperparameters. The team would need to modify the script to read from os.environ instead of argparse, violating the "without modifying" constraint.
  • D is Incorrect: Preprocessing hyperparameters via Lambda before each tuning job adds unnecessary complexity and an extra moving part. SageMaker AMT handles hyperparameter variation natively across training job iterations.

Key Takeaway: SageMaker Script Mode automatically passes hyperparameters from the Estimator as command-line arguments, making argparse-based scripts work without modification.

Question 3 - Framework Container Selection

A healthcare startup is building a medical imaging classification model. The data science team has written a training script that uses Python 3.9 and PyTorch 2.0 with a custom data augmentation library that requires torchvision and opencv-python-headless.

The team needs to run this on Amazon SageMaker AI as a managed training job. The security team requires that all dependencies come from an approved internal PyPI mirror, not from the public internet.

What should the ML engineer configure in the SageMaker PyTorch Estimator to meet these requirements? (Select ONE)

A) Use the default SageMaker PyTorch DLC and add a requirements.txt file listing the dependencies; SageMaker will install them from the public PyPI.

B) Build a custom container extending the SageMaker PyTorch DLC, configure the internal PyPI mirror as the index URL, and register the image in Amazon ECR.

C) Use the SageMaker PyTorch DLC and set the source_dir parameter to local library source code; dependencies will be resolved from the internal mirror automatically.

D) Use Amazon SageMaker Autopilot, which automatically builds a custom container with dependencies from the internal mirror.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: When Script Mode's default DLCs install dependencies from the public PyPI (via requirements.txt), this violates the security requirement for the internal mirror. Building a custom container that extends (FROM) the AWS-maintained PyTorch DLC and configuring the internal PyPI index (--index-url) allows full control over dependency resolution while still leveraging the AWS-maintained base image.

Distractor Analysis

  • A is Incorrect: The default DLC's requirements.txt installation pulls packages from the public PyPI repository, directly violating the security team's requirement for the internal mirror.
  • C is Incorrect: The source_dir parameter bundles the training script's local source code, but it does not control how third-party dependencies are installed. Dependency resolution still defaults to public repositories.
  • D is Incorrect: SageMaker Autopilot is an AutoML service for automated model training and tuning, not a container-building service. It does not provide configuration for specifying custom dependency mirrors.

Key Takeaway: While Script Mode covers most training needs, custom dependency sources (internal mirrors, private packages) require a lightweight custom container that extends the AWS DLC—this is still simpler than a fully custom image.

Question 4 - Distributed Training Configuration

An e-commerce company is training a large Hugging Face transformer model (6 billion parameters) for product recommendation ranking. The model fits on a single GPU, but training on the full dataset takes 6 days with the current single-instance setup.

The team wants to reduce training time using Amazon SageMaker AI while maintaining Script Mode with their custom Hugging Face training script. They need a solution that minimizes code changes.

Which TWO configurations should the ML engineer implement? (Select TWO)

A) Use the SageMaker Hugging Face Estimator and increase the instance_count parameter; add the SageMaker Distributed Data Parallel library to the training script for gradient synchronization.

B) Split the model across multiple GPUs using the SageMaker Model Parallel library to reduce computation time per batch.

C) Use the SageMaker Hugging Face Estimator with multiple instances and enable distribution={"smdistributed": {"dataparallel": {"enabled": True}}}.

D) Configure Amazon SageMaker Automatic Model Tuning with a broader hyperparameter range to find faster-converging settings.

E) Reduce the epochs parameter by 50% and increase learning_rate to compensate, which halves the total training time.

Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: Data parallelism is the correct strategy when the model fits on a single GPU but the dataset is large. By increasing instance_count and using SageMaker's Distributed Data Parallel (SMDDP) library, batches are split across instances, and gradients are synchronized efficiently, reducing total training time.
  • C is Correct: The SageMaker Hugging Face Estimator supports the native distribution parameter with smdistributed.dataparallel enabled. This configuration automatically sets up the cluster, gradient synchronization, and optimized communication without requiring massive code refactoring.

Distractor Analysis

  • B is Incorrect: Model parallelism is designed for models that are too large to fit on a single device. Since the model fits on a single GPU, splitting the model across GPUs would introduce communication overhead without solving the underlying problem (large dataset throughput).
  • D is Incorrect: Hyperparameter tuning seeks better-performing settings, not faster execution per training run. AMT would actually increase total compute time by running many training jobs.
  • E is Incorrect: Arbitrarily reducing epochs and increasing learning rate risks underfitting or divergence, degrading model quality. This is not a principled approach to reducing training time.

Key Takeaway: Use data parallelism when the model fits on one device but the dataset is large; use model parallelism when the model itself exceeds single-device memory.

Question 5 - Script Mode with Custom Metrics

A logistics company is training a Scikit-learn model using SageMaker Script Mode for delivery time prediction. The training script currently prints the following to stdout during training:

Epoch 1 - training loss: 2.456
Epoch 1 - validation loss: 2.789

The team wants to monitor the validation loss in real time on Amazon CloudWatch and use it as the objective metric for SageMaker Automatic Model Tuning.

What must the ML engineer add to the training script to make this work? (Select ONE)

A) No code changes are needed; SageMaker automatically parses all stdout output and creates CloudWatch metrics.

B) Add a metric_definitions parameter to the Estimator specifying a regex pattern that parses validation loss from the log output.

C) Modify the script to call boto3.cloudwatch.put_metric_data() directly from within the training loop.

D) Set the metric_name parameter in the Estimator to validation_loss and SageMaker will automatically capture it.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: SageMaker Script Mode supports custom metric extraction using metric_definitions in the Estimator. Each metric definition includes a Name (metric name in CloudWatch) and a Regex (regular expression pattern that matches the value in the training logs). This enables both real-time CloudWatch visualization and use as the AMT objective metric.

Distractor Analysis

  • A is Incorrect: SageMaker does not automatically parse arbitrary custom metrics from stdout. You must explicitly define regex patterns via metric_definitions for SageMaker to extract and forward metric values to CloudWatch as searchable time-series data.
  • C is Incorrect: While you could call the CloudWatch API directly from the training script, this requires boto3 initialization, error handling, and metric publishing logic. Using metric_definitions in the Estimator is the native, simpler approach.
  • D is Incorrect: There is no metric_name parameter in the SageMaker Estimator API. Metrics must be extracted from log output using metric_definitions or published programmatically.

Key Takeaway: Use metric_definitions with regex patterns in the Estimator to surface custom training metrics in CloudWatch and drive AMT objective functions.

Question 6 - Script Mode vs Built-in Algorithm

A manufacturing company needs to build a predictive maintenance model to classify equipment failure modes from sensor readings. The team has limited ML expertise but extensive domain knowledge about the failure patterns.

They need to choose between using the SageMaker built-in Linear Learner algorithm or Script Mode with a custom Scikit-learn model. The team wants the simplest approach that allows them to use their domain knowledge for feature engineering while minimizing the complexity of the ML pipeline.

Which solution best meets these requirements? (Select ONE)

A) Use the built-in Linear Learner algorithm, which automatically handles feature engineering and requires only raw sensor data.

B) Use Script Mode with Scikit-learn, where the training script applies domain-specific feature engineering before fitting a model.

C) Use Amazon SageMaker Autopilot to automatically engineer features and select the best model from the data.

D) Use Amazon SageMaker JumpStart with a pre-built industrial model that already includes domain-specific feature engineering.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: Script Mode with Scikit-learn allows the team to encode their domain knowledge directly into the feature engineering pipeline (e.g., rolling averages, frequency-domain features, industrial threshold crossovers). The built-in algorithms do not support custom feature transforms, and Autopilot's automatic feature engineering may not capture domain-specific patterns that engineers know to be relevant.

Distractor Analysis

  • A is Incorrect: Built-in Linear Learner performs basic data normalization and works on tabular data, but it does not perform domain-specific feature engineering. The team's expertise would go unused if they relied on raw features alone.
  • C is Incorrect: SageMaker Autopilot automates model selection and basic feature engineering, but it operates generically without the team's domain insight. It may produce a useful baseline but does not prioritize the specific failure patterns that domain experts know matter.
  • D is Incorrect: SageMaker JumpStart provides pre-trained models for common tasks (e.g., computer vision, NLP), but it does not provide domain-specific industrial models for sensor-based failure mode classification out of the box. The team would still need to customize.

Key Takeaway: Choose Script Mode when domain-specific feature engineering is essential—it gives full control over the data transformation pipeline while AWS manages the container.

Question 7 - Source Directory and Dependencies

A media streaming company has a modular ML codebase organized as follows:

ml_project/
├── train.py
├── utils/
│   ├── data_loader.py
│   ├── model_utils.py
│   └── metrics.py
└── requirements.txt

The team wants to run this training job using SageMaker Script Mode with the PyTorch framework. The train.py file imports functions from the utils module.

How should the ML engineer configure the Estimator to ensure all local code is available during training? (Select ONE)

A) Set entry_point="train.py" and upload the entire ml_project/ directory to Amazon S3 manually; SageMaker will detect the structure automatically.

B) Set entry_point="train.py" and source_dir="ml_project/" in the Estimator; SageMaker will bundle and upload the directory to the training instance.

C) Use a Dockerfile to copy the local code into a custom container, push it to Amazon ECR, and reference the image in the Estimator.

D) Convert the utils module into a Python package, publish it to a private PyPI repository, and list it in requirements.txt.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: The source_dir parameter in the SageMaker Estimator specifies the local directory containing the training code, including submodules and helper files. SageMaker automatically bundles this directory, uploads it to Amazon S3 (or uses the specified s3_prefix), and extracts it to the training instance so that relative imports from the utils module work correctly.

Distractor Analysis

  • A is Incorrect: Manual S3 upload bypasses SageMaker's built-in code packaging. While technically possible, SageMaker does not "auto-detect" the structure from a raw S3 bucket. You would need to manage code versioning and file paths manually, introducing unnecessary complexity.
  • C is Incorrect: A custom container is overkill for a standard PyTorch project with standard dependencies. Script Mode with source_dir handles local code bundling natively without any Docker work.
  • D is Incorrect: Publishing utility code as a Python package is a valid engineering approach for shared libraries, but it adds a packaging and versioning step that is unnecessary when source_dir bundles the code directly.

Key Takeaway: Use entry_point for the main script and source_dir for the folder containing all helper modules—SageMaker packages and deploys both automatically.

Question 8 - Framework Version and Region Availability

A global financial institution operates in multiple AWS regions and must maintain consistent training environments across all regions for compliance. The team uses SageMaker Script Mode with TensorFlow and requires a specific TensorFlow version (2.11.0) with GPU support.

The team discovers that the prebuilt SageMaker TensorFlow DLC with version 2.11.0 is not available in one of their required regions (e.g., ap-southeast-3).

What should the ML engineer do to ensure consistent training across all regions? (Select ONE)

A) Use the nearest available TensorFlow version in the unavailable region and document the difference in the compliance report.

B) Build a custom Docker image based on the TensorFlow 2.11.0 DLC from an available region, push it to Amazon ECR in the target region, and use it in the Estimator.

C) Switch the entire pipeline to SageMaker built-in algorithms, which are available uniformly across all regions.

D) Use Amazon SageMaker JumpStart, which provides the same framework versions across all regions.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: When a specific DLC version is unavailable in a region, the solution is to build a custom image based on the DLC from an available region and push it to Amazon ECR in the target region. The Estimator can then reference this image URI, ensuring the exact same TensorFlow version (2.11.0) is used across all regions regardless of DLC regional availability.

Distractor Analysis

  • A is Incorrect: Using a different TensorFlow version in one region creates inconsistent training behavior (numerical differences, performance variations) and violates the compliance requirement for consistency. Documenting a discrepancy does not fix it.
  • C is Incorrect: Switching to built-in algorithms fundamentally changes the modeling approach and may not support the team's custom TensorFlow architectures or training loops. It also does not address the framework version requirement.
  • D is Incorrect: SageMaker JumpStart provides pre-trained models and fine-tuning solutions, not arbitrary framework version containers. Availability of specific framework versions is governed by the same DLC regional availability.

Key Takeaway: For regional DLC gaps, extend the available DLC into a custom image and replicate it via Amazon ECR cross-region replication—maintain consistency without rewriting the pipeline.

Question 9 - Script Mode for Inference Pipeline

A real estate platform has trained a model using SageMaker Script Mode with PyTorch for property price prediction. The inference script must apply the same feature transformations used during training (e.g., log transform on square footage, one-hot encoding on property type) before invoking the model.

The team wants to deploy this as a SageMaker Endpoint with minimal operational overhead. They already have the training script (train.py) and a separate inference script (inference.py) that implements model_fn, input_fn, predict_fn, and output_fn.

Which configuration should the ML engineer use? (Select ONE)

A) Deploy the model using the SageMaker PyTorch Model class with entry_point="inference.py" and source_dir pointing to the inference code directory.

B) Create a Lambda function that performs feature engineering and then calls the SageMaker endpoint; deploy the model without an inference script.

C) Modify the training script to include the inference functions, and deploy using the training script as the entry point.

D) Export the model to ONNX format and deploy it using SageMaker Neo, which automatically applies feature transformations.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: The SageMaker PyTorch Model class supports Script Mode at inference time. By specifying entry_point="inference.py" and source_dir for any helper files, the inference container loads your custom functions (model_fn to load the model, input_fn to preprocess input, predict_fn for the forward pass, output_fn for post-processing). This keeps feature engineering logic inside the endpoint, minimizing operational overhead.

Distractor Analysis

  • B is Incorrect: A Lambda function for feature engineering adds an extra hop in the inference path, increasing latency, introducing a separate codebase to maintain, and potentially creating versioning mismatches between training and inference transformations.
  • C is Incorrect: While embedding inference functions in the training script can work, it violates the separation of concerns and makes the training script harder to maintain. A separate inference.py is cleaner and supports the same Script Mode deployment pattern.
  • D is Incorrect: ONNX/Neo handles model optimization for edge devices and different hardware targets, but it does not automatically apply custom feature transformations. The feature logic would still need to reside in some serving component.

Key Takeaway: SageMaker Script Mode extends to inference—deploy a custom inference script with model_fn, input_fn, predict_fn, and output_fn to keep preprocessing logic co-located with the model.

Question 10 - Debugging Script Mode Training Failures

A telecommunications company is running a SageMaker Script Mode training job with a TensorFlow script. The job fails 10 minutes into training with the following CloudWatch log error:

ModuleNotFoundError: No module named 'tensorflow_datasets'

The team's requirements.txt file lists tensorflow-datasets==4.8.0, but the installation appears to have failed silently during container startup.

What should the ML engineer do to resolve this issue? (Select ONE)

A) Reduce the training instance size to allow more stable pip installations during container initialization.

B) Check the CloudWatch logs for the pip installation output during container startup; verify the package name and version are correct in requirements.txt and the package is available in the configured PyPI index.

C) Install the missing module from within the training script using a subprocess call before importing tensorflow_datasets.

D) Switch to a custom Docker image with all dependencies pre-installed, bypassing the requirements.txt mechanism entirely.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: When Script Mode encounters a ModuleNotFoundError, the first diagnostic step is to examine the container startup logs (in CloudWatch) to see if pip install -r requirements.txt succeeded. Common root causes include: typo in the package name (tensorflow_datasets vs tensorflow-datasets with a hyphen/underscore mismatch), incorrect version string, network restrictions preventing PyPI access, or dependency conflicts. Verifying the package name/version in requirements.txt and checking the pip output directly addresses the likely failure modes.

Distractor Analysis

  • A is Incorrect: Training instance size has no bearing on pip installation stability during container startup. Dependencies are installed before the training script executes, and instance size does not affect package resolution.
  • C is Incorrect: Using subprocess to install packages from within the training script is an anti-pattern—it bypasses SageMaker's dependency management, may install packages in the wrong environment, and creates a race condition between installation and import. Dependencies should be resolved via requirements.txt before the script begins.
  • D is Incorrect: Building a custom Docker image is a heavier solution than needed. While it would resolve the issue, it introduces container maintenance overhead. First diagnose the simpler requirements.txt issue; only escalate to custom containers if the package requires system-level dependencies not available in the DLC.

Key Takeaway: For dependency failures in Script Mode, inspect the container startup logs in CloudWatch—requirements.txt issues (typos, version conflicts, network blocks) are the most common cause and easy to fix without changing the container.

Summary of Skill 2.2.2: Configure SageMaker AI Script Mode

Concept Key Points
What is Script Mode? Supply your own training script (entry_point) that runs inside an AWS-maintained Deep Learning Container (DLC)
When to use Script Mode Custom model architectures, custom loss functions, domain-specific feature engineering, unsupported frameworks
Parameter passing hyperparameters dict → auto-injected as command-line args (compatible with argparse)
Code packaging entry_point points to main script; source_dir bundles supporting modules; requirements.txt installs dependencies
Distributed training Use distribution parameter: smdistributed.dataparallel for data parallelism, smdistributed.modelparallel for model parallelism
Metric monitoring metric_definitions with regex patterns extracts custom metrics from logs → CloudWatch + AMT objective
Inference deployment Same Script Mode pattern: inference.py with model_fn, input_fn, predict_fn, output_fn
When to build custom container Dependency requires non-PyPI source, DLC version unavailable in region, system-level packages needed