AWS Certified AI Practitioner — Task Statement 1.3 Study Notes

Describe the Machine Learning Development Lifecycle

The machine learning (ML) development lifecycle is a repeatable process that starts with a business problem and ends with operating, monitoring, and improving a deployed model.

A typical lifecycle is:

Define the business problem
Determine whether ML is appropriate
Collect and prepare data
Label data and engineer features
Train, tune, and evaluate the model
Deploy the model for inference
Monitor performance, drift, cost, and business outcomes
Retrain, update, or replace the model as required

The lifecycle is iterative. It does not end when a model is deployed.

1. ML Pipeline Versus ML Lifecycle

ML pipeline

An ML pipeline is a sequence of connected processing steps, such as:

Ingesting data
Transforming data
Training a model
Evaluating the model
Deploying the model

ML lifecycle

The ML lifecycle includes the pipeline plus the ongoing activities after deployment:

Monitoring model behavior
Detecting data drift and concept drift
Monitoring bias and data quality
Retraining the model
Managing model versions
Measuring business value
Controlling cost and technical debt

Exam tip

A model is not “finished” after training or deployment. ML systems require continuous evaluation and maintenance because real-world data and business conditions change.

2. Stage 1: Define the Business Goal

Every ML project should begin with a clear business objective rather than starting with a particular algorithm or AWS service.

Key questions

What business problem are we solving?
Is ML the appropriate solution?
What is the expected business value?
How will success be measured?
What are the costs and risks?
What are the consequences of incorrect predictions?
Is enough relevant, high-quality data available?
Can the solution be operated in production?

Formulate the ML problem

Define:

Inputs — What data will the model receive?
Outputs — What should the model predict or generate?
Target variable — What is the model trying to estimate?
Performance metric — How will technical performance be measured?
Business metric — How will business success be measured?

Example

Business goal: Reduce fraudulent credit card transactions.

Input: Transaction amount, location, merchant, time, customer history
Output: Fraud or not fraud
ML task: Binary classification
Technical metric: Recall, precision, F1 score, or AUC
Business metric: Fraud losses avoided, false-decline rate, cost per transaction

Start with the simplest effective solution

Possible approaches should generally be evaluated in increasing order of complexity:

Existing AWS AI service
Existing foundation model or pre-trained model
Fine-tuning or transfer learning
Custom model training
Training a model from scratch

A complex model is not automatically a better business solution.

Cost-benefit analysis

Before progressing, evaluate:

Development cost
Training cost
Inference cost
Data labeling cost
Maintenance and monitoring cost
Security and compliance requirements
Expected business benefit
Cost of incorrect predictions
Scalability requirements

Exam trap

The most technically advanced solution is not necessarily the correct answer. The best solution is usually the one that meets business requirements with the lowest appropriate operational complexity and cost.

3. Sources of Foundation and ML Models

The exam can test how models are obtained and how much customization is required.

Option 1: Fully managed AI services

AWS provides pre-trained AI services for common use cases. You send data to an API and receive a result.

Examples include services for:

Text analysis
Translation
Speech
Image and video analysis
Generative AI

Example: Amazon Comprehend

Amazon Comprehend provides natural language processing capabilities. It also supports custom models such as a custom classifier, where you supply your own labeled examples and categories.

Advantages

Minimal ML expertise required
No need to manage infrastructure
Fastest path to production
Pay-as-you-go pricing
AWS manages the underlying model and service operations

Limitation

A managed service might not achieve the accuracy, customization, latency, or compliance requirements of a particular workload.

Option 2: Foundation models through Amazon Bedrock

Amazon Bedrock provides access to foundation models from AWS and third-party model providers through managed APIs.

You can use foundation models for tasks such as:

Text generation
Summarization
Question answering
Embeddings
Image generation
Conversational applications

You can often customize model behavior through:

Prompt engineering
Retrieval Augmented Generation (RAG)
Fine-tuning, where supported
Custom model import or other Bedrock customization options, depending on the model and feature

Key distinction

Amazon Bedrock is primarily a managed generative AI platform. You generally do not manage the underlying model infrastructure.

Option 3: Pre-trained open or task-specific models

Amazon SageMaker JumpStart provides access to:

Foundation models
Open-source models
Computer vision models
Natural language processing models
Task-specific models

These models are already trained on large datasets and can be used as a starting point.

Benefits

Reduces training time
Reduces compute cost
Requires less labeled data
Accelerates experimentation
Often provides better results than training with limited data from scratch

Option 4: Fine-tuning and transfer learning

Transfer learning uses an existing pre-trained model and adapts it to a new task or domain.

For example:

Start with a general image classification model
Provide organization-specific labeled images
Continue training so it learns the new classification task

This is commonly called fine-tuning, although the exact technical process can vary by model and service.

Advantages

Lower cost than training from scratch
Less training data required
Faster development
Can improve performance for a specific domain

Option 5: Training a model from scratch

Training from scratch requires:

Large amounts of high-quality data
Significant compute resources
Specialized ML expertise
Longer development time
Careful security and compliance controls
More responsibility for model quality, bias, and operational reliability

Exam tip

Training from scratch is usually the most expensive and operationally complex option. It should be selected only when existing models and managed services cannot meet the requirements.

4. Stage 2: Data Collection and Ingestion

Models are only as good as the data used to train and evaluate them.

Identify data requirements

Determine:

What data is needed?
Where is the data generated?
Where is it stored?
Is it structured, semi-structured, or unstructured?
Is it batch or streaming data?
Is it labeled?
Does it contain personally identifiable information (PII)?
Is the data legally and ethically usable?
How frequently will new data arrive?

Batch versus streaming data

Batch data

Data is collected and processed periodically.

Examples:

Daily sales data
Historical customer records
Nightly reporting files

Streaming data

Data is processed continuously or near real time.

Examples:

Application events
IoT sensor data
Financial transactions
Clickstream events

Relevant AWS services can include:

Amazon Kinesis
Amazon Managed Streaming for Apache Kafka (Amazon MSK)
Amazon S3
AWS Glue
Databases and data warehouses

ETL

ETL stands for:

Extract — Obtain data from one or more sources
Transform — Clean, normalize, join, filter, or enrich data
Load — Store the prepared data in a target repository

The process should be repeatable because models are often retrained with new data.

5. Data Preparation

Data preparation can be one of the most time-consuming stages in an ML project.

Common data preparation activities

Removing duplicate records
Handling missing values
Fixing inconsistent schemas
Removing anomalous or invalid records
Normalizing numerical values
Encoding categorical values
Removing or masking PII
Joining multiple datasets
Detecting data quality issues
Transforming raw data into model features

Exploratory data analysis

Exploratory data analysis (EDA) helps teams understand:

Data distributions
Correlations
Outliers
Missing values
Class imbalance
Potential bias
Relationships between features and the target variable

Visualization tools can help data scientists identify issues before training.

6. Data Labeling

Supervised learning requires labeled examples.

Examples of labels:

“Spam” or “not spam”
“Fraud” or “not fraud”
Object bounding boxes in an image
Sentiment categories
Correct answers for document extraction

Accurate labeling can be expensive and time-consuming.

Amazon SageMaker Ground Truth

SageMaker Ground Truth helps create labeled datasets.

It uses active learning, where:

The ML system automatically labels examples it can classify confidently.
More difficult or uncertain examples are sent to human annotators.

Human labeling options include:

Amazon Mechanical Turk
Private workforce
Your own employees or contractors

Exam tip

Ground Truth is associated with creating high-quality labeled training datasets, not with deploying models or monitoring production endpoints.

7. AWS Data Preparation Services

AWS Glue

AWS Glue is a fully managed ETL service.

It can:

Discover data sources
Crawl data
Infer schemas
Generate ETL code
Transform data
Load data into target stores
Work with batch and streaming data sources

AWS Glue Data Catalog

The Data Catalog stores metadata, including:

Data location
Table definitions
Schema
Runtime-related metadata

Important trap

The Glue Data Catalog does not store the actual data. It stores metadata describing the data.

A Glue crawler examines a data source and writes metadata and schema information to the Data Catalog.

AWS Glue DataBrew

AWS Glue DataBrew is a visual, no-code or low-code data preparation tool.

It supports:

Data discovery
Visualization
Cleaning
Normalization
Transformation
Data profiling
Quality rule sets
Reusable transformation recipes

Best fit

Use DataBrew when users need interactive, visual data preparation without writing transformation code.

SageMaker Data Wrangler

SageMaker Data Wrangler provides a visual interface for:

Selecting data
Importing data
Exploring data
Transforming data
Performing feature engineering

It supports many built-in transformations and is intended to simplify the data preparation process for ML.

Amazon SageMaker Canvas

SageMaker Canvas provides a visual, low-code/no-code environment for:

Preparing data
Engineering features
Building models
Analyzing results
Generating predictions

Exam distinction

DataBrew: General visual data preparation
Data Wrangler: ML-focused data selection and transformation
Canvas: Visual ML development and prediction, including data preparation

SageMaker Feature Store

Feature Store is a centralized repository for ML features and their metadata.

It helps teams:

Create and manage features
Discover existing features
Reuse features across projects
Reduce repetitive feature engineering
Maintain consistency between training and inference

A feature is an input characteristic used by a model, such as:

Customer age
Number of transactions
Average order value
Device type

Exam trap

Feature Store stores engineered features and related metadata. It is not the same as the Glue Data Catalog, which stores data-source metadata.

8. Splitting Data into Training, Validation, and Test Sets

A common split is:

80% training
10% validation/evaluation
10% final testing

The exact ratio is not mandatory and depends on the use case and data volume.

Training set

Used to learn model parameters or weights.

Validation set

Used during development to:

Compare models
Tune hyperparameters
Select configurations
Detect overfitting during experimentation

Test set

Used for the final unbiased evaluation before production.

Exam trap

Do not repeatedly tune the model against the final test set. Doing so can cause the test set to influence model selection and make the reported performance overly optimistic.

9. Features and Feature Engineering

A feature is an input variable used by the model.

Feature engineering includes:

Selecting useful input variables
Transforming variables
Combining variables
Encoding categories
Scaling values
Creating derived values

Example:

Raw data: Individual transaction records
Engineered feature: Average transaction value over the last 30 days

Why reduce unnecessary features?

Reducing the feature set can:

Reduce memory requirements
Reduce training time
Reduce inference latency
Reduce compute cost
Reduce noise
Lower the risk of overfitting
Simplify the model

Only features needed for inference should generally be used.

10. Stage 3: Train, Tune, and Evaluate the Model

Model parameters

Parameters, also called weights, are learned during training.

The training algorithm adjusts parameters to reduce the difference between:

The model’s prediction
The expected output

Training is iterative and continues until:

A specified number of iterations is reached
The error falls below a target
The model stops improving
Another stopping condition is met

Hyperparameters

Hyperparameters are configuration values selected before or during training. They are not learned in the same way as model parameters.

Examples:

Learning rate
Number of neural network layers
Number of nodes
Batch size
Number of trees
Regularization settings
Number of training iterations

The best values are usually found through experimentation.

Amazon SageMaker AI training jobs

Amazon SageMaker AI can run training jobs using:

Training data stored in Amazon S3
A selected ML compute instance type
A training algorithm
Hyperparameters
An output location for model artifacts

The algorithm can be provided through:

SageMaker-provided algorithms
Deep learning containers
A custom Docker container stored in Amazon ECR

Training outputs, including model artifacts, are commonly stored in Amazon S3.

Exam tip

Amazon ECR stores container images. Amazon S3 commonly stores training data and model artifacts.

Experimentation

ML development often requires many combinations of:

Datasets
Algorithms
Feature sets
Hyperparameters
Compute configurations

Running multiple training jobs and comparing results is called experimentation.

SageMaker Experiments

SageMaker Experiments helps teams:

Organize training runs
Track inputs and configurations
Compare model versions
Record metrics
Identify the best-performing model
Maintain experiment history

Key distinction

Parameters are learned by the model during training.
Hyperparameters are configured by the practitioner.
Experiments compare different training configurations.

Automatic Model Tuning

Amazon SageMaker Automatic Model Tuning (AMT), also called hyperparameter tuning, runs multiple training jobs to find effective hyperparameter values.

You specify:

The algorithm
Hyperparameter ranges
The objective metric
The goal, such as maximize or minimize
Completion or stopping criteria

For a binary classification model, AMT might optimize:

AUC
F1 score
Accuracy
Precision
Recall

Exam trap

Automatic model tuning does not replace the need to select a suitable algorithm, define an appropriate objective metric, or prepare quality data.

11. Stage 4: Model Deployment and Inference

After a model is trained and evaluated, it must be made available for inference.

The deployment choice depends on:

Latency requirements
Traffic pattern
Payload size
Cost
Availability requirements
Whether the endpoint must remain active
Whether results are needed immediately

Batch inference

Batch inference processes a large set of data at once.

Use it when:

Immediate results are not required
Large datasets must be processed
A persistent endpoint is unnecessary
Cost efficiency is important

Example:

Generate recommendations for all customers overnight
Score the previous day’s transactions
Process a large archive of documents

SageMaker Batch Transform

SageMaker Batch Transform provides offline inference without requiring a persistent real-time endpoint.

Real-time inference

Real-time inference provides immediate responses through an endpoint.

Use it when:

Interactive responses are required
Low latency is important
The application sends requests continuously
The endpoint must be available to receive requests

Examples:

Fraud scoring during checkout
Real-time recommendations
Interactive generative AI applications

Real-time endpoints typically run on continuously available ML compute instances.

Asynchronous inference

Asynchronous inference is useful when:

Requests can be queued
Payloads are large
Processing takes longer
Immediate responses are not required
Traffic is intermittent

SageMaker can scale an asynchronous endpoint down to zero when there are no requests, reducing cost.

Serverless inference

Serverless inference:

Does not require directly managing ML instances
Automatically handles scaling for variable traffic
Is suitable for intermittent workloads
Uses serverless compute characteristics
Charges based on usage and, where applicable, provisioned capacity

It is generally best for workloads with periods of inactivity or unpredictable traffic, subject to service limitations such as model size, startup time, and request duration.

Managed versus self-hosted model APIs

Managed API service

Examples:

Amazon Bedrock API
Amazon Comprehend API
SageMaker hosted endpoint

AWS manages much of the infrastructure, including portions of:

Provisioning
Scaling
Availability
Endpoint operations
Patching and infrastructure maintenance

Self-hosted API

You deploy and operate the inference code and model yourself, commonly in a container.

Possible AWS compute platforms include:

Amazon EC2
Amazon ECS
Amazon EKS
AWS Lambda
AWS Batch

An API can be exposed through:

Amazon API Gateway
An application load balancer
A service-specific endpoint

Self-hosting responsibilities

You may need to manage:

Container images
Patching
Scaling
Health checks
Networking
Security
Deployment updates
Traffic routing
Availability
Capacity planning

Exam tip

If the question emphasizes minimal infrastructure management, consider Bedrock or a managed SageMaker endpoint. If it emphasizes full control over the runtime, custom networking, or deployment environment, self-hosting may be appropriate.

Amazon SageMaker AI inference options

SageMaker AI can deploy model artifacts from Amazon S3 using an inference container from Amazon ECR.

Common options:

Inference type	Best suited for
Batch Transform	Large offline datasets; no persistent endpoint required
Real-time	Low-latency, interactive, sustained traffic
Asynchronous	Large payloads, long processing times, queued requests
Serverless	Intermittent or unpredictable traffic without managing instances

SageMaker endpoints can support automatic scaling.

SageMaker Inference Recommender

Inference Recommender tests configuration options to help select appropriate:

Instance types
Performance settings
Cost/performance configurations

12. Stage 5: Monitoring and Retraining

A deployed model can degrade over time, even if it performed well during testing.

Reasons include:

Data quality changes
Changes in user behavior
Changes in the environment
Changes in business rules
Model bias
Changes in the relationship between inputs and outputs
New types of data not represented during training

Data drift

Data drift occurs when the distribution of input data changes compared with the training data.

Example:

A fraud model was trained using transaction behavior from one year, but customer purchasing patterns change significantly the next year.

Input features may have:

Different averages
Different ranges
Different category frequencies
New or missing categories

Concept drift

Concept drift occurs when the relationship between input variables and the target changes.

Example:

The indicators of fraudulent behavior change because attackers adopt new techniques.

The input data might look similar, but the meaning of the data or the correct prediction has changed.

Exam distinction

Data drift: The input data distribution changes.
Concept drift: The relationship between inputs and target changes.

Both can reduce model quality.

Model monitoring process

A model monitoring solution generally:

Captures production inference data
Compares it with a baseline or training data
Applies monitoring rules
Detects violations or degradation
Sends alerts
Initiates corrective actions
May start retraining

Retraining can be scheduled:

Daily
Weekly
Monthly
When a threshold is exceeded
When a significant event occurs
Through manual approval

Amazon SageMaker Model Monitor

SageMaker Model Monitor can:

Monitor models in production
Capture endpoint data
Compare production data against a baseline
Detect data-quality issues
Detect changes in statistical properties
Apply built-in or custom rules
Display violations in SageMaker Studio
Send results to Amazon CloudWatch

CloudWatch alarms can trigger remediation actions, such as:

Sending notifications
Starting a workflow
Initiating retraining
Rolling back or replacing a model

Important limitation

Monitoring does not automatically guarantee that a model remains accurate. Ground-truth labels may arrive later, and monitoring must be designed to measure the relevant data, model, and business signals.

13. MLOps Fundamentals

MLOps applies software engineering and DevOps practices to the ML lifecycle.

Major MLOps principles

Experimentation

Teams should be able to:

Run repeatable experiments
Compare models
Track configurations
Record metrics
Reproduce successful results

Repeatability

The same source code, data version, configuration, and environment should produce a reproducible result.

Automation

Automate:

Data processing
Training
Evaluation
Testing
Deployment
Monitoring
Retraining
Approval workflows

Scalability

The ML system should scale across:

Data volume
Training workloads
Inference traffic
Number of models
Number of teams and projects

Version control and lineage

Version and track:

Source code
Training data
Validation and test data
Feature definitions
Model artifacts
Hyperparameters
Container images
Infrastructure
Experiment results
Deployment configuration

This enables auditability and rollback.

Production readiness

Before deployment, assess:

Model performance
Latency
Throughput
Availability
Cost
Security
Privacy
Bias
Explainability requirements
Monitoring
Rollback capability
Retraining process

Technical debt

Technical debt is the future cost created by shortcuts or poorly managed complexity.

ML technical debt can result from:

Manual training processes
Untracked data versions
Inconsistent features
Undocumented experiments
Missing monitoring
Hard-coded infrastructure
Unmanaged dependencies
Lack of rollback procedures

MLOps reduces technical debt by making workflows standardized, automated, documented, and observable.

14. AWS Services Supporting MLOps

SageMaker Pipelines

SageMaker Pipelines orchestrates ML workflows.

A pipeline can include:

Data processing
Training
Evaluation
Conditional model approval
Deployment
Batch inference
Monitoring
Retraining

It supports:

Reproducible workflows
Conditional branches
Artifact lineage
Integration with SageMaker jobs
Visualization in SageMaker Studio

Pipelines can be defined using:

SageMaker Python SDK
JSON definitions

SageMaker Model Registry

A centralized repository for:

Model versions
Model packages
Approval status
Model history
Deployment-related metadata

It supports controlled promotion of models through stages such as:

Development
Testing
Staging
Production

SageMaker Feature Store

Used to manage and reuse ML features and feature metadata.

AWS Step Functions

Step Functions provides visual, serverless workflow orchestration.

It can coordinate:

Lambda functions
SageMaker jobs
Glue jobs
Approval steps
Notifications
Custom application logic

Best fit

Use Step Functions when coordinating broader AWS workflows that may include ML and non-ML services.

Amazon Managed Workflows for Apache Airflow

Amazon MWAA is a managed service for Apache Airflow.

Airflow uses Python to define workflows called DAGs. It is useful for:

Scheduling
Authoring
Monitoring
Coordinating complex data and ML workflows

AWS manages much of the underlying Airflow infrastructure.

Amazon ECR

Amazon Elastic Container Registry stores container images, including:

Custom training images
Inference images
Deep learning containers

Amazon S3

Amazon S3 is commonly used for:

Raw training data
Prepared datasets
Validation and test data
Model artifacts
Pipeline outputs
Monitoring data

Amazon CloudWatch

CloudWatch supports:

Metrics
Logs
Alarms
Operational monitoring
Triggering automated responses

15. Relevant AWS AI Services and Features

Amazon Bedrock

Managed access to foundation models through APIs. Supports generative AI application development without managing underlying model infrastructure.

Amazon Q

Amazon Q is a generative AI assistant family designed for workplace and developer use cases.

Examples include:

Answering questions using enterprise information
Assisting developers with coding and troubleshooting
Supporting business users with organizational data

Amazon Q is generally an end-user or developer-facing generative AI application, while Amazon Bedrock is a platform for building applications with foundation models.

Exam distinction

Amazon Bedrock: Build generative AI applications using foundation models.
Amazon Q: Use AWS-provided generative AI assistants for business or developer productivity.

Amazon Quick / QuickSight

AWS analytics and business intelligence tools can help users analyze data and visualize business results. They are relevant to the measurement and decision-making portions of an AI/ML lifecycle.

Use analytics tools to:

Visualize data during exploration
Create dashboards
Track business KPIs
Analyze model outcomes
Communicate ROI and customer impact

Kiro

Kiro is an AI-powered development environment intended to assist with software development workflows. It is more relevant to application and development productivity than to the core model training lifecycle.

Amazon SageMaker AI

Amazon SageMaker AI provides capabilities across the ML lifecycle, including:

Data preparation
Model development
Training
Tuning
Experiment tracking
Deployment
Model monitoring
Pipelines
Model Registry
Feature Store

16. Classification Metrics

Classification metrics are usually derived from a confusion matrix.

Actual / Predicted	Positive prediction	Negative prediction
Positive actual	True Positive (TP)	False Negative (FN)
Negative actual	False Positive (FP)	True Negative (TN)

True positive

The model predicts positive and the actual result is positive.

Example: A fraudulent transaction is correctly identified as fraud.

True negative

The model predicts negative and the actual result is negative.

Example: A legitimate transaction is correctly identified as legitimate.

False positive

The model predicts positive, but the actual result is negative.

Example: A legitimate email is incorrectly classified as spam.

False negative

The model predicts negative, but the actual result is positive.

Example: A patient has a disease, but the model predicts that they do not.

Accuracy

Accuracy is the proportion of all predictions that are correct.

\[ Accuracy = \frac{TP + TN}{TP + TN + FP + FN} \]

Use accuracy when:

Classes are relatively balanced
False positives and false negatives have similar costs

Trap

Accuracy can be misleading for imbalanced datasets.

Example: If 99% of transactions are legitimate, a model that always predicts “legitimate” achieves 99% accuracy but detects no fraud.

Precision

Precision measures the proportion of predicted positives that are actually positive.

\[ Precision = \frac{TP}{TP + FP} \]

Precision answers:

When the model predicts positive, how often is it correct?

Use precision when false positives are costly.

Examples:

Avoiding legitimate email being marked as spam
Avoiding unnecessary medical treatment
Avoiding blocking legitimate customers
Reducing false fraud declines

Memory aid

Precision = predicted positives that were precise/correct.

Recall

Recall measures the proportion of actual positives that the model successfully identifies.

\[ Recall = \frac{TP}{TP + FN} \]

Recall answers:

Of all the actual positive cases, how many did the model find?

Recall is also called:

Sensitivity
True positive rate (TPR)

Use recall when false negatives are costly.

Examples:

Detecting disease
Detecting fraud
Finding safety defects
Identifying security threats

Memory aid

Recall = recover as many actual positives as possible.

Precision-recall tradeoff

Increasing the decision threshold generally:

Reduces positive predictions
Reduces false positives
Increases false negatives
Often increases precision
Often decreases recall

Lowering the threshold generally:

Identifies more possible positives
Increases recall
May increase false positives
May reduce precision

The correct balance depends on business impact.

F1 score

The F1 score combines precision and recall using their harmonic mean.

\[ F1 = 2 \times \frac{Precision \times Recall}{Precision + Recall} \]

Use F1 when:

Both precision and recall matter
The dataset is imbalanced
You want one combined classification metric

Trap

F1 is not the ordinary arithmetic average of precision and recall. It is the harmonic mean, so a very low precision or recall significantly lowers the F1 score.

False positive rate

The false positive rate measures the proportion of actual negatives incorrectly classified as positive.

\[ FPR = \frac{FP}{FP + TN} \]

It answers:

Of all actual negative cases, how many were incorrectly flagged as positive?

True negative rate

The true negative rate is also called specificity.

\[ TNR = \frac{TN}{TN + FP} \]

It measures the proportion of actual negatives correctly identified.

\[ TNR = 1 - FPR \]

AUC and ROC

Many classification models return probabilities rather than direct yes/no results.

A classification threshold converts a probability into a class prediction.

Example:

Model probability: 0.72
Threshold: 0.60
Prediction: Positive

Changing the threshold changes the balance between false positives and false negatives.

ROC curve

The Receiver Operating Characteristic (ROC) curve plots:

True positive rate on the vertical axis
False positive rate on the horizontal axis

Across different classification thresholds.

AUC

AUC means Area Under the ROC Curve.

AUC:

Measures performance across many thresholds
Ranges from 0 to 1
1.0 indicates excellent separation
0.5 indicates performance similar to random guessing
Is useful for comparing binary classifiers that produce probabilities

Exam trap

AUC is not the same as accuracy. A model can have a good AUC but require an inappropriate threshold for the business use case.

17. Regression Metrics

Regression predicts continuous numerical values, such as:

Price
Temperature
Demand
Delivery time
Customer lifetime value

Mean squared error

\[ MSE = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2 \]

Where:

\(y_i\) is the actual value
\(\hat{y}_i\) is the predicted value
\(n\) is the number of observations

MSE:

Is always non-negative
Has a lower-is-better interpretation
Penalizes large errors more heavily because errors are squared
Is sensitive to outliers
Uses squared units

Root mean squared error

\[ RMSE = \sqrt{MSE} \]

RMSE:

Is easier to interpret than MSE
Uses the same units as the target variable
Still emphasizes large errors and outliers

Example:

If the target is measured in dollars:

MSE is in dollars squared
RMSE is in dollars

Mean absolute error

\[ MAE = \frac{1}{n}\sum_{i=1}^{n}|y_i - \hat{y}_i| \]

MAE:

Measures the average absolute prediction error
Uses the same units as the target
Is less sensitive to outliers than MSE or RMSE
Is useful when large errors should not receive disproportionate emphasis

Metric selection

Use MSE/RMSE when large errors are especially undesirable.
Use MAE when you want a more robust measure of typical error.

18. Business Metrics

Technical model metrics do not necessarily demonstrate business success.

Business metrics may include:

Cost per user
Cost per prediction
Development cost
Training cost
Inference cost
Operational cost
Customer feedback
Customer retention
Sales or revenue increase
Percentage increase in users
Fraud losses avoided
Processing time reduction
Employee productivity
Conversion rate
Return on investment (ROI)

ROI

A basic ROI concept is:

\[ ROI = \frac{Benefit - Cost}{Cost} \]

The exact calculation depends on the organization’s accounting approach.

Costs may include:

Data acquisition
Data labeling
Model development
Compute
Storage
Inference
Monitoring
Retraining
Human review
Security and compliance
Incorrect predictions

Important principle

The best technical model is not always the best business solution. A slightly less accurate model might produce better ROI if it is much cheaper, faster, easier to maintain, or more scalable.

AWS cost allocation tags

AWS cost allocation tags can be applied to resources used by an ML project.

Example:

Key: MLProject
Value: FraudDetection

Tags can help identify project-specific charges in:

AWS Cost Explorer
AWS cost and usage reporting

This allows teams to compare actual AWS costs with the original cost-benefit analysis.

19. High-Value Exam Tips and Traps

Data and services

Glue Data Catalog stores metadata, not the actual data.
Glue is a managed ETL service.
DataBrew is visual data preparation.
Ground Truth is for creating labeled datasets.
Feature Store stores reusable features and metadata.
S3 commonly stores datasets and model artifacts.
ECR stores container images.

Models

Managed AI APIs are usually the simplest option.
Bedrock provides managed access to foundation models.
JumpStart provides pre-trained and open-source models.
Transfer learning starts from an existing model.
Training from scratch requires the most data, compute, expertise, and responsibility.

Inference

Batch: large offline jobs, results can wait.
Real-time: low-latency interactive responses.
Asynchronous: queued requests, large payloads or long processing.
Serverless: intermittent traffic without managing instances.
Managed SageMaker endpoints reduce infrastructure management.
Self-hosted models provide more control but create more operational responsibility.

Monitoring

Data drift concerns changes in input distributions.
Concept drift concerns changes in the relationship between input and target.
Model Monitor detects production data and model-quality issues.
CloudWatch can raise alarms and trigger remediation.
Retraining should be repeatable and may be scheduled or event-driven.

Metrics

Accuracy can be misleading with imbalanced data.
Precision focuses on reducing false positives.
Recall focuses on reducing false negatives.
F1 balances precision and recall.
AUC evaluates ranking/separation across thresholds.
MSE and RMSE emphasize outliers.
MAE is less sensitive to outliers.
Business metrics measure whether the ML system creates organizational value.

MLOps

MLOps is ML development combined with software engineering and DevOps practices.
Version everything necessary to reproduce a model.
Automate training, testing, deployment, monitoring, and retraining.
SageMaker Pipelines orchestrates reproducible ML workflows.
Model Registry manages model versions and approval history.
Step Functions and MWAA can orchestrate broader workflows.
MLOps improves productivity, repeatability, reliability, auditability, and model quality.