2.3

Task 2.3: Analyze model performance

Knowledge of:

• Model evaluation techniques and metrics (for example, confusion matrix, heat maps, F1 score, accuracy, precision, recall, Root Mean Square Error [RMSE], receiver operating characteristic [ROC], Area Under the ROC Curve [AUC])
• Methods to create performance baselines
• Methods to identify model overfitting and underfitting
• Metrics available in SageMaker Clarify to gain insights into ML training data and models
• Convergence issues

Skills in:

• Selecting and interpreting evaluation metrics and detecting model bias
• Assessing tradeoffs between model performance, training time, and cost
• Performing reproducible experiments by using AWS services
• Comparing the performance of a shadow variant to the performance of a production variant
• Using SageMaker Clarify to interpret model outputs
• Using SageMaker Model Debugger to debug model convergence