Skill 2.1.6: Assess Tradeoffs Between ML Model Performance, Training Time, and Cost
Task 2.1: ML Model and Foundation Model (FM) Development - Choose appropriate modeling approaches for ML and AI solutions
Core Principle: Model selection balances performance, training time, and cost against hard constraints (interpretability, latency, compliance). No single optimum—match the approach to the business requirement.
Core Tradeoff Triangle
- Performance: Accuracy, precision/recall, F1, AUC, RMSE. Driven by data quality, feature richness, algorithm complexity, and tuning.
- Training Time: Data prep, feature engineering, training compute, hyperparameter tuning, retraining cadence. Often >80% of total effort.
- Cost: Training compute, storage, labeling, inference hosting, operational overhead. Training is one-time; inference scales with traffic.
- Hard Constraints: Interpretability, latency, compliance, security. These can override raw performance (e.g., finance rejects black-box models).
Decision Matrix: Prioritize by Requirement
| Use Case Profile | Prioritize | Sacrifice | AWS Approach |
|---|---|---|---|
| Real-time fraud detection | Low latency, high recall | Training time, cost | XGBoost, SageMaker real-time endpoint |
| Weekly demand forecast | Interpretability, cost | Training time | SageMaker built-in linear/regression, Batch Transform |
| Medical imaging diagnosis | High accuracy | Cost, training time | Deep learning, GPU, SageMaker training |
| High-volume text classification | Cost, latency | Some accuracy | Distilled model, Serverless Inference |
| Rapid prototyping | Training time | Cost, performance | SageMaker JumpStart, AutoGluon, managed AI |
| Fixed business logic | Simplicity, auditability | N/A | Plain code rules |
Levers & Tradeoffs
| Lever | Performance | Training Time | Cost | Notes |
|---|---|---|---|---|
| More data | ↑ | ↑ | ↑ | Diminishing returns, overfitting risk |
| More features | ↑ then ↓ | ↑ | ↑ | Feature engineering cost |
| Complex algorithm | ↑ | ↑ | ↑ | Black box, harder to explain |
| Hyperparameter tuning | ↑ | ↑↑ | ↑↑ | Use AMT, early stopping |
| Early stopping | ↔ | ↓ | ↓ | May miss best model |
| Spot instances | ↔ | ↔ | ↓↓ | Requires checkpointing |
| Distributed training | ↔ | ↓↓ | ↑↑ | Data vs model parallelism |
| Smaller model | ↓ | ↓ | ↓↓ | Good for latency/cost |
| Distillation | ↓ | ↑ | ↓ | Teacher-student, needs prompts |
| Managed AI service | ↔ | ↓↓ | ↓ | For solved tasks (Textract, etc.) |
| Serverless inference | ↔ | N/A | ↓ | Cold starts; intermittent traffic |
| Provisioned throughput | ↔ | N/A | ↑ | Predictable high volume |
FM-Specific Tradeoffs (Adjacent)
- Prompting: Fastest, cheapest, no training. Start here.
- RAG: Fresh, citable data; adds retrieval latency/cost; no model retraining.
- Fine-tuning: Domain behavior/style; training cost, versioning, catastrophic forgetting risk.
- Distillation: Smaller, cheaper, faster; some quality loss.
- Provisioned Throughput: Predictable latency for high volume; higher cost.
- On-Demand: Spiky traffic; per-token cost.
Exam Gotchas
Performance metric must match business objective: Recall for fraud, RMSE for forecast, F1 for imbalance. Training time includes data prep and tuning: Often >80% of effort. Managed services reduce it. Inference cost dominates at scale: Optimize total cost of ownership, not just training cost. Interpretability is a hard requirement in finance/healthcare: Simpler model may beat complex one. Prompt before fine-tune: Fine-tune only when evaluations show persistent gap. Spot training needs checkpointing: Otherwise interruption loses progress. Serverless inference has cold starts: Not for strict low-latency SLAs. Provisioned throughput is for steady high volume: Not cost-effective for spiky traffic. Distillation uses prompts, not labels: Teacher generates responses. RAG vs Fine-tuning: RAG for fresh/citable knowledge; fine-tuning for behavior/style.
Security & Compliance (Cost/Time Impact)
- KMS: Encrypt training data, model artifacts, endpoints at rest.
- IAM: Execution roles for SageMaker, Bedrock; least privilege.
- VPC: Use VPC endpoints for private access to S3/Bedrock; no internet.
- Audit: CloudTrail, SageMaker Model Monitor for compliance; adds cost.
Anti-Patterns
- Custom model for solved task: Use Textract, Rekognition, Comprehend, Transcribe.
- ML for fixed logic: Use rules.
- Fine-tuning before prompting: Prompt engineering first.
- Ignoring inference cost: Training cheap, inference expensive at scale.
- Over-tuning on wrong metric: Align with business KPI.
- Real-time endpoint for batch: Use Batch Transform.
- Provisioned throughput for spiky traffic: Use on-demand.
- No checkpointing with Spot: Lost progress.