Skill 2.2.6: Combine Multiple ML Models to Improve Performance or Reduce Cost
Task 2.2: ML Model and Foundation Model (FM) Development — Train, fine-tune, and customize models for ML and AI solutions
Core objective: Extract more accuracy per model (ensembling at train/serve time) or host many models for less money (consolidation at serve time). Every combining decision trades latency + operational complexity against accuracy or unit cost — identify which one the scenario optimizes before choosing.
1. Combining at Training Time — Ensemble Methods
| Method | Mechanics | Fixes | Base learner fit | Cost / Latency | Watch out |
|---|---|---|---|---|---|
| Bagging (bootstrap aggregating) | Parallel, independent models on bootstrap samples; average or vote | Variance (overfitting) | High-variance, low-bias (deep trees, KNN) | N× train; N× infer unless consolidated | Little gain on stable low-variance learners (linear models) |
| Boosting | Sequential; each learner fits the residual errors of the last | Bias (underfitting) | Weak, high-bias (shallow trees) | Sequential train; N× infer | Too many rounds / high LR → overfits; sensitive to label noise |
| Stacking | Meta-learner trained on base models' out-of-fold predictions | Both — usually best accuracy | Diverse heterogeneous models | Highest complexity | Meta-features must be out-of-fold or the target leaks |
| Voting / Blending | Hard vote (labels) or soft vote (averaged probabilities) | Variance | Diverse, comparable models | Cheap if probabilities already produced | Soft voting requires calibrated probabilities |
| Model averaging | Mean of continuous predictions | Variance | Regression | Cheap | One outlier model drags the mean |
Exam rule: bagging ⇒ variance, boosting ⇒ bias. "High training accuracy, poor validation accuracy" ⇒ variance reduction, not boosting.
AWS mapping: SageMaker built-ins already contain ensembles — XGBoost (boosting), Linear Learner, KNN, Random Cut Forest. Random Forest / Extra Trees are not built-ins; they arrive via scikit-learn in script mode. Distractor trap: "use a built-in algorithm" ≠ "build an ensemble" — heterogeneous members + meta-learner means custom stacking.
2. Combining at Serving Time — Hosting Decision Matrix
| Pattern | What it is | Best for | Cost behavior | Latency |
|---|---|---|---|---|
| Single-model endpoint | 1 model, 1 container | One high-traffic model; strict SLA | Baseline; scales with instance count | Lowest, predictable |
| Multi-Model Endpoint (MME) | Many artifacts behind one container/endpoint; loaded on demand, unloaded when idle | Hundreds–thousands of similar models with sparse per-model traffic (per-tenant, per-device) | Pay for one endpoint — large savings | Cold-start on first call to an unloaded model |
| Multi-Container Endpoint (MCE) | Up to 15 containers on one endpoint; invoke a specific one, or random routing | Different frameworks/algorithms, A/B testing, ensemble members | No savings — all containers resident | No per-model cold start |
| Serial Inference Pipeline | Up to 15 containers in a linear chain; output of one feeds the next | Pre-process → model → post-process; ensemble → aggregator | Additive (one instance per stage) | Adds per-stage latency; real-time/async only |
| Serverless Inference | Scale-to-zero, pay per request | Intermittent/spiky traffic, dev/test | Cheapest at low volume | Cold start |
| Asynchronous Inference | Queue + S3 payloads, near-real-time | Large payloads, long inference, scale-to-zero | Cheap at low utilization | Seconds–minutes |
| Batch Transform | Offline scoring of a dataset, no endpoint | Bulk / non-real-time scoring | Cheapest for bulk | Offline |
Hard rules: MME and MCE are not interchangeable. MME = same container, many artifacts, dynamic load/unload, saves money. MCE = different containers, all resident, saves ops effort, not money. "Thousands of models" or "reduce cost" ⇒ MME. "Two frameworks / A/B test on one endpoint" ⇒ MCE.
MME prerequisite: each model is a separate S3 prefix packaged as a single
.tar.gz, all loadable by the same inference container, and the request must target a specific model. MME does not support production variants / traffic splitting — for A/B use MCE or a single-model endpoint with multiple production variants (traffic weights must sum to 1; up to 10 variants).Pipeline ≠ MCE: pipeline is sequential (chained); MCE is parallel/independent (pick one).
3. Performance vs Cost — Which Lever
| Goal | Lever | Why |
|---|---|---|
| Higher accuracy, latency tolerated | Ensemble (stacking/boosting), served via inference pipeline or MCE | Members offset each other's errors |
| Higher accuracy, latency-sensitive | Distillation into one student model | Single forward pass at serve time |
| Lower cost, many similar models | MME consolidation | Shared endpoint + shared memory |
| Lower cost, mixed frameworks | MCE | Consolidates endpoints, not instances |
| Lower cost, spiky/unknown traffic | Serverless Inference | Scale to zero |
| Lower cost, bulk scoring | Batch Transform | No always-on endpoint |
| Lower cost, right size | Inference Recommender + autoscaling (target-tracking on invocations per instance) | Matches instance type/count to load |
| Lower cost, same accuracy | Neo compilation / quantization / smaller instance | Cheaper per inference |
Bedrock path (FM): Intelligent Prompt Routing routes a prompt across FMs to balance quality vs cost — the managed "combine models" answer for generative AI. Model Evaluation compares candidates; tuning retrieval (embedding model, chunking, ranking) beats swapping the generator when retrieved passages lack the answer.
4. Security, Governance, Compliance
- Execution role: the endpoint assumes an IAM role to pull images from ECR and read artifacts from S3 — scope to the exact bucket/prefix; MME needs read access to the entire model prefix.
- Encryption: artifacts at rest with SSE-KMS; endpoint-attached storage volumes can be KMS-encrypted; TLS in transit by default.
- Network isolation: VPC-only mode (subnets + security groups) plus interface VPC endpoints (PrivateLink) for the SageMaker API and a gateway endpoint for S3/ECR.
- Governance: Model Registry (versioning, approval status) + Model Cards — register the ensemble as one versioned artifact so all members are traceable.
- Monitoring: Data Capture + Model Monitor (data quality, model quality, bias, drift) per variant/model; CloudWatch for invocation, latency, and ModelLatency alarms.
Gotcha: a network-isolated container cannot download models at runtime, which breaks MME's on-demand load — verify the design allows artifact access.
5. Anti-Patterns & Distractor Traps
- Ensembling a model that isn't overfitting — pay N× latency for zero accuracy gain.
- Stacking with in-sample base predictions — leaks the target; must be out-of-fold.
- MME under a strict p99 SLA — cold starts violate it; use single-model endpoints or keep hot models pinned.
- MCE to save money — it doesn't; every container consumes memory continuously.
- Inference pipeline for batch scoring — use Batch Transform.
- Assuming more epochs or tuning fixes poor validation scores — that is a generalization problem, not a search problem.
- Optimizing the generator when retrieval is broken — fix embedding/chunking/ranking first.
- Ignoring metric direction — tuning toward a minimized error metric when the objective is maximized silently selects the worst configuration.
- Deploying an ensemble without versioning its members — the combined model becomes irreproducible and unrollbackable.