Skill 3.1.4: Evaluate and select appropriate foundation model (FM) deployment options

Task 3.1: Deployment and Orchestration of ML and AI Workflows – Manage deployment infrastructure for ML and AI model types

Core concept: FM deployment splits on one axis — who owns the model host. A managed FM API (Amazon Bedrock) removes instance sizing but replaces it with a capacity posture (on-demand vs. reserved). A self-hosted endpoint (Amazon SageMaker AI, Amazon ECS/EKS, Amazon EC2) gives you accelerator, container, and scaling control and inherits the operational burden. Every other requirement (custom weights, latency SLA, cost shape, data residency) then narrows the choice inside that axis.

The Deployment Ladder

Option	Who hosts the model	Capacity & billing	Best fit	Key constraint
Amazon Bedrock – on-demand	AWS (serverless FM API)	Per input/output token; per-model, per-region RPM/TPM quotas	Variable traffic, fastest path to a base FM, no infra	Throttles under sustained load; no throughput guarantee; per-region model availability
Amazon Bedrock – Provisioned Throughput	AWS	Reserved Model Units, 1- or 6-month commitment	Steady high-volume, latency-sensitive production; custom models	Mandatory for models customized inside Bedrock; idle MUs still bill
Amazon Bedrock – Custom Model Import	AWS	On-demand token pricing	Open-weight model fine-tuned outside AWS, served through the same FM API	Only supported architectures/file layouts; on-demand only; feature parity gaps
Amazon SageMaker JumpStart	You (endpoint you own)	Instance-hour + storage	You need the model artifact, custom inference config, or fine-tune-then-host in one place	You size, scale, and secure the endpoint; no Bedrock invocation API
SageMaker endpoint, self-hosted FM (LMI/DJL container)	You	Instance-hour + autoscaling	Custom weights/architecture, custom container, GPU vs. Inferentia choice	Ops burden: scaling policy, warm capacity, idle accelerator cost
Amazon ECS / Amazon EKS / Amazon EC2	You	Compute + orchestration	Existing Kubernetes/container control plane, bespoke serving stack	Highest ops burden; no managed endpoint features
AWS Lambda	You (event-driven)	Per-request + duration	Glue/orchestration around a deployment; lightweight non-FM inference	15-min limit, ≤10 GB memory, no GPU → not an FM host

Bedrock Capacity: On-Demand vs. Provisioned Throughput

On-demand — pay per token, no commitment, bounded by per-model/per-region quotas (RPM/TPM). Raise limits via Service Quotas, or spread load with cross-region inference profiles.
Provisioned Throughput — reserve Model Units for guaranteed input/output throughput; 1-month or 6-month term. The answer when the requirement names guaranteed throughput, consistent latency at volume, exceed on-demand quota, or serve a Bedrock-customized model.
Cross-region inference profiles — route requests across regions to absorb spikes without a commitment; the request may be processed outside the originating region.

Exam Gotcha: Throughput spikes that produce throttling are fixed by Provisioned Throughput or cross-region inference profiles — not by retries or a larger max_tokens setting.

Exam Gotcha: Model availability is per region. "The model isn't invocable" is often a region/access-request problem, not a configuration problem.

The Custom-Model Fork (highest-yield discriminator)

Path	How the model was customized	Capacity model
Bedrock custom model	Fine-tuned or continued-pre-trained inside Bedrock	Provisioned Throughput required
Bedrock Custom Model Import	Fine-tuned outside AWS (EC2, SageMaker, on-premises), weights in Amazon S3	On-demand only — no Provisioned Throughput

Both end in "my own model." The capacity posture is what separates them on the exam.
Import gate: only specific open-weight architectures, sizes, quantization, and file layouts are accepted (e.g., Llama, Mistral/Mixtral, Flan-T5 families). Confirm the current supported list before planning the migration.
Feature parity: imported models do not support every Bedrock capability available to base models (Provisioned Throughput, fine-tuning, and some agent/guardrail integrations). Validate per feature.
Self-hosted alternative: if the architecture isn't importable, package it into a SageMaker endpoint (or ECS/EKS) instead.

SageMaker Hosting for FMs: Which Endpoint Type Applies

Endpoint type	FM suitability	Notes
Real-time	Default for interactive inference on your own weights	60-second timeout; auto scaling on an invocation metric keeps latency stable under spikes
Asynchronous	Strongest cost posture for bursty self-hosted FM inference	Queue-backed, payloads up to ~1 GB, minutes-long generation, can scale instance count to zero
Batch Transform	Offline bulk generation/scoring only	Input dataset in S3 → predictions in S3 → compute released; never for conversational traffic
Serverless	Ruled out for FMs	No accelerated instances, ~6 GB memory ceiling, 60-second timeout, no VPC config, no multi-model hosting
Inference components	Multi-model / multi-tenant on one fleet	Route by component name; per-component scaling; scale-to-zero needs a policy on the service's no-capacity metric
Multi-model endpoint	Many small models sharing one framework	Same serving container, load/unload from memory → cold-start penalty on rarely invoked models; wrong shape for a few large FMs with different stacks

Cost/latency levers for self-hosted FMs: right-size the accelerator (GPU g5/p4d/p5 vs. AWS Inferentia inf2/Trainium), quantization, tensor parallelism, and continuous batching via the LMI/DJL serving stack.
Real-time scale-in to zero requires inference components plus an out-of-capacity scaling policy; invocations during the provisioning window fail.

Exam Gotcha: If the requirement names "no infrastructure to manage," "no endpoint," or "invoke through a managed API," the answer is Bedrock — not a SageMaker endpoint, and not JumpStart.

Security, Networking & Compliance

IAM — Bedrock authorizes both control plane and data plane per model/inference-profile ARN; agents invoke tools under a scoped execution role, not the caller's identity.
KMS — customer managed keys for custom model artifacts, the S3 bucket holding imported weights, knowledge base stores, and log destinations. Encryption in transit is default for all Bedrock and SageMaker endpoints.
VPC endpoints (AWS PrivateLink) — interface endpoints for bedrock-runtime, bedrock, and the agent endpoints keep FM traffic off the public internet for no-internet/private-subnet designs.
Guardrails — content filters, denied topics, and PII redaction applied at the invocation layer; a governance control independent of hosting choice.
Model invocation logging — CloudWatch Logs or S3, metadata-only or full request/response, for audit and troubleshooting.
Data residency — cross-region inference profiles may process data outside the source region; a residency requirement can force a single-region deployment with Provisioned Throughput instead.

Anti-Patterns & Distractor Traps

SageMaker Serverless Inference for an LLM — no GPU support; the option is structurally disqualified.
Lambda as an FM host — no GPU, 15-minute ceiling.
Batch Transform for a chat UI or live scoring — offline bulk only.
Provisioned Throughput for spiky dev/test or intermittent traffic — you pay for idle Model Units; on-demand is correct.
Assuming on-demand throughput is unlimited — it is quota-bound per model per region.
Assuming Custom Model Import accepts any checkpoint — architecture and layout gates apply.
Assuming an imported model behaves like a Bedrock base model — verify Provisioned Throughput, fine-tuning, and agent/guardrail support.
Self-hosting when the requirement says "no infrastructure to manage" — managed FM API wins regardless of cost arguments.
Confusing capacity with compute — Bedrock answers "how much throughput is reserved," SageMaker answers "which instances are running."

Adjacent FM-Layer Options (deploy the application, not the model)

Amazon Bedrock AgentCore — serverless agent runtime plus memory, gateway (exposes existing APIs/functions as tools over an open agent protocol such as MCP), identity, and observability. Use when the requirement is production agent hosting, not model hosting.
Amazon Bedrock Knowledge Bases — managed RAG ingestion and retrieval. Tune independently: top-k for recall, hybrid search for exact terms (requires a filterable text field in the vector store), metadata filters to exclude stale/out-of-scope documents, reranking to fix ordering of already-retrieved candidates, and query decomposition for compound questions. Reranking cannot recover a passage the first pass never retrieved.