Skill 3.1.7: Configure FM deployment, model hosting, and resource allocation
Task 3.1: Deployment and Orchestration of ML and AI Workflows – Manage deployment infrastructure for ML and AI model types
Core objective: Place an FM on the smallest infrastructure that satisfies latency, throughput, payload, data-residency, and cost requirements. Choose managed API vs. self-hosted endpoint, then allocate capacity deliberately: on-demand vs. reserved, instance/accelerator sizing, autoscaling, and scale-to-zero posture.
Primary decision signals: Who waits? How fast? How work arrives? Payload/processing duration? Traffic pattern (steady/spiky/idle)? Base FM vs. custom/imported? Data sensitivity? Existing control plane (Kubernetes, containers, pipelines)?
FM Hosting Paths
| Path | Hosting Model | Capacity Model | Best For | Watch-outs |
|---|---|---|---|---|
| Amazon Bedrock (base FMs) | Serverless managed API | On-demand or Provisioned Throughput (model units) | No infra, fast time-to-market, variable traffic, multi-model access | Model access request; regional availability; quotas; no endpoint sizing |
| Bedrock Custom Model Import | Serverless managed API for supported open-weight architectures | On-demand only | Bring externally fine-tuned open-weight models without operating endpoints | Only supported architectures/file layouts; import from S3; same InvokeModel/Converse API; no Provisioned Throughput |
| Bedrock custom model (fine-tuned in Bedrock) | Serverless managed API | Provisioned Throughput required | Customized model inside Bedrock | Must reserve capacity; cost commitment |
| SageMaker JumpStart | Preconfigured SageMaker endpoint | Instance-based | Quick deploy of pretrained models | Still SageMaker endpoint; size instances |
| SageMaker AI endpoint | Managed endpoint you size | Real-time, serverless, async, batch | Custom models, control over instance/accelerator, VPC, MME/MCE | Ops for scaling, patching, model loading |
| ECS / EKS / EC2 | Self-managed | You manage | Existing Kubernetes/container control plane; specialized hardware/networking | Undifferentiated heavy lifting |
| AWS Lambda | Event-driven serverless | Per invocation | Lightweight inference/glue, pipeline events | 15-min timeout, payload limits, no GPU, cold starts |
Hard rule: A model customized inside Bedrock must be served through Provisioned Throughput. A model brought in via Custom Model Import is invoked on-demand only.
SageMaker Inference Options for FMs
| Option | Latency / Pattern | Payload & Timeout | Scaling & Cost | FM Use |
|---|---|---|---|---|
| Real-Time Endpoint | ms latency; steady or spiky; caller blocked | ~6 MB; 60 s timeout | Auto Scaling on invocation/CPU/GPU metrics; warm capacity | Interactive chat, fraud scoring, embeddings |
| Serverless Inference | Intermittent with real idle gaps | ~4 MB; 60 s timeout | Scales to zero; pay per execution; Provisioned Concurrency keeps warm | Low-volume, sporadic, cost-sensitive |
| Asynchronous Inference | Queued; no caller blocked | Up to 1 GB; up to 15 min | Managed queue; scales to zero when idle | Large payloads, long-running per-request work |
| Batch Transform | Pre-assembled dataset on schedule | S3 in / S3 out; no endpoint | Compute starts, processes, writes to S3, terminates | Bulk scoring, overnight document processing |
| Multi-Model Endpoint (MME) | Many models, same framework | Shared serving container | Loads/unloads models from memory; cold start for rare models | Many small models, cost optimization |
| Multi-Container Endpoint (MCE) | Up to 15 containers, different frameworks | Pipeline or direct invocation by name | Each container isolated | Heterogeneous models, inference pipeline |
Serverless limits: No accelerated instances, no VPC configuration, no multi-model hosting, 60 s max processing time. Any requirement naming GPU, VPC, or MME rules out serverless.
Real-time scale-to-zero: Not a single setting. Requires inference components plus a scaling policy on the service metric emitted when no capacity can serve a request (e.g.,
HasBacklogWithoutCapacity). Async can scale to zero naturally via its queue.
Resource Allocation & Scaling
- Bedrock Provisioned Throughput: Reserve model units for base FMs or Bedrock-customized models. Guarantees input/output tokens per minute. Use for predictable high-volume production, latency-sensitive apps, or when on-demand quotas are insufficient.
- SageMaker instance sizing: GPU memory must fit model weights + KV cache + batch size. Right-size accelerators: Inferentia for cost-effective inference, GPU for broad framework support, Trainium for training/inference where supported.
- Inference components: Pack multiple models on one endpoint and scale them independently. Enables real-time scale-to-zero when paired with the correct policy.
- Autoscaling metrics:
InvocationsPerInstance,CPUUtilization,GPUUtilization, custom CloudWatch metrics. Target tracking is common for latency/throughput SLAs. - Provisioned Concurrency: Serverless only; keeps capacity warm to avoid cold starts after idle periods.
- Batch Transform controls: Instance count, max concurrent transforms, max payload, strategy (single record vs. mini-batch).
- Throughput posture: Managed FM API replaces instance sizing with capacity reservation. Self-hosting inverts the trade-off: you choose accelerator, instance count, and scaling policy, and inherit the work of keeping them right.
Custom Model Import & External Models
- Bedrock Custom Model Import: Bring supported open-weight architectures from S3 into Bedrock. Served through the same invocation API. No endpoint or instance to size.
- Pre-import gate: Only certain model architectures and file layouts are accepted. Confirm support before planning migration.
- SageMaker path for external models: Package artifacts and host on a managed endpoint. You control container, instance type, VPC, and scaling.
- JumpStart: Catalog of pretrained models deployable to a preconfigured endpoint without authoring container settings first.
Agents & RAG Hosting
- Bedrock AgentCore: Modular services for production agents: serverless runtime, memory, gateway (exposes APIs/functions as tools via open agent protocols such as MCP), identity, observability.
- Bedrock Knowledge Bases: Manages retrieval: ingestion, chunking, embedding, vector store, query answering.
- RAG configuration levers:
- Top-k: Widens recall; more passages = more context but more noise.
- Hybrid search: Adds keyword match alongside vector match; recovers exact terms embeddings blur. Requires vector store with filterable text field.
- Metadata filters: Narrow candidate set before similarity. Best for stale/out-of-scope documents.
- Reranking: Second pass reorders candidates already retrieved. Improves ordering, not recall. Cannot rescue a passage never retrieved.
- Query decomposition: Breaks compound questions into sub-questions, each retrieving against its own topic.
Security, Compliance & Networking
- IAM roles: SageMaker execution role for S3, ECR, KMS access. Bedrock requires permissions for
bedrock:InvokeModel,bedrock:InvokeModelWithResponseStream, etc. Custom Model Import needs S3 read access. - Encryption: KMS at rest for S3, ECR, SageMaker volumes. TLS in transit. Use customer-managed KMS keys where compliance requires.
- Network isolation: VPC endpoints (PrivateLink) for SageMaker, Bedrock, S3, ECR. SageMaker endpoints can run in VPC for private traffic. Serverless does not support VPC.
- Compliance controls: CloudTrail, model invocation logging, Guardrails for Bedrock, data residency constraints, model access approvals.
- Data residency: Confirm regional availability of Bedrock models and Custom Model Import before design.
Anti-Patterns & Exam Gotchas
Serverless for strict latency: Cold starts after idle periods violate ms-level SLAs. Use real-time with autoscaling for continuous/spiky low-latency traffic.
Real-time for batch: Idle endpoint cost with no interactive caller. Use Batch Transform for pre-assembled datasets.
Batch for dynamic requests: Cannot process individual real-time requests arriving throughout the day. Use async inference for queued single requests.
Async for interactive: Caller is not blocked, but queuing adds latency. Use real-time when a user or downstream service waits mid-transaction.
MME for different frameworks: MME assumes same framework and shared serving container. Use MCE for heterogeneous containers.
Provisioned Throughput for imported models: Imported models are on-demand only. Do not plan reserved capacity for them.
Unsupported architecture import: Custom Model Import only accepts supported open-weight architectures and file layouts. Verify before migration.
Over-provisioning GPUs: Right-size instance and use autoscaling. Monitor GPU memory and utilization; unused capacity is pure cost.
No VPC for sensitive data: Use VPC endpoints and private subnets for SageMaker and Bedrock. Serverless cannot join a VPC.
Quick Decision Matrix
| Requirement Signal | Recommended Hosting | Avoid |
|---|---|---|
| Interactive, ms latency, spiky traffic | Real-time endpoint + Auto Scaling | Serverless (cold starts), Batch |
| Intermittent, long idle gaps, cost-sensitive | Serverless endpoint | Real-time (idle cost) |
| Large payload (>6 MB) or minutes-long processing, no caller waiting | Asynchronous inference | Real-time (timeout), Batch (dynamic requests) |
| Pre-assembled dataset on schedule | Batch Transform | Real-time, Async |
| Many small models, same framework | Multi-Model Endpoint | Single endpoint per model |
| Heterogeneous containers, pipeline | Multi-Container Endpoint | MME |
| Base FM, no infra management | Bedrock on-demand | Self-hosted SageMaker |
| Predictable high-volume FM throughput | Bedrock Provisioned Throughput | On-demand only |
| Externally fine-tuned open-weight FM, no endpoint | Bedrock Custom Model Import | SageMaker MME |
| Kubernetes/container control plane already exists | EKS / ECS | Managed SageMaker |
| Lightweight inference/glue, event-driven | Lambda | GPU-heavy inference |