Skip to content

Skill 3.1.2: Select deployment orchestrators and multi-model or multi-container deployment strategies

Question 1 - Multi-Model Endpoint Selection

A financial services company has trained 200 specialized fraud detection models, each tailored to a specific merchant category. All models use the same XGBoost framework version and are relatively small. Traffic patterns show that 20 models handle 90% of daily inference requests, while the remaining 180 models are invoked only occasionally when transactions from less common merchant categories appear. The team wants to minimize infrastructure costs while serving all models with acceptable performance.

Which deployment strategy should the ML engineer recommend? (Select ONE)

A) Deploy each model to its own single-model endpoint with Application Auto Scaling configured for each endpoint.

B) Deploy all 200 models to a multi-model endpoint, accepting occasional latency penalties when rarely used models require loading from Amazon S3.

C) Deploy all 200 models to a multi-container endpoint with one container per model, invoking each container directly by name.

D) Deploy the 20 frequently used models to single-model endpoints and the remaining 180 models to batch transform jobs scheduled every 30 minutes.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: Amazon SageMaker Multi-Model Endpoints (MMEs) allow hosting hundreds of models on a shared compute fleet behind a single serving container. Since all models use the same XGBoost framework, they share the same serving container. SageMaker dynamically manages model caching—frequently used models stay in memory for fast inference, while rarely used models are loaded from Amazon S3 when requested, incurring only an occasional cold-start latency penalty. This dramatically reduces hosting costs compared to individual endpoints.

Distractor Analysis

  • A is Incorrect: Deploying 200 individual single-model endpoints would require maintaining 200 separate compute fleets, leading to extremely high idle infrastructure costs, especially for the 180 rarely used models.
  • C is Incorrect: Multi-container endpoints support up to 15 containers on a single endpoint, not 200. Additionally, multi-container endpoints are designed for different container configurations (e.g., different frameworks or inference pipelines), not for hosting large numbers of models sharing the same framework.
  • D is Incorrect: Hybrid deployment adds unnecessary architectural complexity. Batch transform cannot serve real-time inference requests for the 180 less common merchant categories—customers would wait up to 30 minutes for fraud scoring, which is unacceptable.

Key Takeaway: Use Multi-Model Endpoints when hosting many models that share the same framework and have uneven traffic patterns; the shared fleet dramatically reduces costs at the price of occasional cold-start loads for infrequently used models.

Question 2 - Inference Pipeline with Different Frameworks

A healthcare analytics company is deploying an inference workflow for processing X-ray images. The pipeline consists of three stages: a PyTorch-based image preprocessing model, a TensorFlow-based anomaly detection model, and a PyTorch-based classification model. The output of each stage feeds directly into the next stage. The team wants to deploy this pipeline as a single endpoint invocation that runs the models sequentially without building a custom orchestration layer.

Which deployment strategy should the ML engineer implement? (Select ONE)

A) Deploy all three models to a SageMaker multi-model endpoint and invoke each model sequentially from the client application.

B) Deploy all three models to a SageMaker multi-container endpoint with a serial inference pipeline configuration.

C) Deploy each model to a separate real-time endpoint and create an AWS Step Functions workflow to orchestrate the calls.

D) Deploy all three models to a SageMaker multi-container endpoint with direct invocation mode, where a custom Flask container routes requests between containers.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: Amazon SageMaker Multi-Container Endpoints (MCEs) support inference pipelines where up to 15 containers run sequentially. The endpoint automatically chains requests from one container to the next—the first container's response becomes the second container's request, and so on. Since the models use different frameworks (PyTorch and TensorFlow), each can have its own serving container, and the pipeline handles the orchestration without requiring client-side coordination.

Distractor Analysis

  • A is Incorrect: Multi-model endpoints require all models to share the same serving container and framework. These models use different frameworks (PyTorch vs. TensorFlow), so they cannot be hosted behind a single serving container in an MME.
  • C is Incorrect: While Step Functions could orchestrate calls between separate endpoints, this approach incurs additional network latency between services, higher management overhead, and unnecessary costs from maintaining multiple endpoints. The multi-container inference pipeline natively handles sequential execution.
  • D is Incorrect: Direct invocation mode requires the client to specify which container to call, rather than automatically routing between containers sequentially. Building a custom Flask routing container introduces unnecessary operational complexity and defeats the purpose of the native inference pipeline feature.

Key Takeaway: Use Multi-Container Endpoints with serial inference pipelines when models use different frameworks and need to execute sequentially; the endpoint natively chains container requests without custom orchestration code.

Question 3 - Bedrock Custom Model Import vs. SageMaker Hosting

A startup has fine-tuned a Llama 3 model on proprietary training data using Amazon SageMaker. The team wants to serve this customized model through a managed API without provisioning or managing any instances. They require on-demand pricing (pay-per-token) rather than committing to reserved throughput capacity. The model architecture is supported by Amazon Bedrock Custom Model Import.

Which inference approach should the ML engineer recommend? (Select ONE)

A) Import the fine-tuned model into Amazon Bedrock using Custom Model Import and invoke it via the InvokeModel API.

B) Configure Provisioned Throughput in Amazon Bedrock and deploy the fine-tuned model to it.

C) Deploy the fine-tuned model to a SageMaker real-time endpoint with Application Auto Scaling.

D) Deploy the fine-tuned model to a SageMaker serverless endpoint.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Amazon Bedrock Custom Model Import enables teams to import supported open-weight models (like Llama 3) that were fine-tuned outside Bedrock into Bedrock's managed infrastructure. Imported models are served through the standard Bedrock InvokeModel API with on-demand pricing, meaning the team pays per token without committing to reserved capacity or managing any endpoints or instances.

Distractor Analysis

  • B is Incorrect: Provisioned Throughput in Bedrock requires committing to reserved model units for a set period of time. This violates the requirement for on-demand pricing. Additionally, Provisioned Throughput is typically used for models customized directly within Bedrock or for base models requiring guaranteed throughput, not for serving imported custom models with pay-per-token pricing.
  • C is Incorrect: Deploying to a SageMaker real-time endpoint requires the team to manage instance types, scaling policies, and endpoint configuration—directly contradicting the requirement to serve the model through a managed API without provisioning or managing instances.
  • D is Incorrect: While SageMaker serverless endpoints eliminate instance management, they still require the team to define and manage endpoint configuration. They also don't provide the managed foundation model API experience (like standardized InvokeModel or Converse APIs) that Bedrock offers. Serverless endpoints also have cold-start latency that may affect performance.

Key Takeaway: Bedrock Custom Model Import serves externally fine-tuned open-weight models through the managed Bedrock API with on-demand pricing, distinguishing it from Provisioned Throughput (reserved capacity) and SageMaker hosting (managed endpoint, not managed API).

Question 4 - Orchestrator Selection for Kubernetes-Native Workflows

A large enterprise has standardized its entire ML infrastructure on Kubernetes. Their ML engineering team uses Helm charts, Kubernetes operators, and custom controllers to manage deployments. They are deploying a real-time inference service that requires integration with Istio service mesh, custom Kubernetes ingress controllers, and workload identity via IAM Roles for Service Accounts (IRSA). The team wants to leverage their existing Kubernetes tooling while minimizing operational overhead.

Which deployment orchestrator should the ML engineer recommend? (Select ONE)

A) Amazon SageMaker real-time endpoint with inference components

B) Amazon Elastic Kubernetes Service (EKS) with a SageMaker-compatible model server

C) Amazon ECS with service discovery and AWS App Mesh

D) AWS Lambda with an API Gateway REST API

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: Amazon EKS is the managed Kubernetes service that allows the team to leverage their existing Kubernetes manifests, Helm charts, operators, and controllers. EKS supports Istio for service mesh, custom ingress controllers, and IRSA for workload identity—all implemented using Kubernetes-native patterns. This allows the team to deploy their real-time inference service using their existing Kubernetes tooling without re-architecting their deployment pipeline.

Distractor Analysis

  • A is Incorrect: SageMaker real-time endpoints abstract away the underlying infrastructure and do not support Kubernetes-native controls like custom Istio configurations, Kubernetes ingress controllers, or Kubernetes operators. The team would need to abandon their existing Kubernetes tooling, which violates the requirement.
  • C is Incorrect: ECS uses AWS-native orchestration (task definitions, services) rather than Kubernetes. It would require the team to translate all their Kubernetes manifests and operators to ECS patterns, which is significant operational overhead and doesn't support Kubernetes-specific features like custom controllers.
  • D is Incorrect: Lambda is event-driven serverless compute unsuitable for deploying large ML inference workloads that require GPU acceleration or long-running model serving containers. It also provides no Kubernetes integration and cannot leverage Helm charts or Kubernetes operators.

Key Takeaway: When an organization's control plane is already Kubernetes (Helm, operators, Istio), choose Amazon EKS to preserve that tooling; switch to managed options like SageMaker only when there's no existing Kubernetes investment.

Question 5 - Asynchronous Inference vs. Batch Transform for Large Payloads

An insurance company receives individual claims documents as high-resolution scanned PDFs throughout the day. Each document is between 10–20 MB and takes approximately 8 minutes to process through a custom document understanding model. Users upload documents through a web portal but do not wait for results—they receive email notifications when processing completes. The system must scale down to zero instances during periods of inactivity to minimize costs.

Which inference strategy should the ML engineer implement? (Select ONE)

A) A SageMaker real-time endpoint with a 10-minute timeout exception

B) A SageMaker serverless endpoint with an S3-presigned URL for result retrieval

C) A SageMaker asynchronous endpoint with auto scaling down to zero instances

D) A SageMaker batch transform job triggered by an S3 event notification

Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

  • C is Correct: Amazon SageMaker Asynchronous Inference is designed for exactly this workload: individual requests with payloads up to 1 GB and processing times up to 15–60 minutes. Requests are placed in a managed internal queue, and callers receive an immediate acknowledgment (202 status). The endpoint can scale down to zero instances when the queue is empty, eliminating costs during idle periods. Results are written to S3, and SNS email notifications can be triggered upon completion.

Distractor Analysis

  • A is Incorrect: SageMaker real-time endpoints enforce a strict 60-second timeout on synchronous invocation requests. An 8-minute processing time would consistently time out. The reference to "10-minute timeout exception" is invalid—this is not a configurable feature of SageMaker endpoints.
  • B is Incorrect: While SageMaker serverless endpoints also enforce a 60-second maximum processing timeout, they are synchronous and would fail on 8-minute tasks.
  • D is Incorrect: Batch transform processes static, pre-assembled datasets in bulk. It cannot handle individually arriving documents throughout the day. Additionally, batch transform jobs must be explicitly orchestrated on a schedule or manually—they cannot be triggered by individual S3 event notifications matching this dynamic arrival pattern.

Key Takeaway: Asynchronous inference is the right choice for large payloads (>6 MB) or long processing times (minutes) that arrive as individual requests but don't require synchronous responses; it supports scale-to-zero and queued processing.

Question 6 - Serverless Endpoint Limitations

An autonomous vehicle company needs to deploy a computer vision model for real-time obstacle detection. The model requires NVIDIA A100 GPU acceleration, must run within a specific VPC to access an internal PostgreSQL metadata database, and must host multiple model variants on a single endpoint for A/B testing. Traffic is intermittent—vehicles operate primarily during daytime shifts, with 8 hours of complete inactivity overnight.

Which deployment approach meets ALL requirements? (Select ONE)

A) A SageMaker serverless endpoint with GPU acceleration enabled, VPC configuration, and multi-model support

B) A SageMaker real-time endpoint with GPU instances, VPC configuration, and multiple inference components for A/B testing

C) A SageMaker multi-model endpoint with auto scaling down to zero instances during overnight hours

D) An Amazon Bedrock Custom Model Import deployment with Provisioned Throughput

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: SageMaker real-time endpoints support all three requirements: (1) GPU instance selection including NVIDIA A100 instances, (2) VPC configuration for accessing private resources like an internal PostgreSQL database, and (3) inference components that allow multiple model variants to share an endpoint for A/B testing with traffic routing.

Distractor Analysis

  • A is Incorrect: SageMaker serverless endpoints have critical limitations: they do not support GPU instances, cannot be configured with a VPC, and do not support multi-model deployment. All three required features are unavailable with serverless hosting, making this option impossible.
  • C is Incorrect: While multi-model endpoints support GPU instances and can host multiple models, enabling access to a private VPC requires VPC configuration. Additionally, scale-to-zero requires inference components, which may not be compatible with all features. This option doesn't explicitly address the VPC requirement or the inference component configuration for A/B testing.
  • D is Incorrect: Amazon Bedrock serves foundation models through a managed API and does not support custom computer vision models for autonomous vehicle obstacle detection. Provisioned Throughput is designed for foundation models, not specialized perception models, and Bedrock provides no VPC configuration for accessing internal databases.

Key Takeaway: Serverless endpoints exclude GPU acceleration, VPC configuration, and multi-model hosting—any requirement naming these features rules out serverless immediately; use real-time endpoints with inference components.

Question 7 - Retrieval Configuration for Stale Document Filtering

A legal research platform uses Amazon Bedrock Knowledge Bases to retrieve relevant case law for user queries. Users report that the system frequently returns cases from outdated legal precedents and documents from jurisdictions outside their current matter. The knowledge base contains documents from multiple states and across many years. The team wants to prevent stale or out-of-scope documents from surfacing in retrieval results before similarity search is performed.

Which retrieval configuration should the ML engineer implement? (Select ONE)

A) Add a reranking model to reorder retrieved results

B) Increase the number of returned passages to improve recall

C) Configure metadata filtering on jurisdiction and case year fields

D) Enable hybrid search to combine vector and keyword matching

Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

  • C is Correct: Metadata filtering in Bedrock Knowledge Bases cuts the candidate set before vector similarity search is performed. By filtering on metadata fields such as jurisdiction (e.g., "California", "New York") and case year (e.g., ">= 2020"), documents from wrong jurisdictions or outdated years are excluded from the search entirely. This directly addresses the complaint that stale or out-of-scope documents keep surfacing.

Distractor Analysis

  • A is Incorrect: Reranking models reorder candidates that were already retrieved by vector search, improving ordering rather than recall or candidate exclusion. It cannot exclude documents that the first pass erroneously retrieved—it would simply reorder those same stale or out-of-scope documents.
  • B is Incorrect: Increasing the number of returned passages (top_k) widens recall, bringing back MORE documents—including potentially more stale or out-of-scope documents. This would worsen the problem rather than fix it.
  • D is Incorrect: Hybrid search adds keyword matching alongside vector matching, which can help with exact term matching (like specific case citations). However, it does not filter by metadata or document attributes, so it would not prevent old cases or out-of-jurisdiction documents from surfacing.

Key Takeaway: Metadata filtering is the right lever when documents are being retrieved from wrong categories (stale, out-of-scope); it narrows the candidate pool before vector search executes, while reranking only reorders already retrieved results.

Question 8 - Throughput Posture for Internally Customized Bedrock Models

A media company has fine-tuned an Amazon Bedrock Titan model (customized within Amazon Bedrock) for generating marketing copy in their brand voice. The model is used by their content management system, which makes requests sporadically throughout the day, with periods of high volume during weekday mornings and near-zero usage overnight and weekends. The team wants to minimize costs while guaranteeing that the customized model remains available when needed.

Which deployment configuration should the ML engineer implement? (Select ONE)

A) Provisioned Throughput with 1 model unit to the customized Titan model, accepting that the model is unavailable during idle periods.

B) On-demand pricing for the customized Titan model.

C) Provisioned Throughput with sufficient model units to handle peak demand, keeping the capacity reserved 24/7.

D) Custom Model Import to serve the fine-tuned model on-demand.

Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

  • C is Correct: Models customized within Amazon Bedrock (such as fine-tuned Titan models) can ONLY be served through Provisioned Throughput—they cannot be invoked on-demand. Provisioned Throughput reserves model units for a specific duration, guaranteeing capacity availability. While this means paying for reserved capacity 24/7 (even during overnight/weekend idle periods), it is the only way to serve internally customized Bedrock models and keep them available when needed.

Distractor Analysis

  • A is Incorrect: Provisioned Throughput requires committing to a specific number of model units, and 1 model unit may not provide sufficient throughput for weekday morning peak demand. This option incorrectly suggests the model would be "unavailable during idle periods"—Provisioned Throughput maintains capacity 24/7 during the commitment period.
  • B is Incorrect: On-demand pricing is NOT available for models customized within Amazon Bedrock. Internally fine-tuned models must be served through Provisioned Throughput—they cannot be invoked on-demand like base models.
  • D is Incorrect: Custom Model Import is for importing externally customized open-weight architectures (like Llama, Mistral) into Bedrock. It is not applicable to Titan models fine-tuned directly within Bedrock, as Titan is proprietary and not an open-weight architecture.

Key Takeaway: Models customized inside Amazon Bedrock (not imported) require Provisioned Throughput—they cannot be served on-demand; the difference in operating posture between internal fine-tuning and Custom Model Import paths is critical.

Question 9 - Agent Tool Invocation via Model Context Protocol

A customer service team is building an Amazon Bedrock agent that needs to interact with their existing internal tools. These tools include a ticket management system API, a CRM API, and a document approval workflow. The tools are currently exposed as REST APIs behind an internal API gateway. The team wants to integrate these tools without rewriting them as native AWS Lambda functions and wants to use an open protocol for tool communication. The agent must authenticate with the internal APIs using the organization's federated identity system.

Which Bedrock AgentCore capability should the ML engineer use? (Select ONE)

A) Bedrock AgentCore Gateway to expose existing MCP-based APIs and functions as tools

B) Bedrock AgentCore Memory to cache API responses

C) Bedrock Knowledge Bases to index API documentation

D) Bedrock Guardrails to filter API responses

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Amazon Bedrock AgentCore Gateway allows existing APIs and functions to be exposed as tools to agents using the Model Context Protocol (MCP), an open protocol for agent-tool communication. The Gateway handles identity propagation using federated identity (such as OAuth 2.0 or SAML), allowing the agent to authenticate with internal APIs without rewriting them as AWS Lambda functions or creating custom connectors.

Distractor Analysis

  • B is Incorrect: AgentCore Memory provides long-term memory for agents (conversation history, user preferences, session context), not tool integration or API authentication. It cannot expose existing APIs as callable tools.
  • C is Incorrect: Bedrock Knowledge Bases manages document retrieval for RAG workflows (indexing unstructured documents and answering questions with retrieved passages). It cannot expose APIs as executable tools or handle API authentication.
  • D is Incorrect: Bedrock Guardrails apply content filtering, PII redaction, and safety checks to model inputs and outputs. They do not provide tool integration capabilities or API gateway functionality.

Key Takeaway: Bedrock AgentCore Gateway is the intended mechanism for exposing existing APIs/functions as agent tools via the open Model Context Protocol, including federated identity propagation—use it instead of rewriting APIs as Lambda functions.

Question 10 - Multi-Container Endpoint Direct Invocation for A/B Testing

An e-commerce company is conducting A/B testing on two product recommendation models deployed using different serving stacks: Model A uses NVIDIA Triton Inference Server on PyTorch, and Model B uses a TensorFlow Serving container. The team wants to deploy both models behind a single endpoint where the client application can explicitly specify which model variant to use for each request. They also want to run both containers simultaneously, with each container independently scalable.

Which deployment strategy should the ML engineer implement? (Select ONE)

A) A SageMaker multi-model endpoint with both Triton and TensorFlow Serving containers registered

B) A SageMaker multi-container endpoint in direct invocation mode, with separate inference components for each container

C) A SageMaker multi-container endpoint in serial inference pipeline mode

D) Two separate SageMaker endpoints behind an Application Load Balancer with path-based routing

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: Amazon SageMaker Multi-Container Endpoints (MCEs) in direct invocation mode allow running multiple containers (up to 15) directly addressable by the client. Using inference components, each container can be independently scaled (different instance counts, different instance types, or scaling policies). The client application specifies which container (model variant) to invoke for each request, enabling explicit A/B testing control.

Distractor Analysis

  • A is Incorrect: Multi-model endpoints assume all models share the same serving container and framework. Triton and TensorFlow Serving are different serving stacks, so they cannot be hosted behind a single serving container in an MME.
  • C is Incorrect: Serial inference pipeline mode chains containers sequentially—the first container's output becomes the second container's input. This is inappropriate for A/B testing where only ONE model should process each request based on the client's routing decision, not both models sequentially.
  • D is Incorrect: Deploying two separate endpoints behind an ALB introduces additional infrastructure management (load balancer, path-based routing rules, health checks) and overhead. Multi-container endpoints with direct invocation achieve the same A/B testing goal natively within SageMaker without requiring external service orchestration.

Key Takeaway: For A/B testing with different serving stacks requiring explicit client-side model selection, use Multi-Container Endpoints with direct invocation mode and inference components; serial pipelines are for sequential execution, and MMEs are for same-framework model libraries.