Skip to content

Task Statement 3.1: Describe design considerations for applications that use foundation models

Objectives:

  • Identify selection criteria to choose FMs (for example, cost, modality, latency, multi-lingual, model size, model complexity, customization, input/output length, prompt caching).
  • Describe the effect of inference parameters on model responses (for example, temperature, input/output length).
  • Define Retrieval Augmented Generation (RAG) and describe its business applications (for example, Amazon Bedrock Knowledge Bases).
  • Identify AWS services that help store embeddings within vector databases (for example, Amazon OpenSearch Service, Amazon Aurora, Amazon Neptune, Amazon RDS for PostgreSQL).
  • Explain the cost tradeoffs of various approaches to FM customization (for example, pre-training, fine-tuning, in-context learning, RAG, model distillation).
  • Define the role of AI agents and describe AI agents' business applications.

Lesson 1

Let's get started with the first task statement from Domain 3, which is to describe design considerations for applications that use foundation models. This task statement is split into four lessons. This task statement is focused on understanding the design consideration for applications that use foundation models. For this task statement, our task is to identify selection criteria, to choose pre-trained models such as cost, modality, latency, multilingual, model size, model complexity, customization, input, and output length.

In Domain 2, we talked about how pre-trained models have advantages in machine learning and artificial intelligence, but they also come with considerations. Pre-trained models are large and require a lot of compute resources for training and inference. These requirements can be a challenge for individuals and organizations with limited computing capabilities. We also talked about considerations to ensure alignment with your requirements before selecting a pre-trained model.

For this task statement, we will focus on the requirements for cost, latency constraints, and required modalities. Let's dive deeper into cost considerations. The duration and cost of training a model are important considerations because it can be expensive for hardware storage and more.

Here is a question, do you choose a model that's 98% accurate but costs hundreds of thousands of dollars to train, or do you choose a less accurate model that is 97% accurate, but only costs a few thousand dollars? The answer will depend on your requirements. You need to find the balance between training time, cost, and model performance. Your goal is to architect a scalable solution that is efficient and does not reduce the performance of the model. The cost to build and maintain a model is an important consideration in the success of a project. The more complex a model is, the greater its impact will be throughout the model's lifecycle.

Let's dive deeper into considerations for latency constraints, inference speed, and real-time requirements. If the AI application needs to process data and provide results in real time, then that fact is another consideration for the model selection. For example, some complex models might have longer inference times and will not be able to meet your latency requirements. Inference speed or the duration it takes a model to process data and produce a prediction is another consideration. Here is another question. Suppose that you have a self-driving vehicle system that will require instant decisions and models with slow inference times will not meet this requirement. What model do you choose? You might use a k-nearest neighbors, or KNN, model that performs most of its computational work during the inference stage. However, this model might be slower to generate predictions. This self-driving vehicle system is an extremely high dimensional problem, and a KNN model would not be the best choice. So understanding the requirements for your use case in terms of inference speed is important when deciding on AI model.

Also, let's dive deeper into modalities. There are several considerations that you should keep in mind here. In most AI systems, each modality has a specific embedding, but ensemble methods combine several models to achieve better performance than a single model. And another consideration might be the need to choose multilingual models trained on relevant languages. What are other considerations of choosing a pre-trained model? Architecture and complexity come to mind. We talked about how different architectures have different strengths and weaknesses and might be more suitable for different types of tasks. For example, you might choose convolutional neural networks, CNNs, for image recognition, and you might choose recurrent neural networks, RNNs, for natural language processing. The complexity of the model can be measured by the number of parameters, layers, and operations. All these factors will affect its speed, memory, and accuracy, and remember that more complex models have higher accuracy, but also require more computational resources and data.

A subset of architecture and complexity is the performance and metrics of the pre-trained model on the original dataset and the new dataset. You can use standard metrics to evaluate and compare different models. Such metrics might include accuracy, precision, recall, F1 score, root mean squared error or RMSE, mean average precision or MAP, and mean absolute error, MAE. However, you should also consider the tradeoffs between these metrics and choose the ones that are most relevant for your task. For example, suppose that you are doing object detection. You might care more about MAP than accuracy because it measures how well the model can locate and classify multiple objects in an image. Accuracy is not recommended with datasets that are not evenly distributed or imbalanced. It's important when you are choosing an appropriate metric or set of metrics to assess your model's performance before selecting a model. I'm going to pause this lesson here, and in the next lesson we will continue with task statement 3.1.

Lesson 2

Let's continue with task statement 3.1, and talk about another consideration, biases that might be present in the training data. It's important to understand how to mitigate risks, address ethical concerns, and make informed decisions about model selection and fine-tuning. Another consideration is the availability and compatibility of the pre-trained model. You can find many pre-trained models online, such as on TensorFlow hub, PyTorch hub, Hugging Face, or other repositories. However, you should check whether the model is compatible with your framework, language, and environment, and check to ensure it has a license and documentation. You should also check whether the model has been updated and maintained regularly and whether it has any known issues or limitations. Another consideration is customization and explainability of the pre-trained model. You might want to modify or extend the pre-trained model to suit your task, such as adding new layers, classes, or features. You might also want to understand how the pre-trained model works and how it makes predictions or decisions.

For these purposes, you should look for models that are flexible, modular, transparent, and provide tools or methods to visualize or interpret their inner workings. The ability to interpret and explain model outcomes is important. Being transparent refers to interpretability. For example, it means being able to explain mathematically through coefficients and formulas why a model makes a certain prediction. This interpretability is possible if the model is simple enough, but foundation models are not interpretable by design because they are extremely complex. They are the opposite of transparent and often referred to as black boxes.

Explainability attempts to explain this black box, by approximating it locally with a simpler model that is interpretable. Explainability is different from interpretability and pre-trained models cannot be transparent by design. If interpretability is a requirement, then pre-trained foundation models might not be the best choice. For example, certain models such as linear regression and decision trees might be better when it comes to explainability. The complexity of a model is important and can help you uncover intricate patterns within the data, but it can add challenges to maintenance and interpretability. There are a few key considerations for this. Greater complexity might lead to enhanced performance, but can increase costs. The more complicated the model is, the harder it is to explain the outputs of the model. And there are more considerations too, such as hardware constraints, maintenance updates, data privacy, transfer learning, and more. I'm going to pause this lesson here, and in the next lesson we will continue with task statement 3.1.

Lesson 3

Let's continue talking about task statement 3.1 and talk about the effect of inference parameters on model responses such as randomness, diversity, and length. Inference parameters help control the behavior and output characteristics of the foundation models. So let's pause and define inference. The inference is where you process new data through the model to make predictions. It is the process of generating an output from an input that you provided to model. Amazon Bedrock gives you the ability to run inference in the foundation model you choose.

Here is a question, when you run inference, what inputs do you provide? A prompt, which is an input, is provided to the model for it to generate a response. And inference parameters are a set of values that can be adjusted to limit or influence the model response. You can run inference with base models, custom models, or provision models to test foundation model responses with different prompts and inference parameters. After you have sufficiently explored these methods, you can set up your application to run model inference by calling these APIs. Amazon Bedrock foundation models support the inference parameters of temperature, Top K, Top P to control randomness and diversity in the response. Amazon Bedrock also supports parameters such as response length, penalties, and stop sequences to limit the length of the response. It is important to consider these different parameter settings and experiment to find the optimal balance between diversity, coherence, and resource efficiency. Then continuously monitor and adjust these parameters in production to maintain optimal performance and align with evolving requirements. I have added flashcards for the inference parameters and the link under additional resources.

We will talk more about prompt engineering in the next lesson, but for this task statement, let's pause for fundamentals and talk about prompt engineering. AWS defines prompts as a specific set of inputs provided by you the user. These inputs guide LLMS to generate an appropriate response or output for a given task or instruction. You can also add contextual data from your internal databases to enrich the prompt. You can integrate additional domain-specific data from these data stores or vector data stores that add to your prompts semantically relevant inputs. This method is called retrieval augmented generation, RAG, and we will cover this topic in a few minutes. Remember, the best AI model for you depends on your needs, resources, and constraints. It's all about finding the right balance to meet your project's objectives and requirements. For the exam, ensure that you understand the cost tradeoffs of various approaches to foundation model customization, such as pre-training, fine-tuning, in-context learning, and RAG.

Let's pause here and ask a question. What is the difference between vector databases and a machine learning model? A vector database is a collection of data that is stored as mathematical representations. And remember, we related to this as the Excel spreadsheet. Vector databases store structured and unstructured data, such as text or images with the vector embeddings. Vector embeddings are a way to convert words and sentences and other data into numbers. Embeddings are the numerical representation of that data that represents the meaning and relationships. A vector database is filled with dense vectors by processing input data, generally text data, and using an ML model, generally an embedding model. So it's important to understand that a machine learning model is a prerequisite to create a vector database and the indexing technology itself. Vector databases are the factual reference of foundation model based applications, helping the model retrieve trustworthy data. Foundation models use vector databases as an external data source to improve their capabilities by enhancing search recommendations and text generation use cases. Vector databases add additional capabilities for efficient and fast lookup, and to provide data management, fault tolerance, authentication, and access control and query engine.

For example, knowledge bases for Amazon Bedrock give you the ability of collecting data sources into a repository of information. This way, you build an application that takes advantage of retrieval augmented generation, RAG. RAG enhances language models to retrieve and use external knowledge during the generation process. It is a technique in which the retrieval of information from data sources augments the generation of model responses. I'm going to pause this lesson here, and in the next lesson, we will continue with task statement 3.1 and talk more about RAG.

Lesson 4

Let's continue with task statement 3.1 and pick up from the last lesson. RAG combines two components, a retriever component, which searches through a knowledge base and a generator component, which produces outputs based on the retrieved information. This combination helps the model access up-to-date and domain-specific knowledge beyond their training data. Now, let's think back on vector databases. What is an example of how you would use a vector database in the real world? When I want to query the model, the prompt is passed into the query encoder, which encodes or embeds the data into the same format as the external data. Then the embedding can be passed to the vector database to search and return similar embeddings that have been through the model. Those embeddings are then attached to my new query and can also be mapped back to their original location. If the vector database finds similar data, then the retriever retrieves that data, the LLM combines or augments the new data or text with the original prompt, then the prompt is sent to the LLM to return a completion. We have mentioned that generative LLMs can be prone to hallucinations, which is a situation where the model generates a believable but factually incorrect response. RAG solves this problem by complimenting generative LLMs with an external knowledge base that is typically built using a vector database, hydrated with vector-coded knowledge articles.

Amazon Bedrock offers RAG models that integrate with custom knowledge bases. RAG finds applications in various domains, such as question-answering, dialect systems, and content generation using external knowledge to provide accurate and contextually relevant responses. Let's also talk about AWS services that help store embeddings within vector databases. Examples include Amazon OpenSearch Service, Amazon Aurora, Redis, Amazon Neptune, Amazon DocumentDB with MongoDB compatibility, and Amazon RDS with PostgreSQL.

The OpenSearch search engine delivers low-latency search and aggregations, OpenSearch dashboards, visualization, and dashboarding tools. It also has plugins that provide advanced capabilities such as alerting, fine-grained access control, observability, security monitoring and vector storage and processing. With OpenSearch Service's vector database capabilities, you can implement semantic search, retrieval of augmented generation, RAG with LLMs, recommendation engines, and search media too. With semantic search, you can improve the retrieved results by using language-based embeddings on search documents. I added a link to a workshop example to improve search relevance with ML in Amazon OpenSearch Service. It explores the difference between keyword and semantic search based on a model called Bidirectional Encoder Representations from Transformers, BERT. It's hosted by Amazon SageMaker to generate vectors and store them in OpenSearch. The vector engine from Amazon OpenSearch Serverless provides vector storage and search capabilities. In this way, it helps build machine learning, ML, augmented search experiences, and generative artificial intelligence, AI, applications without having to manage the vector database infrastructure. With a fully managed RAG offered by knowledge bases for Amazon Bedrock, you can securely connect foundation models, FMs, to your company data. It is stored as embeddings in the vector engine for more relevant, context-specific, and accurate responses without continuously re-training the FM.

Amazon RDS for PostgreSQL also supports the pgvector extension to store embeddings and perform efficient searches. We added a link for more options that are available for more advanced vectors. In Domain 2 and the lesson for task statement 2.1, I said one additional thing to keep in mind is that larger models of any architecture are typically more capable of carrying out their tasks well. Researchers have found that the larger a model, the more likely it is to work because you need it without additional in-context learning or further training. For this task statement, ensure that you understand the role of agents in multi-step tasks.

Foundation models can understand and respond to queries based on their pre-trained knowledge. However, they are unable to complete any real world tasks such as booking a flight or processing a purchase order. Why? Well, these tasks require organization-specific data and workflows that typically need custom programming. Agents for Amazon Bedrock is a fully managed AI capability from AWS to help you build applications foundation models. Agents can automatically break down tasks and generate the required orchestration logic or write custom code, and agents can securely connect to your databases through APIs, they can ingest and structure the data for machine consumption and augment it with contextual details to produce more accurate responses and fulfill requests. Agents are an additional piece of software that orchestrates the prompt completion workflows and interactions between the user requests, foundation model, and external data sources or applications. Also, agents automatically call APIs to take actions and invoke knowledge bases to supplement information for these actions. For example, you can create an agent that helps customers process reservations for your next scuba diving vacation. All right, let's get started with task statement 3.2 in the next lesson.