Task Statement 3.4: Describe methods to evaluate foundation model performance

Objectives:

• Determine approaches to evaluate FM performance (for example, human-in-the-loop evaluation, benchmark datasets, Amazon Bedrock Model Evaluation).
• Identify relevant metrics to assess FM performance (for example, Recall-Oriented Understudy for Gisting Evaluation [ROUGE], Bilingual Evaluation Understudy [BLEU], BERTScore, LLM-as-a-judge).
• Determine whether an FM effectively meets business objectives (for example, productivity, user engagement, task engineering).
• Identify approaches to evaluate the performance of applications built with FM (for example, RAG, agents, workflows).
• Identify business objective alignment metrics for AI applications (for example, task completion rate, user satisfaction, cost per interaction).

Lesson 1

Let's get started with a fourth and final task statement from Domain 3, which is to describe methods to evaluate foundation model performance. This task statement is split into two lessons. For this task statement, let's talk about the things that you need to consider to integrate your model into applications. Several important questions should be asked at this stage. The first set of questions is related to how your model will function in deployment. How fast do you need your model to generate completions? What compute budget do you have available? Are you willing to trade off model performance for improved inference speed or lower storage? You might have inference challenges for LLMs when you deploy, on premises, to the cloud or to edge devices. These challenges can include compute storage and low-latency requirements for the users that are consuming your application. One optimization technique is to improve application performance by reducing the size of the LLMs. This action can reduce the inference latency because the smaller size model loads more quickly. However, remember that reducing the size of the model might decrease its performance.

Other optimization techniques include making a more concise prompt, reducing the size of the retrieved snippets and their number, and reducing generation through inference parameters and prompt. Some techniques work better than others for generative models, but you might have tradeoffs between accuracy and performance. We have mentioned many different metrics to evaluate the performance of your generative AI application. We also talked about how evaluation metrics, such as accuracy or root mean square error, RMSE, are more straightforward to calculate. This is because the predictions are deterministic and can be compared against the labels. The output of generative AI models is non-deterministic and makes the evaluation more difficult to determine. Evaluation metrics for generative AI models are more task-specific. For example, Recall Oriented Understudy for Gisting Evaluation, or ROUGE, is a set of metrics and a software package. It is used to evaluate automatic summarization tasks and machine translation software in natural language processing. It evaluates how well the input compares to the generated output. Bilingual Evaluation Understudy, or BLEU, is an algorithm that is used for translation tasks. It evaluates the quality of text which has been machine translated from one natural language to another.

To evaluate and compare LLMs without a task-specific focus, you can use existing benchmarks and datasets that LLM researchers have established specifically for this purpose. Examples include: General Language Understanding Evaluation, GLUE, Holistic Evaluation of Language Models, HELM, Massive Multitask Language Understanding, MMLU, and Beyond the Imitation Game Benchmark, BIG-bench. GLUE was created to help the development of models that can generalize across multiple tasks. It is a collection of natural language tasks, such as sentiment analysis and question answering. You can use these tasks to evaluate and compare model performance across a set of language tasks. Then, you can use the benchmark to measure and compare the model performance. SuperGlue was introduced in 2019 and adds additional tasks, such as multi-sentence reasoning and reading comprehension. Both the GLUE and SuperGlue benchmarks have leadership boards that can be used to compare and contrast evaluated models. Massive Multitask Language Understanding, MMLU, evaluates the knowledge and problem-solving capabilities of the model.

To perform well, models must have extensive world knowledge and problem-solving ability. The models are tested on more than basic language understanding, such as history, mathematics, laws, computer science, and more. The Beyond the Imitation Game Benchmark, BIG-bench, focuses on tasks that are beyond the capabilities of the current language models. It contains tasks such as math, biology, physics, bias, linguistics, reasoning, childhood development, software development, and more. Another benchmark is the Holistic Evaluation of Language Models, HELM, which is a benchmark to help improve model transparency. It offers users guidance on which model performs well for a given task. HELM is a combination of metrics for tasks such as summarization, question and answer, sentiment analysis, and bias detection. You can also use human workers to manually evaluate your model responses. For example, you can use human workers to compare the responses of SageMaker JumpStart models, and you can also specify responses from models outside AWS.

Additionally, you can use Amazon SageMaker Clarify to evaluate large language models, LLMs, and create model evaluation jobs. A model evaluation job helps to evaluate and compare model quality and metrics for text-based foundation models from SageMaker JumpStart. Amazon Bedrock provides an evaluation module that can automatically compare generated responses and calculate a semantic similarity base score, BERTscore, against a human reference. It is suitable to evaluate faithfulness and hallucinations in text-generation tasks. I'm going to pause this lesson here, and in the next lesson, we will continue with task statement 3.4.

Lesson 2

Let's continue with task statement 3.4. In the last lesson, we talked about a first set of questions related to how your LLM will function in deployment. The second set of questions to consider, focus on how to integrate your model into applications. What additional resources might your model need? Do you intend for your model to interact with external data or other applications? And, if so, how will you connect to those resources? We have talked about retrieval augmented generation, RAG, a few times. Remember, it is a framework for LLM systems to use as external data sources. RAG helps with the challenge of internal knowledge of models being outdated. If your model is outdated, then it'll not know newer information. Also, models can have difficulty with complex math. The model might return a number that is close to the correct answer, but it's not correct. RAG also helps by providing a context, which helps to avoid hallucinations and improve factuality by grounding responses. However, you need more configurations to connect your LLM to external components and integrate deployment within your application. You can use an orchestration library to configure and manage the passing of user input to the large language model and the return of completions.

RAG helps overcome the outdated knowledge issue if your model uses older information. Re-training the model on new data adds additional costs and it will require repeated re-training to keep the model updated with the new knowledge. But RAG helps your model access additional external data at inference times. And the additional external data can help improve the relevance and accuracy of completions. Let's talk about a high-level overview of a stack that you can reference for building generative AI applications. You can use it to determine whether a foundation model effectively meets business objectives such as productivity, user engagement, task engineering. Here are a few questions. How will your model be consumed? What will be the design of the intended application or API interface that your model will be consumed through? We know that we can use foundation models and LLMs to meet various business objectives and enhance user experiences. And by using the evaluation techniques that we've discussed, you can determine whether your objectives are being met.

First, you must define your business goals with a specific problem that you're solving. You must determine the metrics for success and your infrastructure to support the model and application. And remember to measure, monitor, and review your metrics to evaluate the performance. Integrating generative AI models also requires the ability to interact with existing systems, software applications, or services in real time by using APIs and interfaces. Here are some key components to build end-to-end solutions for your application. The infrastructure layer provides the compute, storage, and network to serve and host your LLMs and to host your application components. For this layer, consider security recommendations. You must ensure that your data is being handled securely across the AI lifecycle for data preparation, training, and inferencing.

Next, you choose the large language models to use with your application and the appropriate infrastructure for your inference needs. Remember to include any additional storage and consider whether you need real-time or a near real-time interaction with a model. Here is a quick question. Why would you need additional storage? You might need to implement the collecting and storing of user completions or feedback to use for additional fine-tuning evaluation or alignment to your objectives. And remember to add security to isolate your data. For the next layer, you might also need additional tools and frameworks for large language models, or you might need to use model hubs to centrally manage and share models for applications. For the final layer, you'll need to have user interface that will be used to consume the application, such as website or a rest API. To ensure secure connections, security is a focus of this layer.

Additionally, you might require various stack components for interacting with your application. Consider requirements such as real-time or near real-time interaction with a model, along with retrieving data from external sources, such as RAG. Also, depending on your objectives, you might need to add storage to collect and store the outputs. Or you could store feedback from users that might be useful for additional fine-tuning, alignment, and evaluation as your application matures. The next layer might require additional tools and frameworks for LLMs, such as model hubs to centrally manage and share models for use in applications. In the final layer, you typically have a user interface that the application will be consumed through, such as a website or a rest API. This layer is where you also include the security components required for interacting with your application. Your users, whether they are human end users or other systems that can access your application through its APIs, will interact with this entire stack. I'm going to conclude this lesson here and let's start with our walkthrough questions.