Task Statement 2.2: Understand the capabilities & limits of GenAI for solving business problems

Objectives:

• Describe the advantages of GenAI (for example, adaptability, responsiveness, conversational capabilities, ability to generate content).
• Identify disadvantages of GenAI solutions (for example, hallucinations, interpretability, inaccuracy, nondeterminism).
• Identify factors to consider when selecting GenAI models (for example, model types, performance requirements, capabilities, constraints, compliance, cost, latency, model complexity).
• Determine business value and metrics for GenAI applications (for example, cross-domain performance, ROI, efficiency, conversion rate, average revenue per user, accuracy, customer lifetime value).

Lesson 1

Let's get started with the second task statement from Domain 2, which is to describe the capabilities and limitations of generative AI for solving business problems. This task statement is split into three lessons.

So far in this course, we have learned that generative AI and LLMs are a general-purpose technology. Therefore, similar to other general-purpose technologies, such as deep learning, it has many uses. It is useful not just for a single application, but for a lot of different applications that span many corners of the economy. Let's talk about the advantages of generative AI, such as adaptability, responsiveness, and simplicity. We talked about how we use artificial intelligence almost daily in Domain 1. Every time you do a web search, that's AI. Every time you use your credit card, there is probably an AI checking whether it's really you using your credit cards. Or every time you go to amazon.com, AI recommends products to you.

Many AI systems have been complex and expensive to build. However, generative AI is making many AI applications much more straightforward to build. Generative AI might help your business build valuable AI applications at lower costs, and faster too. I have added flashcards to review best practices on using generative AI. Generative AI is a fascinating technology, but it can't do everything. It's important to understand generative AI's limitations to ensure that you're building models that are responsible, ethical, and fair. We must ensure that we use AI responsibly and benefit people. We will dive deeper into responsible AI under Domain 4.

If you're trying to figure out what prompting an LLM can do, there is one question that I find helpful. Could I as a 10-year-old child follow the instructions in the prompt and complete the task? For example, could I at the age of 10 follow the instructions to read an email and determine whether the email is a complaint? Well, I think most people can do this, and an LLM can do that too. But could you or a child write an article about a new AWS service without any information about that service? Probably not. We could write a generic document, but it would not include specifics about the new AWS service. However, if you read a blog post or a press release on the topic, then you could write a document with much more detail, and a large language model could do that too.

Every time you prompt your LLM, the LLM does not actually remember earlier conversations. It is similar to asking a different child for every single task. Therefore, you don't get to train them over time on specifics of your business or the style you want them to write, but you could with fine-tuning. I'm going to pause this lesson here, and in the next lesson, we will continue with task statement 2.2.

Lesson 2

Let's continue with task statement 2.2 and talk about how people are using LLM applications and revisit the generative AI project lifecycle. The goal of fine-tuning with instructions is to further train your model to better understand human-like prompts and generate more human-like responses. This approach can improve a model's performance sustainability over the original pre-trained based version and lead to more natural-sounding language. Natural-sounding human language can be a challenge, and there have been even articles about some LLMs behaving badly.

Issues include models that use toxic language in their completions, replying in combative and aggressive voices, and providing detailed information about dangerous topics. These problems exist because large models are trained on vast amounts of text data from the internet where such language appears frequently. Because of this, an LLM could give misleading or incorrect answers. Suppose that you ask the LLM about the disproven health advice for diabetics, that eating a carb-focused diet is the key to good health. The model should refute the story. Instead, the model might give a confident and totally incorrect response. Definitely not the truthful and honest answer a person is seeking. This action is called a hallucination.

To get the correct answer, you should double check the answer with an authoritative source before counting on it. Also, the LLM shouldn't create harmful completions, such as being offensive, discriminatory, or illicit criminal behavior. Again, we'll talk about these important human values: helpfulness, honesty, and harmlessness under Domain 4. These values are a set of principles that guide developers in the responsible use of AI. We can add fine-tuning with human feedback to better align models with human preferences and to increase the helpfulness, honesty, and harmlessness of the completions. This further training can also decrease the toxicity and reduce the generation of incorrect information.

When selecting a model or an algorithm, consider interpretability. The higher the interpretability of a machine learning model, the easier it is to comprehend the model's predictions, but you have a tradeoff to consider. It's between what the model has predicted, which is the model performance, and why the model made such a prediction, which is the model interpretability. And these are methods for model interpretability that can be classified into intrinsic analysis and post hoc analysis. Intrinsic analysis can be applied to interpret models that have low complexity or simple relationships between the input variables and the predictions.

The simple relationship between the inputs, input variables, output results, and predictions in high model interpretability can lead to lower performance. The reason is that algorithms are unable to capture complex non-linear interactions. Post hoc analysis can be applied to interpret simple relationship models and more complex models, such as neural networks, which can capture non-linear interactions. These methods are often model agnostic and provide mechanisms to interpret a trained model based on the inputs and the output predictions. Post hoc analysis can be performed at a local level and zoom in on a single data point, or it can be performed at a global level and zoom out and view the overall behavior of the model.

Let's talk about how you can formalize the improvement in performance of your fine-tuned model over the pre-trained model that you started with. Developers of large language models use specific metrics. You can use these metrics to assess the performance of your models and compare it to other models out in the world. In traditional machine learning, you can assess how well a model is performing on training and validation datasets where the output is already known. You're able to calculate basic metrics, such as accuracy, which states the fraction of all predictions that are correct because the models are deterministic.

But with large language models, the output is non-deterministic and language-based evaluation is much more challenging. For example, consider these two sentences: I drink coffee and I do not drink coffee. As humans, we can see and understand the differences and similarities too, but when you train a model on millions of sentences, you need an automated and structured way to make measurements. ROUGE and BLEU are two widely used evaluation metrics for different tasks. ROUGE, or Recall-Oriented Understudy for Gisting Evaluation, is primarily employed to assess the quality of automatically-generated summaries by comparing them to human-generated reference summaries. On the other hand, BLEU, or Bilingual Evaluation Understudy, is an algorithm designed to evaluate the quality of machine-translated texts by comparing it to human-generated translations. I will continue talking about ROUGE and BLEU under task statement 3.4. For now, let's pause this lesson and continue with task statement 2.2.

Lesson 3

Let's continue talking about task statement 2.2. In this lesson, we'll talk about various factors to select appropriate generative AI models, such as model type, performance, requirements, capabilities, constraints, and compliance.

How do you choose the correct model architecture to ensure the success of your generative AI project? We covered different architectures under task statement 2.1 and I also provided flashcards for review. But for this task statement, let's dive a bit deeper.

Generative AI foundation models are designed to generate different types of content, such as text and chat, images, code, video, and embeddings. You can modify these models to fit specific domains and tasks by adjusting the algorithms or model structures.

When it comes to data generation, it's important to select the most appropriate model because it can significantly impact the quality of data. The most used models are variational autoencoder, VAEs, generative adversarial networks, GANs, and auto regressive models.

Each of these models has advantages and disadvantages depending on the complexity and quality of the data. These are just some of the types of generative AI models and ongoing research and development leads to more new advanced generative models. The number and size of foundation models on the market have grown at a rapid pace. Dozens of models are now available.

According to AWS, this list contains the prominent foundation models released since 2018. Let's wrap up this task statement and talk about how to determine business metrics for generative AI applications. Examples of these metrics include cross-domain performance, efficiency, conversion rate, average revenue per user, accuracy, customer lifetime value, and more.

Foundation models are trained on huge, unlabeled broad datasets and they underpin the capabilities of generative AI. As a result, they are considerably larger than traditional machine learning models, which are generally used for more specific functions. FMs are used as the baseline starting point for developing and creating models. These models can be used to interpret and understand language, have conversational messaging, and create and generate images. Different foundation models specialize in different areas.

For example, the stable diffusion model by Stability AI is great for image generation. And the GPT-4 model is used by ChatGPT for natural language. FMs are able to produce a range of outputs based on prompts with high levels of accuracy. But we need to ensure that we understand the metrics or key performance indicators, KPIs, to use.

These metrics evaluate the performance, impact, and success of our generative AI applications.

Earlier in this lesson, we mentioned accuracy. By tracking, monitoring, and analyzing the right business metrics, you can learn from the past, monitor the present, and plan for the future. By analyzing large amounts of business data to forecast their future values or to detect outliers and understand the root cause is complex, time consuming, and not always accurate.

Foundation models are creating opportunities and challenges for organizations. Some of these are ensuring high-quality outputs that align with business needs and minimizing hallucinations or false information. For FMs to be truly useful in an enterprise context, they need to integrate and inter-operate with existing business systems and workflows.

For example, they must access data from databases and use enterprise resource planning, ERP, and customer relationship management, CRM. To gain real business value, organizations need employees with strong technical skills to implement, customize, and maintain FMs. They also require computational resources and infrastructure to deploy the model cost effectively and ensure that customers are receiving value.

The quality of the outputs of the FMS will determine the adoption and use particularly in customer-facing applications like chatbots. Output quality metrics are relevance, accuracy, coherence, and appropriateness, which all contribute to overall user satisfaction. Your output quality should be measured with predefined standards to ensure that AI systems meet efficiency requirements.

Efficiency impacts the generative AI workflow. It can be tracked with metrics such as task completion rates and reduction in manual efforts, making a direct contribution to operational productivity. Also, a low error rate helps to maintain both accuracy and credibility of the AI applications.

Organizations need to evaluate potential return on investment and weigh in the cost and benefits of FMs considering their application. Additionally, it's important to understand the metrics for comparing operational costs and efficiencies gained.

Here's a question. What are strategies to maximize customer lifetime value, CLTV, which is a key metric for scaling your business? Loyalty programs, creating brand loyalty, collecting feedback, cross selling, personalized experiences, and more. You also need metrics for cross-domain performance to evaluate the transfer and application of knowledge and skills across different domains to generate or predict cross-domain data and content.

Remember that AI is also evolving, so ensure that you measure, monitor, review, and reevaluate your model to ensure that you are meeting your business requirements and goals.

All right, let's continue with task statement 2.3 in the next lesson.