AWS AI Practitioner Exam Notes

Task Statement 2.2: Understand the Capabilities and Limits of Generative AI for Solving Business Problems

This task statement focuses on determining:

What generative AI is good at
Where generative AI can fail
How to select an appropriate generative AI model
How to measure technical performance and business value

1. Generative AI as a General-Purpose Technology

Generative AI is a general-purpose technology, meaning it can be applied to many different use cases rather than only one specific task.

Examples include:

Generating text and summaries
Answering questions conversationally
Creating images
Generating or explaining code
Translating languages
Extracting information from documents
Drafting emails and reports
Creating marketing content
Supporting customer service
Producing embeddings for semantic search
Assisting with business analysis

Generative AI can make some AI applications:

Faster to build
Less expensive to develop
Easier for users to interact with
More adaptable to new tasks
Accessible through natural-language prompts

However, generative AI is not automatically suitable for every business problem. A model must be evaluated against business, technical, ethical, compliance, and cost requirements.

2. Advantages and Capabilities of Generative AI

2.1 Adaptability

Generative AI models can often perform multiple tasks without being retrained from scratch.

A single foundation model may be able to:

Summarize a document
Classify customer feedback
Translate text
Answer questions
Generate an email
Extract entities
Create a structured response from unstructured text

This is different from traditional machine learning systems, which are frequently designed for a narrowly defined task.

Example

A traditional model might be trained specifically to classify emails as either:

Complaint
Not a complaint

A generative AI model could potentially classify the email, explain the reasoning, summarize the complaint, and draft a response using a prompt.

Exam point

Generative AI is generally more flexible and adaptable than task-specific models, but this flexibility does not guarantee accuracy or consistency.

2.2 Responsiveness to Natural-Language Instructions

Generative AI models can respond to prompts written in natural language.

Users do not necessarily need to know:

Programming languages
Machine learning algorithms
Model architecture
Database query syntax

A user can describe the desired task in a prompt.

Example

“Read this customer email, identify the main issue, classify its urgency, and draft a polite response.”

The model may perform several related tasks in one interaction.

Important limitation

A model can only reliably use information that is:

Present in its training data
Provided in the prompt or context
Retrieved from an authorized data source
Learned through additional training such as fine-tuning

A model cannot reliably provide accurate details about information it has never encountered.

Example

If asked to write about a newly released AWS service without any source material, the model may produce a plausible but generic answer. If provided with an official AWS announcement or documentation, it can create a more detailed response grounded in that information.

2.3 Conversational Capabilities

Generative AI can interact with users using conversational language.

Common applications include:

Customer-service chatbots
Employee assistants
Technical-support assistants
Virtual tutors
Conversational search
Voice assistants
Internal knowledge assistants

Conversational interfaces can make complex systems easier to use because users can express their needs naturally.

Limitation: conversation context

A model does not necessarily permanently remember previous conversations.

The model usually responds based on the context included in the current request. An application may need to explicitly provide:

Previous messages
User preferences
Account information
Relevant documents
Conversation history

This is often handled by the surrounding application rather than by the base model itself.

Exam trap

Do not assume that a foundation model automatically remembers all previous user interactions or learns a company’s preferences simply because it has been used before.

2.4 Content Generation

Generative AI can create new content, including:

Text
Images
Code
Video
Audio
Summaries
Embeddings

The generated output is based on patterns learned from training data and the instructions provided by the user or application.

Important distinction

“Generate” does not necessarily mean “know” or “verify.”

A model may generate fluent content that is:

Incorrect
Outdated
Unsupported
Biased
Unsafe
Inappropriate for the business context

2.5 Lower Development Effort and Cost

Generative AI can reduce the amount of effort required to build certain applications.

Potential benefits include:

Faster prototyping
Reduced need to create a task-specific model
Natural-language interfaces
Reuse of foundation models
Reduced development time
Faster deployment of AI-assisted workflows

This does not mean every generative AI application is inexpensive. Costs may include:

Model inference
Input and output tokens
Fine-tuning
Data preparation
Retrieval systems
Storage
Monitoring
Security controls
Human review
Application integration
Specialized infrastructure

3. Limitations and Disadvantages of Generative AI

3.1 Hallucinations

A hallucination occurs when a generative AI model produces information that is false, fabricated, or unsupported, often with a confident tone.

Examples

Inventing a citation
Providing a nonexistent product feature
Giving an incorrect medical recommendation
Claiming that a policy exists when it does not
Producing incorrect calculations
Creating fictional facts about a company or person

The answer can sound fluent and convincing while still being wrong.

Why hallucinations occur

Potential causes include:

Incomplete or outdated training data
Ambiguous prompts
Lack of relevant context
Statistical generation rather than direct fact lookup
Conflicting information in training data
Model limitations
Poorly designed retrieval or grounding workflows

How to reduce risk

Possible controls include:

Provide authoritative context
Use retrieval-augmented generation where appropriate
Require citations or source references
Validate outputs against trusted systems
Apply rules and guardrails
Use human review for high-risk decisions
Measure factuality and error rates
Restrict the model from answering outside its supported domain

Exam tip

A confident response is not proof of correctness. Hallucination is one of the most important limitations of generative AI.

3.2 Inaccuracy

Generative AI outputs may be inaccurate even when they are not deliberately misleading.

Accuracy can be affected by:

Poor prompt quality
Insufficient context
Ambiguous instructions
Domain-specific terminology
Out-of-date information
Incorrect source data
Inadequate model selection

For customer-facing or high-impact use cases, accuracy must be measured using predefined standards.

3.3 Nondeterminism

Generative AI models can produce different outputs for the same prompt.

Reasons include:

Probabilistic token generation
Sampling settings
Temperature
Top-p or similar generation controls
Model updates
Context differences

Example

The same prompt may produce:

Different wording
Different examples
Different response structures
Different conclusions
In some cases, inconsistent factual claims

Contrast with traditional deterministic systems

A deterministic system generally produces the same output for the same input and configuration.

Generative AI is often nondeterministic, so evaluating it is more difficult than calculating simple accuracy on a fixed classification problem.

Exam trap

Do not assume that providing the same prompt always results in the exact same response.

3.4 Limited Interpretability

Interpretability refers to how easily people can understand why a model produced a prediction or output.

Generative AI models, especially large neural networks, can be difficult to interpret because:

They contain many parameters
They learn complex nonlinear relationships
Their outputs are generated through many internal steps
A simple human-readable rule may not explain the output

This creates a tradeoff between:

Performance: what the model predicts or generates
Interpretability: why the model produced that result

A highly complex model may have better performance but be harder to explain.

3.5 Toxic, Harmful, or Inappropriate Outputs

Models trained on large amounts of internet content may encounter:

Offensive language
Toxic language
Aggressive or combative responses
Stereotypes
Dangerous instructions
Discriminatory content
Illegal or harmful content

A model may reproduce or generate such content if it is not properly controlled.

Organizations should consider:

Safety testing
Content filters
Guardrails
Human oversight
Input and output monitoring
Responsible AI policies
Domain-specific restrictions

The goal is to promote outputs that are:

Helpful
Honest
Harmless

3.6 Lack of Current or Private Knowledge

A foundation model may not know:

Recent events
Newly released products
Internal company policies
Private customer data
Current inventory
Current account status
Organization-specific procedures

The model must receive relevant information through mechanisms such as:

Prompt context
Retrieval from enterprise data
Fine-tuning
Tool or API integration
Access to approved business systems

Important distinction

Fine-tuning is not the same as giving the model unlimited real-time access to company data. Real-time or current information generally requires application integration or retrieval from an authoritative source.

3.7 Cost and Resource Requirements

Foundation models are often considerably larger than traditional machine learning models.

Costs may arise from:

Large-scale inference
High request volume
Long prompts
Long generated responses
Fine-tuning
Data storage
Networking
Evaluation
Monitoring
Application infrastructure

Larger or more complex models may provide stronger capabilities, but they may also have:

Higher cost
Higher latency
Greater infrastructure requirements
More difficult deployment
More complex monitoring

3.8 Integration and Operational Complexity

To create business value, a model usually must integrate with existing systems and workflows.

Examples include:

Databases
Customer relationship management systems
Enterprise resource planning systems
Document repositories
Identity systems
Business applications
Monitoring and logging platforms

A model that performs well in a demonstration may still fail to deliver value if it cannot securely access the information needed by the business.

Organizations also need personnel who can:

Implement the model
Customize prompts and workflows
Manage data access
Monitor performance
Maintain integrations
Control costs
Review safety and compliance

4. Fine-Tuning and Human Feedback

4.1 Fine-Tuning

Fine-tuning involves additional training of a pretrained model using task-specific or domain-specific data.

Potential goals include:

Improving performance on a particular task
Teaching a desired style
Improving understanding of domain-specific language
Producing more consistent output formats
Adapting the model to organizational terminology

Example

A business could fine-tune a model to produce customer responses in a particular tone and format.

Important limitation

Fine-tuning does not guarantee:

Factual accuracy
Real-time knowledge
Elimination of hallucinations
Perfect compliance
Consistent behavior in every situation

4.2 Instruction Fine-Tuning

Instruction fine-tuning trains a model to better understand human-like instructions and produce useful responses.

It can improve:

Following prompts
Naturalness of language
Task performance
Response format
User experience

4.3 Human Feedback

Fine-tuning with human feedback can help align model behavior with human preferences.

It may improve:

Helpfulness
Honesty
Harmlessness
Safety
Response quality
Reduction of toxic content
Reduction of some incorrect responses

Exam tip

Human feedback can improve alignment, but it does not make the model perfectly truthful or safe.

5. Model Interpretability

5.1 Intrinsic Analysis

Intrinsic interpretability applies to models that are naturally easier to understand because they use:

Simple relationships
Fewer variables
Lower complexity
Clear decision rules

Examples may include simple linear relationships or decision rules.

Advantages

Easier to explain
Easier to audit
Easier to debug
Often easier to justify to regulators or business stakeholders

Tradeoff

Simple models may have lower performance when the problem involves complex nonlinear relationships.

5.2 Post Hoc Analysis

Post hoc analysis attempts to explain a model after it has been trained.

It can be applied to:

Simple models
Complex models
Neural networks
Other black-box models

These techniques are often model agnostic, meaning they can be used across different model types.

Local interpretation

Explains one specific prediction or output.

Example:

Why did the model classify this particular customer message as high priority?

Global interpretation

Explains general model behavior across many inputs.

Example:

Which factors generally influence the model’s predictions?

Exam distinction

Intrinsic analysis: the model is interpretable by design.
Post hoc analysis: an explanation is created after the model generates an output.

6. Selecting an Appropriate Generative AI Model

Model selection should be based on the business problem, not simply on choosing the largest or newest model.

6.1 Model Type and Modality

Different foundation models specialize in different content types.

Potential modalities include:

Text
Chat
Images
Code
Video
Audio
Embeddings

Examples from the lesson

Stable Diffusion: image generation
GPT-4: natural-language generation and conversation

Model type examples

Variational Autoencoders, or VAEs

VAEs are generative models that learn a compressed representation of data and can generate new samples.

Potential strengths:

Useful for learning latent representations
Can generate variations of data
Often used in image and data-generation applications

Potential limitations:

Generated outputs may be blurry or less sharp
Quality depends heavily on training data and architecture

Generative Adversarial Networks, or GANs

GANs use two competing networks:

A generator creates samples
A discriminator attempts to distinguish generated samples from real samples

The generator improves by trying to fool the discriminator.

Potential strengths:

Can generate highly realistic images or data
Useful for synthetic data and image generation

Potential limitations:

Training can be unstable
The generator may suffer from mode collapse
Difficult to train and tune

Autoregressive Models

Autoregressive models generate output sequentially, using previously generated content to predict the next element.

They are commonly used for:

Text generation
Language modeling
Code generation
Sequential data

Potential limitations include:

Latency from sequential generation
Potential error propagation
Sensitivity to prompt and context

Exam tip

Know that VAEs, GANs, and autoregressive models are different generative model types. The correct choice depends on the data modality, task, performance requirements, and constraints.

6.2 Performance Requirements

Evaluate how well the model performs the required task.

Consider:

Accuracy
Relevance
Coherence
Factuality
Appropriateness
Robustness
Domain performance
Ability to follow instructions
Output consistency

A model that is excellent at general conversation may not be the best model for:

Specialized legal documents
Medical text
Software code
Image generation
Real-time classification
Enterprise search

6.3 Required Capabilities

The selected model must support the capabilities required by the application.

Examples:

Text generation
Multilingual responses
Image generation
Code generation
Document summarization
Structured output
Function or tool use
Embedding generation
Long-context processing
Domain-specific terminology

6.4 Latency Requirements

Latency is the time required to produce a response.

Low-latency applications may include:

Customer chat
Interactive assistants
Fraud-related workflows
Real-time recommendations
Voice applications

Higher latency may be acceptable for:

Overnight report generation
Batch document processing
Periodic data analysis
Marketing content generation

Tradeoff

Larger models may provide better quality but often require more computation and may have higher latency.

6.5 Cost Requirements

Consider the total cost, not only the model’s direct usage price.

Potential costs include:

Input and output processing
Fine-tuning
Hosting or inference
Storage
Data transfer
Retrieval systems
Monitoring
Human review
Security
Application development
Maintenance

Exam trap

The most capable model is not always the best business choice. A smaller model may meet requirements at lower cost and latency.

6.6 Model Complexity

A more complex model may offer:

Greater capability
Better performance on difficult tasks
Stronger language understanding
Improved cross-domain performance

But it may also introduce:

Higher cost
More latency
More difficult deployment
Greater operational complexity
More challenging interpretability
Higher resource requirements

Choose the simplest model that satisfies the business requirements.

6.7 Compliance, Security, and Privacy

Model selection must consider organizational and regulatory constraints.

Evaluate:

Where data is processed
Whether sensitive data is exposed
Data retention behavior
Access controls
Encryption
Auditability
Data residency
Industry regulations
Model usage restrictions
Protection of personally identifiable information
Protection of confidential business data

A model that produces high-quality output may still be unsuitable if it violates compliance or privacy requirements.

6.8 Integration Constraints

The model should work with the organization’s existing technology environment.

Consider whether it can:

Access approved enterprise data
Integrate with CRM systems
Integrate with ERP systems
Connect to databases
Work with existing applications
Support required APIs
Operate within the organization’s identity and security model
Produce outputs in a format that downstream systems can use

7. Foundation Models

A foundation model is a large, pretrained model trained on broad datasets, often using vast amounts of unlabeled data.

Foundation models serve as a starting point for developing AI applications.

They can be adapted through:

Prompt engineering
Retrieval and grounding
Fine-tuning
Application workflows
Tool or system integration

Foundation models may support:

Language understanding
Conversational messaging
Text generation
Image generation
Code generation
Embedding generation

Different foundation models specialize in different areas. There is no single foundation model that is best for every use case.

8. Evaluating Generative AI Model Performance

Traditional machine learning evaluation is often easier because the expected output is known.

For example, in a classification problem:

\[ \text{Accuracy} = \frac{\text{Number of correct predictions}}{\text{Total predictions}} \]

Generative AI is more difficult to evaluate because:

Multiple answers may be valid
Outputs are language-based
Responses may vary between runs
Fluency does not guarantee factual correctness
Exact string matching may be inappropriate

8.1 Output Quality Metrics

Important output quality dimensions include:

Relevance

Does the response address the user’s request?

Accuracy

Is the information correct?

Coherence

Is the response logically structured and understandable?

Appropriateness

Is the response suitable for the audience, context, and business use case?

Other possible measures

Factuality
Completeness
Safety
Citation quality
Consistency
Toxicity rate
Error rate
Task success rate

8.2 ROUGE

ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation.

It is primarily used to evaluate automatically generated summaries by comparing them with human-created reference summaries.

It is especially associated with:

Text summarization
Comparing generated text to reference text
Measuring overlap between generated and expected content

Exam tip

ROUGE is primarily associated with summarization.

8.3 BLEU

BLEU stands for Bilingual Evaluation Understudy.

It is primarily used to evaluate machine translation by comparing machine-generated translations with human-generated reference translations.

Exam tip

BLEU is primarily associated with machine translation.

Exam trap

ROUGE and BLEU are not general replacements for all generative AI evaluation. The appropriate metric depends on the task.

9. Business Value and Metrics for Generative AI

Technical quality alone does not prove that a generative AI application is valuable.

An application should be evaluated using both:

Model and output quality metrics
Business performance metrics

9.1 Return on Investment, or ROI

ROI compares the value produced by the application with its total cost.

A simplified formula is:

\[ \text{ROI} = \frac{\text{Benefits} - \text{Costs}}{\text{Costs}} \]

Benefits may include:

Increased revenue
Reduced labor costs
Faster operations
Lower support costs
Improved customer retention
Reduced errors
Increased employee productivity

Costs may include:

Model usage
Infrastructure
Implementation
Integration
Monitoring
Training
Human review
Security and compliance

9.2 Efficiency

Efficiency measures whether the application improves operational productivity.

Possible metrics include:

Task completion rate
Time to complete a task
Reduction in manual effort
Average handling time
Number of cases handled per employee
Processing time per document
Cost per transaction
Error reduction
Employee productivity

Example

A support assistant may be considered successful if it reduces average handling time while maintaining or improving customer satisfaction.

9.3 Conversion Rate

Conversion rate measures the percentage of users who complete a desired action.

Examples:

Completing a purchase
Submitting an application
Clicking a recommendation
Signing up for a service
Accepting an offer

Generative AI may influence conversion through:

Personalized recommendations
Conversational shopping assistants
Improved product descriptions
Automated sales responses
Personalized marketing content

9.4 Average Revenue per User, or ARPU

ARPU measures the average revenue generated per user during a specified period.

Generative AI may improve ARPU through:

Cross-selling
Up-selling
Personalized experiences
Targeted recommendations
Improved customer engagement

9.5 Customer Lifetime Value, or CLTV

Customer lifetime value estimates the total value a customer generates throughout the relationship with a business.

Potential strategies for increasing CLTV include:

Loyalty programs
Creating brand loyalty
Personalized customer experiences
Cross-selling
Up-selling
Collecting and acting on feedback
Improving customer support
Increasing retention

Exam distinction

Conversion rate: whether a user completes a desired action.
ARPU: average revenue per user.
CLTV: total expected value over the customer relationship.

9.6 Accuracy and Error Rate

Accuracy measures how often the system produces a correct result.

Error rate measures how often it produces an incorrect result.

These metrics are especially important for:

Document processing
Classification
Information extraction
Customer support
Compliance workflows
High-impact decisions

For generative applications, accuracy may need to be evaluated together with:

Relevance
Factuality
Completeness
Safety
Human review scores

9.7 Cross-Domain Performance

Cross-domain performance evaluates how well a model transfers knowledge or capabilities across different subject areas.

Example

A model trained or evaluated on one domain may also be tested on:

Finance
Healthcare
Retail
Manufacturing
Customer service

A model with strong cross-domain performance may be more adaptable across an organization.

Exam point

Cross-domain performance is related to generalization and transfer of knowledge across domains.

10. Measuring Success Over Time

AI systems should not be evaluated only at launch.

Organizations should:

Define success criteria before deployment
Establish baseline measurements
Monitor quality and business KPIs
Track errors and hallucinations
Review customer and employee feedback
Monitor cost and latency
Reevaluate the model periodically
Check for changes in business requirements
Validate that the application continues to produce value

Model performance and business value can change over time because of:

New data
Changing user behavior
Model updates
Changing regulations
Changing business priorities
New competitors
Changes in costs
Distribution shifts in inputs

11. High-Value Exam Comparisons

Concept	Meaning
Adaptability	Ability to perform or support different tasks and domains
Responsiveness	Ability to follow natural-language instructions
Conversational capability	Ability to interact through natural dialogue
Hallucination	False or unsupported content presented by the model
Nondeterminism	Same input can produce different outputs
Interpretability	How understandable the model’s reasoning or behavior is
Intrinsic analysis	Interpretability built into a simple model
Post hoc analysis	Explanation applied after model training or prediction
Fine-tuning	Additional training for a specific task, domain, or style
Foundation model	Large pretrained model used as a starting point for applications
ROUGE	Primarily evaluates generated summaries
BLEU	Primarily evaluates machine translations
ROI	Business benefits compared with total costs
ARPU	Average revenue per user
CLTV	Total expected customer value over the relationship
Cross-domain performance	Ability to transfer capabilities across domains

12. Exam Tips and Common Traps

Tip 1: Fluent does not mean factual

A well-written answer may still be a hallucination.

Trap: Assuming confidence, detail, or good grammar proves accuracy.

Tip 2: Fine-tuning does not automatically provide current information

Fine-tuning can adapt behavior, style, and domain performance, but current information generally requires updated data, retrieval, or system integration.

Tip 3: The largest model is not always the best model

Model selection must balance:

Quality
Cost
Latency
Complexity
Compliance
Capability
Integration requirements

Tip 4: Generative AI is often nondeterministic

Do not assume identical prompts always produce identical outputs.

Tip 5: Simple models are generally easier to interpret

Complex neural networks may provide stronger performance but are harder to explain.

Tip 6: Distinguish local and global interpretability

Local: explains one prediction
Global: explains overall model behavior

Tip 7: Match the metric to the use case

Summarization → ROUGE may be appropriate
Machine translation → BLEU may be appropriate
Business chatbot → task completion, relevance, accuracy, customer satisfaction
Sales assistant → conversion rate, ARPU, CLTV
Automation workflow → time saved, manual effort reduction, error rate

Tip 8: Technical metrics and business metrics are different

A model may have good output quality but fail to generate business value.

Conversely, a system may improve efficiency but have unacceptable accuracy or safety problems.

Evaluate both.

Tip 9: Integration is essential for enterprise value

A foundation model must often work with:

CRM systems
ERP systems
Databases
Document stores
Existing workflows

A standalone model demonstration is not the same as a production business solution.

Tip 10: High-risk outputs require validation

For medical, legal, financial, safety, or compliance-related use cases, authoritative validation and human oversight may be necessary.

13. Final Summary

Generative AI offers significant advantages:

Adaptability
Natural-language interaction
Conversational capabilities
Content generation
Faster development
Potential cost and productivity improvements

However, it also has important limitations:

Hallucinations
Inaccurate outputs
Nondeterministic behavior
Limited interpretability
Toxic or harmful responses
Outdated or missing knowledge
Cost and latency
Integration and compliance challenges

Model selection should consider:

Modality and model type
Required capabilities
Quality and performance
Latency
Cost
Complexity
Compliance
Security
Privacy
Integration requirements

Success must be measured using both technical and business metrics, such as:

Relevance
Accuracy
Coherence
Appropriateness
Task completion rate
Reduction in manual effort
Error rate
ROI
Conversion rate
ARPU
CLTV
Cross-domain performance

The key exam principle is:

Choose and evaluate generative AI based on the complete business problem—not just the model’s ability to generate impressive text or other content."