AIF-C01 Task 3.3: Training and Fine-Tuning Foundation Models

Overview

Foundation models are trained in stages. The major stages and techniques include:

Pre-training: Build broad general capabilities from large datasets.
Fine-tuning: Adapt a pre-trained model for a specific task, domain, style, or behavior.
Instruction tuning: Train the model to follow natural-language instructions.
Continuous pre-training: Continue training an existing model on additional domain-specific data.
Reinforcement learning from human feedback: Use human preferences to improve behavior.
Model distillation: Transfer useful behavior from a larger model to a smaller model.

For the exam, focus on the differences between these approaches, their data requirements, and their cost and complexity tradeoffs.

1. Foundation Model Training Stages

1.1 Pre-training

Pre-training is the initial process of training a foundation model on a very large and diverse dataset.

The goal is to learn general patterns, such as:

Language structure.
Facts and concepts present in the training data.
Relationships between words and concepts.
Image or audio patterns, depending on the modality.
General reasoning and generation capabilities.

Pre-training usually requires:

Large datasets.
Significant compute capacity.
Distributed training infrastructure.
Specialized machine learning expertise.
Extensive testing and evaluation.
Substantial time and financial investment.

The result is a general-purpose foundation model that can later be adapted for specific applications.

Exam takeaway

Pre-training creates broad, general capabilities. It is usually much more expensive and complex than fine-tuning an existing model.

1.2 Fine-tuning

Fine-tuning adapts a pre-trained foundation model using a smaller, task- or domain-specific dataset.

Fine-tuning can help a model learn:

A particular response style.
A specific classification task.
Organization-specific terminology.
A consistent output format.
Domain-specific patterns.
Specialized instruction-following behavior.

The model starts with capabilities learned during pre-training and adjusts them using the new training data.

Examples

A company could fine-tune a model to:

Classify insurance claims.
Summarize legal documents using a defined format.
Respond in a particular brand voice.
Extract fields from industry-specific forms.
Generate support responses using approved terminology.

Exam takeaway

Fine-tuning generally changes model behavior or task performance. It is not usually the best way to provide frequently changing factual information. RAG is often more suitable for that requirement.

1.3 Continuous pre-training

Continuous pre-training continues training an existing foundation model on additional data, often from a specific domain.

It can help the model better understand:

Industry terminology.
Specialized language.
Domain-specific writing styles.
Large collections of domain documents.
New patterns not sufficiently represented in the original training data.

Examples of domains include:

Medicine.
Law.
Finance.
Engineering.
Scientific research.

Continuous pre-training is generally more extensive than task-specific fine-tuning.

Advantages

Can broaden the model’s domain knowledge.
Useful when the domain has a large amount of specialized text.
May improve understanding of industry vocabulary and concepts.

Limitations

More expensive and complex than ordinary fine-tuning.
Requires substantial high-quality data.
Requires additional training infrastructure.
Can introduce errors or biases from the new data.
Must be evaluated to ensure that general capabilities are not degraded.

Exam distinction

Fine-tuning usually targets a task, format, behavior, or smaller specialized dataset.
Continuous pre-training adds broader domain exposure through additional pre-training data.

1.4 Instruction tuning

Instruction tuning fine-tunes a model using examples of instructions and desired responses.

Training records commonly contain:

An instruction.
Optional context or input.
An expected response.

Example:

Instruction: Summarize the following customer complaint.
Input: The customer received the wrong product and wants a replacement.
Expected response: The customer received an incorrect product and requests a replacement.

Instruction tuning helps a model:

Follow natural-language requests.
Produce more useful answers.
Follow specified output formats.
Perform a range of related tasks.
Respond more consistently to user instructions.

Instruction tuning is a type of fine-tuning focused on instruction-following behavior.

1.5 Transfer learning

Transfer learning uses knowledge learned from one training task or dataset to improve performance on another related task.

Fine-tuning a pre-trained foundation model is an example of transfer learning because the model transfers general knowledge learned during pre-training to a specialized task.

Benefits

Requires less data than training from scratch.
Requires less compute than pre-training.
Can reduce training time.
Can improve performance on a specific task.

Example

A general language model can be adapted using labeled customer-support examples instead of training a new language model from the beginning.

1.6 Reinforcement learning from human feedback

Reinforcement learning from human feedback, or RLHF, uses human preferences to improve model behavior.

A high-level RLHF process may include:

Generate multiple model responses to the same prompts.
Ask human reviewers to rank or compare the responses.
Use the preference data to train a reward model or otherwise estimate preferred behavior.
Optimize the FM toward responses that receive higher preference scores.
Evaluate the resulting model for quality, safety, and unintended behavior.

RLHF can improve:

Helpfulness.
Instruction following.
Tone.
Safety behavior.
Alignment with human preferences.

Limitations

Human review is expensive and time-consuming.
Reviewers may disagree.
Feedback may be subjective or biased.
Poorly designed feedback can cause unintended behavior.
The resulting model still requires evaluation and monitoring.

Exam takeaway

RLHF uses human preference feedback, not just traditional labeled input-output examples.

1.7 Model distillation

Model distillation transfers knowledge or behavior from a larger teacher model to a smaller student model.

A typical process is:

Use the teacher model to generate outputs or labels.
Use those outputs as training data for the student model.
Train the student to approximate the teacher’s useful behavior.
Evaluate the student against the target requirements.

Benefits

Lower inference cost.
Lower latency.
Smaller infrastructure requirements.
Useful for high-volume applications.
May allow deployment in more constrained environments.

Limitations

The student may have lower quality or reasoning ability.
The student may not retain all teacher capabilities.
The distillation process requires data, compute, and evaluation.
Incorrect teacher outputs may be transferred to the student.

Exam distinction

Distillation is focused on creating a smaller or more efficient model. It is not the same as simply reducing the prompt length or using a smaller model without training.

2. Comparing FM Training and Customization Approaches

Approach	Primary purpose	Data requirement	Relative cost and complexity
Pre-training	Create a general foundation model	Very large and diverse dataset	Very high
Fine-tuning	Adapt a model to a task or behavior	Smaller task-specific dataset	Moderate to high
Instruction tuning	Improve instruction following	Instruction and response examples	Moderate to high
Continuous pre-training	Add broad domain knowledge	Large domain-specific dataset	High
Transfer learning	Reuse learned knowledge for a related task	Task-specific examples	Lower than training from scratch
RLHF	Align behavior with human preferences	Human rankings or preferences	High
Model distillation	Create a smaller, efficient model	Teacher-generated or labeled outputs	Moderate to high
Prompt engineering	Change behavior through instructions only	Prompts and examples	Low
RAG	Provide external information at inference time	Documents and embeddings	Moderate application complexity

Important exam rule

Choose the least complex approach that meets the requirement:

Change instructions or formatting → Prompt engineering.
Add current or private information → RAG.
Learn a specialized task or style → Fine-tuning.
Add broad domain language and knowledge → Continuous pre-training.
Build a new general-purpose model → Pre-training.
Reduce inference cost after achieving desired behavior → Distillation.

3. Preparing Data for Fine-Tuning

The quality of fine-tuning data is often more important than simply increasing the amount of data.

A data preparation process should cover:

Data collection.
Data curation.
Data governance.
Data formatting.
Data labeling.
Data splitting.
Quality checks.
Representativeness.
Privacy and security review.
Evaluation after training.

4. Data Curation

Data curation is the process of selecting, cleaning, organizing, and preparing data for training.

4.1 Remove poor-quality data

Remove or correct data that is:

Duplicated.
Incomplete.
Irrelevant.
Incorrect.
Contradictory.
Inappropriately formatted.
Excessively noisy.
Based on obsolete policies.
Unrelated to the target use case.

Poor-quality examples can teach the model undesirable behavior.

4.2 Remove duplicates

Duplicates can cause problems such as:

Overrepresenting certain examples.
Inflating apparent performance.
Causing data leakage between training and evaluation sets.
Making the model memorize specific content.

Deduplicate both exact copies and, where practical, substantially similar records.

4.3 Normalize data

Normalization may include:

Consistent spelling and formatting.
Standardized labels.
Consistent units.
Consistent capitalization.
Removal of unnecessary markup.
Standardized response structures.

Do not remove meaningful domain-specific terms simply to make data look uniform.

4.4 Protect data quality during transformation

Document transformations so the organization knows:

What was changed.
Why it was changed.
Which data was removed.
Which tools performed the transformation.
Which version of the dataset was used.

This supports reproducibility and auditability.

5. Data Governance for Fine-Tuning

Fine-tuning data may contain sensitive information. Governance requirements apply before, during, and after training.

5.1 Data ownership and permissions

Confirm that the organization:

Owns or is authorized to use the data.
Has permission to use the data for model training.
Understands licensing and contractual restrictions.
Has documented data owners.
Has approved the intended use.

Do not assume that data available on the internet is automatically permitted for training.

5.2 Sensitive and personal data

Identify and appropriately handle:

Personally identifiable information.
Credentials and secrets.
Financial information.
Health information.
Confidential business information.
Customer communications.
Proprietary source code.

Possible controls include:

Removing unnecessary sensitive data.
Masking or anonymizing fields.
Tokenizing identifiers.
Restricting access.
Encrypting data.
Applying retention and deletion policies.
Separating training data from production data.

5.3 Data residency and location

Check requirements for:

Where training data is stored.
Where customization jobs run.
Where logs and model artifacts are stored.
Whether data crosses Regions.
Whether backups or replicas are created.
Whether a third-party service receives the data.

Use approved AWS Regions and service configurations where geographic restrictions apply.

5.4 Access control

Apply least privilege to:

Data scientists.
Developers.
Training jobs.
Storage locations.
Model artifacts.
Evaluation data.
Logs.

Restrict who can:

Upload training data.
Start a customization job.
Modify datasets.
Access the resulting model.
Deploy the customized model.
Delete or export artifacts.

6. Data Size and Quality

6.1 Data volume

The amount of data required depends on:

The task.
The complexity of the domain.
The size and capability of the base model.
The diversity of expected inputs.
The quality of the examples.
The degree of behavior change required.

More data is not always better. A small, carefully curated dataset can be more useful than a large noisy dataset.

6.2 Quality over quantity

High-quality data should be:

Accurate.
Relevant.
Consistent.
Representative.
Properly labeled.
Aligned with the desired behavior.
Free of unnecessary sensitive data.

6.3 Avoid overfitting

Overfitting occurs when a model performs well on training data but poorly on new, unseen inputs.

Possible causes include:

Too little training data.
Too many training iterations.
Excessively repetitive examples.
A dataset that does not represent real-world usage.

Mitigations include:

More representative data.
Validation and test datasets.
Regular evaluation.
Appropriate training configuration.
Removing duplicate or overly similar examples.

6.4 Avoid underfitting

Underfitting occurs when the model has not learned the target task well enough.

Possible causes include:

Insufficient or poor-quality training data.
Unclear examples.
A mismatch between the model and the task.
Insufficient training.

The solution is not always “train longer.” The organization should first review data quality, task definition, model choice, and training configuration.

7. Data Labeling

Data labeling assigns the expected answer, category, or annotation to a training example.

Examples include:

Assigning a support category.
Identifying entities in a document.
Marking whether a response is safe.
Providing a preferred response.
Ranking multiple responses.
Identifying sentiment.
Defining the correct output format.

Good labeling practices

Define clear labeling instructions.
Use consistent categories.
Provide examples to labelers.
Use qualified domain experts when necessary.
Measure agreement between labelers.
Review disagreements.
Track label changes.
Remove ambiguous examples or document how they should be handled.

Label quality

Incorrect labels can teach the model the wrong behavior. Labelers should understand:

The task.
The business definitions.
Edge cases.
Privacy requirements.
Escalation procedures.

8. Representativeness

Fine-tuning data should represent the inputs and conditions the model will encounter in production.

Include variation in:

User wording.
Writing style.
Language.
Geography, where relevant.
Device or channel.
Document formats.
Customer types.
Common and uncommon cases.
Difficult or ambiguous cases.
Expected edge cases.

Why representativeness matters

A model trained only on ideal examples may fail when:

Users use informal language.
Documents contain errors.
Inputs are incomplete.
Different languages are used.
Users ask questions in an unexpected way.
Real-world cases do not match the training format.

Representative data also helps identify unfair or uneven performance across groups.

9. Training, Validation, and Test Data

Separate data into different sets.

Training set

Used to adjust the model during fine-tuning.

Validation set

Used during development to compare training configurations and detect problems such as overfitting.

Test set

Used for final evaluation on data that was not used to train or tune the model.

The test set should remain protected from the training process. If test examples appear in training data, reported performance may be misleading.

Evaluation data should include

Normal use cases.
Edge cases.
Failure cases.
Safety-related cases.
Ambiguous inputs.
Adversarial inputs.
Different languages or user groups, where applicable.

10. Preparing Instruction-Tuning Data

Instruction-tuning data generally contains examples of:

An instruction.
Optional context or input.
The desired output.

A conceptual record might look like this:

{
  "instruction": "Classify the customer request.",
  "input": "The customer was charged twice for the same order.",
  "output": "billing"
}

The exact file format and field names depend on the model and customization service. Always follow the required format for the selected model.

Best practices

Use clear instructions.
Ensure the expected response is correct.
Use consistent output formats.
Include representative examples.
Include difficult cases.
Avoid contradictory examples.
Do not include secrets or unnecessary personal data.
Keep the examples aligned with the intended production task.

11. Reinforcement Learning from Human Feedback

11.1 Purpose

RLHF aligns model behavior with human preferences rather than only matching a fixed answer.

It is useful when quality depends on preferences such as:

Helpfulness.
Tone.
Safety.
Relevance.
Completeness.
Following instructions.
Avoiding undesirable content.

11.2 High-level process

A simplified RLHF process is:

Select representative prompts.
Generate multiple candidate responses.
Ask human reviewers to rank or compare the responses.
Record preference data.
Use the preferences to estimate which outputs are better.
Optimize the model toward preferred behavior.
Evaluate the result for quality, safety, and unintended consequences.

11.3 Challenges

RLHF requires:

Human reviewers.
Clear review guidelines.
Quality control.
A representative prompt set.
Consistent preferences.
Significant time and cost.

Potential problems include:

Reviewer disagreement.
Reviewer bias.
Inconsistent labeling.
Over-optimization for the review criteria.
Reduced performance on cases not represented in the feedback.

Exam takeaway

RLHF uses human preference or ranking data to improve model behavior. It is different from ordinary supervised fine-tuning, which typically uses desired input-output pairs.

12. AWS Services and Features for Customization

Amazon Bedrock model customization

Amazon Bedrock provides managed capabilities for customizing supported foundation models.

Depending on the model and feature, customization may include:

Fine-tuning.
Continued pre-training.
Custom model import or related customization options.
Training from data stored in Amazon S3.
Creating a customized model artifact.
Evaluating and deploying the customized model.

Availability, supported data formats, hyperparameters, and customization methods vary by model.

Exam tip

The question may describe a company that wants to customize a supported FM without managing the underlying training infrastructure. Amazon Bedrock model customization is a likely choice.

Amazon SageMaker AI

Amazon SageMaker AI provides broader machine learning development and training capabilities.

It may be used when organizations need:

More control over training workflows.
Custom training code.
Specialized algorithms.
Data processing pipelines.
Model evaluation workflows.
Greater control over infrastructure and deployment.

Comparison

Requirement	Amazon Bedrock customization	Amazon SageMaker AI
Managed FM customization	Strong fit	Possible, but more developer-controlled
Custom training code	More limited and model-dependent	Strong fit
Infrastructure management	More abstracted	More configurable
Use of supported commercial FMs	Common use case	Possible through appropriate workflows
Complete ML pipeline control	More limited	Strong fit

For the AIF-C01 exam, the key concept is the tradeoff between a managed customization experience and a more flexible, developer-controlled training environment.

13. Training and Fine-Tuning Risks

Data leakage

Data leakage occurs when information from evaluation or test data is included in training data.

This can make performance appear better than it really is.

Memorization

A model may memorize training examples, particularly when data is duplicated, overly specific, or sensitive.

This creates risks of:

Reproducing private information.
Exposing proprietary content.
Failing to generalize to new inputs.

Bias

Biased or unrepresentative training data can cause biased outputs.

Review:

Data composition.
Labeling decisions.
Performance across relevant user groups.
Differences in error rates.
Sensitive attributes and proxies.

Catastrophic forgetting

A customized model may lose some general capabilities after being trained too heavily on narrow domain data.

Evaluate both:

Performance on the target task.
Retention of important general capabilities.

Model drift

The real-world data or business requirements may change after customization.

Monitor whether the model’s performance changes because of:

New terminology.
New products.
New policies.
New user behavior.
Changing regulations.
Changes in source data.

14. Exam Tips and Common Traps

Tip 1: Pre-training versus fine-tuning

Pre-training creates broad general capabilities.
Fine-tuning adapts an existing model to a more specific task or behavior.

Tip 2: Continuous pre-training versus fine-tuning

Continuous pre-training usually uses a larger domain corpus to broaden domain knowledge.
Fine-tuning usually uses task-specific examples to change behavior or performance.

Tip 3: RAG versus fine-tuning

Frequently changing or private facts → RAG.
Stable behavior, style, format, or task adaptation → Fine-tuning.

Tip 4: RLHF uses preferences

If humans rank or compare multiple responses, the technique is likely RLHF or preference-based alignment.

Tip 5: Distillation reduces operational cost

Distillation generally creates a smaller student model from a larger teacher model. It is useful when lower latency or inference cost is more important than retaining every capability of the larger model.

Tip 6: Data quality matters more than raw volume

The best answer is not always “collect more data.” Look for:

Accurate examples.
Good labels.
Representative inputs.
Deduplication.
Privacy controls.
Train, validation, and test separation.

Tip 7: Keep test data separate

Using test examples during training creates data leakage and invalidates the evaluation.

Tip 8: Fine-tuning data must be authorized

Data that can be accessed is not necessarily data that can legally or contractually be used for model training.

Tip 9: Training data can contain security risks

Review training data for:

Prompt-injection content.
Malicious instructions.
Secrets.
Personal information.
Incorrect labels.
Poisoned examples.

Tip 10: Customization does not eliminate evaluation

After fine-tuning, evaluate for:

Accuracy.
Safety.
Bias.
Generalization.
Regression in general capabilities.
Privacy leakage.
Robustness to unexpected inputs.

Practice Questions and Answers

## Question 1 Which training stage creates broad general-purpose capabilities by using a very large and diverse dataset? A. Fine-tuning B. Pre-training C. Prompt engineering D. Vector search **Answer: B. Pre-training** **Explanation:** Pre-training creates the general capabilities of a foundation model. Fine-tuning adapts an existing model to a more specific task or behavior. --- ## Question 2 A company wants its model to understand specialized medical terminology across a large collection of medical documents. Which approach is most appropriate? A. Continuous pre-training on suitable domain data B. Increasing the model temperature C. Using a shorter prompt D. Removing all domain-specific data **Answer: A. Continuous pre-training on suitable domain data** **Explanation:** Continuous pre-training can provide broader exposure to domain-specific language and terminology. It is more extensive and costly than narrow task-specific fine-tuning. --- ## Question 3 A company needs a model to produce responses in a consistent legal-document format using labeled examples. Which approach is most appropriate? A. Fine-tuning with high-quality task-specific examples B. Pre-training a new foundation model from scratch in every case C. Increasing temperature D. Deleting the expected outputs **Answer: A. Fine-tuning with high-quality task-specific examples** **Explanation:** Fine-tuning can adapt a model to a specialized task, format, or response style. --- ## Question 4 Which statement best describes transfer learning in the context of foundation models? A. Using knowledge learned by a pre-trained model to improve performance on a related specialized task B. Deleting all pre-trained model parameters C. Storing documents as vectors D. Changing the model’s temperature during inference **Answer: A. Using knowledge learned by a pre-trained model to improve performance on a related specialized task** **Explanation:** Fine-tuning a pre-trained FM for a specific task is an example of transfer learning. --- ## Question 5 An organization wants to improve a model’s helpfulness and safety by having reviewers rank multiple candidate responses. Which technique does this describe? A. Zero-shot prompting B. Reinforcement learning from human feedback C. Vector indexing D. Prompt caching **Answer: B. Reinforcement learning from human feedback** **Explanation:** RLHF uses human preference information, such as rankings or comparisons, to optimize model behavior. --- ## Question 6 Which dataset is most appropriate for fine-tuning a customer support classification model? A. A large dataset containing unrelated public articles with no labels B. A small but accurate and representative dataset of support requests with correct categories C. A dataset containing only one repeated support request D. A test dataset that will also be used for training **Answer: B. A small but accurate and representative dataset of support requests with correct categories** **Explanation:** Fine-tuning requires task-relevant, correctly labeled, and representative examples. The test set must remain separate from the training data. --- ## Question 7 What is a primary risk of including duplicate or highly similar records in a fine-tuning dataset? A. It guarantees better generalization. B. It can overrepresent certain examples and cause the model to memorize them. C. It eliminates the need for validation data. D. It automatically reduces training cost. **Answer: B. It can overrepresent certain examples and cause the model to memorize them.** **Explanation:** Duplicates can bias the training process, contribute to overfitting, and create misleading evaluation results. --- ## Question 8 A company has private product documentation that changes every week. It wants an FM to answer questions using the latest version without retraining the model each week. Which approach is generally most appropriate? A. RAG B. Pre-training C. Fine-tuning every week D. Model distillation **Answer: A. RAG** **Explanation:** RAG retrieves current information at inference time. Updating the knowledge source is generally easier and less costly than retraining the model frequently. --- ## Question 9 Which practice helps prevent data leakage during model evaluation? A. Include the evaluation data in the fine-tuning dataset. B. Use the same data for training and testing. C. Keep validation and test data separate from the training data. D. Use only duplicate examples in the test set. **Answer: C. Keep validation and test data separate from the training data.** **Explanation:** Separating data ensures that evaluation measures performance on unseen examples. --- ## Question 10 A company wants to reduce inference latency and cost after obtaining desired behavior from a large model. Which technique may help create a smaller model with similar behavior? A. Model distillation B. Increasing the context length C. Adding more examples to every prompt D. Continuous data replication **Answer: A. Model distillation** **Explanation:** Distillation transfers useful behavior from a larger teacher model to a smaller student model, potentially reducing inference cost and latency. --- ## Question 11 Which statement about fine-tuning data governance is correct? A. Any publicly accessible data can be used for training without review. B. Training data should be reviewed for authorization, privacy, security, quality, and retention requirements. C. Fine-tuning data does not require access controls. D. Data governance is required only after model deployment. **Answer: B. Training data should be reviewed for authorization, privacy, security, quality, and retention requirements.** **Explanation:** Training data may contain sensitive, restricted, or malicious content. Governance applies throughout collection, preparation, training, evaluation, and deployment.

Final Summary

Remember these core relationships:

Pre-training creates broad FM capabilities from very large datasets.
Fine-tuning adapts an existing FM to a specific task, domain, style, or format.
Instruction tuning improves the ability to follow natural-language instructions.
Continuous pre-training adds broader domain knowledge using additional domain data.
Transfer learning reuses learned capabilities for a related task.
RLHF uses human preferences to align model behavior.
Distillation creates a smaller, potentially faster and cheaper student model.
Fine-tuning data should be accurate, authorized, secure, representative, well labeled, and properly separated into training, validation, and test sets.
High-quality data is generally more valuable than simply collecting more data.
RAG is usually preferable for frequently changing external knowledge, while fine-tuning is usually better for stable behavior or task adaptation.