Skip to content

Skill 2.2.8: Apply customization techniques for AI solutions (for example, task-specific prompt engineering, fine-tuning)

Question 1 - Prompt Engineering vs. Fine-Tuning Selection

A financial services company is building a customer support chatbot using Amazon Bedrock with the Anthropic Claude model. The chatbot must handle account balance inquiries, transaction disputes, and loan application status questions. The team has noticed inconsistent response formatting and occasional off-topic answers. They have a compliance requirement to complete the solution quickly with minimal cost and no model training data management overhead.

Which customization technique should the team apply FIRST to address these issues?

A) Use Amazon Bedrock fine-tuning with labeled examples of support conversations to train the model on the company's specific response patterns.

B) Implement task-specific prompt engineering with explicit instructions, output format constraints, and few-shot examples in the prompt template stored in Amazon Bedrock Prompt Management.

C) Use Amazon Bedrock Reinforcement Fine-Tuning (RFT) with a reward function that scores responses based on compliance and formatting accuracy.

D) Deploy a custom SageMaker training job to fine-tune a smaller open-source model from SageMaker JumpStart on the company's support data.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: Prompt engineering is the recommended first-line customization approach for foundation models when dealing with output formatting inconsistencies and instruction adherence. Storing prompts in Bedrock Prompt Management allows for versioning, tracking, and systematic improvement without training data preparation or model training costs. It addresses the "minimal cost" and "no model training data management overhead" constraints directly.

Distractor Analysis

  • A is Incorrect: Fine-tuning requires preparing labeled training datasets, managing training jobs, and maintaining fine-tuned model versions. This introduces significant operational overhead and cost, violating the "minimal cost" and "no training data management" constraints when prompt engineering hasn't been exhausted first.
  • C is Incorrect: Reinforcement Fine-Tuning requires designing reward functions and preparing evaluation datasets, which is even more complex than supervised fine-tuning. This approach is premature when basic prompt engineering hasn't been attempted.
  • D is Incorrect: This introduces the highest operational overhead—custom SageMaker training infrastructure, model deployment, and ongoing maintenance of a separate model endpoint. It contradicts the requirement for minimal cost and quick implementation.

Key Takeaway: Prompt engineering is always the first customization technique to attempt for foundation models; only exhaust prompting before escalating to fine-tuning or RLHF approaches.

Question 2 - Supervised Fine-Tuning Data Requirements

A healthcare organization wants to customize Amazon Bedrock Titan Text model to generate medical coding suggestions from clinical notes. The team has collected 10,000 de-identified clinical notes paired with correct medical codes. They want to use Amazon Bedrock fine-tuning.

Which data preparation requirement must be satisfied for the fine-tuning job to succeed?

A) The training data must be stored in Amazon DynamoDB with a specific schema including prompt and completion fields.

B) The training data must be formatted as a JSONL file in Amazon S3, with each line containing a prompt and completion pair matching the model's expected structure.

C) The training data must be embedded using Amazon SageMaker Embeddings before submission to Bedrock.

D) The training data requires manual annotation by AWS certified data scientists before training can begin.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: Amazon Bedrock fine-tuning requires training data to be stored in Amazon S3 in JSONL (JSON Lines) format. Each line contains a complete prompt-completion pair. For Titan models specifically, the format follows the structure used for inference prompts, with clear delineation between input and expected output text.

Distractor Analysis

  • A is Incorrect: Amazon Bedrock fine-tuning does not read training data directly from DynamoDB. The data must be staged in S3 to be accessible by the Bedrock training pipeline.
  • C is Incorrect: Bedrock fine-tuning operates on raw text data and manages the conversion internally. Pre-embedding the training data would corrupt the training process, as fine-tuning works on token sequences, not embedded vectors.
  • D is Incorrect: While data quality and annotation accuracy are critical, AWS does not require AWS-certified data scientists to annotate the data. The organization can prepare its own training data using domain experts.

Key Takeaway: Amazon Bedrock fine-tuning requires JSONL files in S3 with prompt-completion pairs—not DynamoDB, pre-embedded vectors, or AWS-provided annotation services.

Question 3 - Reinforcement Fine-Tuning (RFT) Scenarios

A cybersecurity company wants to train an Amazon Bedrock model to generate security incident response playbooks. The quality of a playbook is measured by multiple factors: actionability (10-point scale), compliance with internal security policies (pass/fail), and technical accuracy (validated by an automated checker). The team has 500 example scenarios but no pre-written correct playbooks. They want the model to learn optimal response generation through iterative improvement.

Which customization approach best fits this scenario?

A) Supervised fine-tuning using the 500 scenarios with manually written playbooks created by security analysts.

B) Prompt engineering with detailed security policy documentation included in the system prompt.

C) Amazon Bedrock Reinforcement Fine-Tuning (RFT) using a reward function that evaluates actionability, compliance, and technical accuracy.

D) Model distillation from a larger model that already generates security playbooks.

Click to reveal Answer & Explanation

Correct Answer: C

Correct Option Analysis

  • C is Correct: Reinforcement Fine-Tuning (RFT) is specifically designed for scenarios where you have evaluation criteria (reward signals) but not necessarily ground-truth outputs. The team has clear quality metrics (actionability score, compliance pass/fail, technical accuracy) that can be encoded into a reward function. RFT allows the model to explore response generation and learn from the reward signals iteratively.

Distractor Analysis

  • A is Incorrect: Supervised fine-tuning requires labeled input-output pairs. The team explicitly lacks pre-written correct playbooks, so they cannot create the prompt-completion pairs needed for SFT.
  • B is Incorrect: While prompt engineering can provide policy context, it cannot train the model to iteratively improve based on quantitative quality metrics. Complex multi-factor quality evaluation is beyond prompt engineering capabilities.
  • D is Incorrect: Model distillation requires a larger teacher model that already performs the task well. The scenario doesn't mention having access to such a model, and distillation primarily transfers existing capabilities rather than improving through reward optimization.

Key Takeaway: Choose Reinforcement Fine-Tuning when you have clear evaluation criteria (reward signals) but lack ground-truth outputs for supervised fine-tuning.

Question 4 - Distillation for Cost Optimization

A media company currently uses a large proprietary model through Amazon Bedrock to generate video metadata tags from descriptions. The model is very accurate but costs $0.05 per request. With 10 million requests per month, the cost is substantial. The company can tolerate slightly lower accuracy but requires lower inference costs and faster response times.

Which customization technique should the company use?

A) Use Amazon Bedrock Model Distillation to create a smaller, faster student model that replicates the teacher model's tagging behavior using a managed prompt set.

B) Use supervised fine-tuning on the existing large model to make it more efficient at generating tags.

C) Switch to a different large model on Bedrock with lower per-token pricing.

D) Implement caching of common video descriptions to reduce the number of Bedrock API calls.

Click to reveal Answer & Explanation

Correct Answer: A

Correct Option Analysis

  • A is Correct: Amazon Bedrock Model Distillation is specifically designed for this use case—reducing inference costs and latency by training a smaller student model to replicate the behavior of a larger teacher model. The managed distillation process takes a prompt set and generates teacher responses automatically, then distills these into a student model that shows significantly lower latency and cost. The scenario explicitly mentions tolerance for slightly lower accuracy, which aligns with the typical trade-off in distillation.

Distractor Analysis

  • B is Incorrect: Supervised fine-tuning on the same large model does not reduce the model's size or inference cost—it only customizes the output behavior. The per-request cost remains the same because the infrastructure requirements are unchanged.
  • C is Incorrect: While switching to a lower-priced model might reduce cost, it's not a customization technique and doesn't provide the systematic trade-off of preserving specific task performance while reducing latency. Distillation creates a purpose-built student model optimized for this specific tagging task.
  • D is Incorrect: Caching only helps for duplicate or highly similar requests. Video descriptions may vary significantly, limiting cache hit rates. This approach sidesteps the actual model optimization need and doesn't address response latency for novel content.

Key Takeaway: Model distillation is the optimal Bedrock customization technique when the goal is reducing inference cost and latency, and minor accuracy loss is acceptable.

Question 5 - Catastrophic Forgetting Prevention

An e-commerce company is fine-tuning Amazon Bedrock Amazon Titan model on their internal customer service transcripts to improve support-specific responses. However, they want to maintain the model's general language understanding and broad knowledge capabilities. During the fine-tuning process, they observe the model becoming highly specialized but losing its ability to handle general queries.

Which TWO strategies should the company implement to prevent catastrophic forgetting? (Select TWO)

A) Increase the learning rate during fine-tuning to accelerate convergence on the specialized data.

B) Blend general-domain data with the specialized customer service transcripts in the fine-tuning dataset.

C) Use a lower epoch count and monitor validation loss on a held-out general language understanding benchmark.

D) Remove the base model's original knowledge by training exclusively on the customer service data to maximize specialization.

E) Use only 10 customer service examples to minimize the impact on the base model's general capabilities.

Click to reveal Answer & Explanation

Correct Answers: B and C

Correct Options Analysis

  • B is Correct: Mixing general-domain data with specialized fine-tuning data is the standard defense against catastrophic forgetting. The blended dataset allows the model to maintain general language understanding while learning the specialized domain patterns.
  • C is Correct: Limiting training epochs and monitoring performance on a general language benchmark helps detect early signs of catastrophic forgetting. Lower epochs mean fewer gradient updates that could overwrite general capabilities.

Distractor Analysis

  • A is Incorrect: Increasing the learning rate accelerates weight changes, making catastrophic forgetting worse by more aggressively overwriting general knowledge with specialized patterns.
  • D is Incorrect: Training exclusively on specialized data is the primary cause of catastrophic forgetting—it removes the general knowledge signal entirely from the training process.
  • E is Incorrect: While using fewer examples reduces catastrophic forgetting risk, 10 examples would provide insufficient training signal for the model to learn the specialized domain (this would lead to underfitting, not successful adaptation).

Key Takeaway: Prevent catastrophic forgetting by blending general-domain data into fine-tuning datasets and limiting epochs while monitoring performance on general tasks.

Question 6 - Prompt Management and Versioning

A software company develops an AI assistant for code review using Amazon Bedrock. The team has developed a prompt template that includes system instructions, code context variables, and few-shot examples. As they experiment with different prompt structures, they need to track versions, compare prompt performance across iterations, and roll back to previous versions if new prompts perform worse.

Which AWS service feature should the team use?

A) Store prompt templates in AWS Systems Manager Parameter Store with version labels.

B) Use Amazon Bedrock Prompt Management to create, version, and compare prompt performance across iterations.

C) Embed prompt templates directly in the application code using Git for version control.

D) Use Amazon SageMaker Feature Store to store prompt versions as feature groups.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: Amazon Bedrock Prompt Management is purpose-built for storing, versioning, and managing prompt templates. It allows teams to create named versions, compare performance across prompt iterations, and roll back to previous versions. This directly addresses all three requirements (track versions, compare, rollback) with a managed service.

Distractor Analysis

  • A is Incorrect: While Parameter Store can technically store prompt strings with version labels, it is not designed for prompt management workflows. It lacks native prompt comparison features, A/B testing capabilities, and direct integration with Bedrock model invocations.
  • C is Incorrect: Embedding prompts in application code couples the prompt to the codebase and requires code deployments for prompt changes. It lacks the ability to compare prompt performance at runtime or roll back without a full code deployment.
  • D is Incorrect: SageMaker Feature Store is designed for machine learning feature data (structured data points for model training), not for storing conversation prompt templates.

Key Takeaway: Amazon Bedrock Prompt Management provides native versioning, comparison, and rollback for prompt templates—essential for systematic prompt iteration and deployment.

Question 7 - Task-Specific Prompt Engineering Patterns

A legal research company is building an AI assistant using Amazon Bedrock that helps attorneys find relevant case law. The assistant receives a legal question and must output a structured analysis with three sections: "Relevant Precedents", "Legal Reasoning", and "Recommended Next Steps". The model often deviates from this structure or adds unsolicited legal advice.

Which prompt engineering technique will MOST effectively enforce the required output structure?

A) Increase the temperature parameter to encourage more creative responses that naturally produce varied structures.

B) Add a system prompt instruction specifying the required three-section structure, along with a few-shot example demonstrating the exact output format.

C) Reduce the number of input tokens to force the model to produce shorter, more structured outputs.

D) Use a larger foundation model on Bedrock that inherently follows output formatting instructions better.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: Explicit output format instructions in the system prompt, combined with few-shot examples showing the exact structure, is the most effective prompt engineering technique for enforcing consistent output formatting. The few-shot examples provide concrete demonstrations of the expected pattern, which significantly improves format adherence compared to instructions alone.

Distractor Analysis

  • A is Incorrect: Increasing temperature increases randomness and variability in the response, which would make format deviations MORE likely, not less likely.
  • C is Incorrect: Reducing input tokens (context length) has no direct relationship to output structure adherence. It may even hurt performance by removing relevant context needed for the analysis.
  • D is Incorrect: While some models are better at instruction following, switching models is not a prompt engineering technique and may introduce unnecessary cost or complexity. The team should optimize the prompt first before changing models.

Key Takeaway: Combine explicit format instructions with few-shot examples to enforce output structure; this costs nothing and improves adherence dramatically.

Question 8 - Retrieval Optimization vs. Model Fine-Tuning

A manufacturing company has implemented a Retrieval-Augmented Generation (RAG) system using Amazon Bedrock Knowledge Bases with OpenSearch Serverless. The system answers questions about equipment maintenance procedures. Users report that the answers often reference the wrong equipment model or miss critical safety warnings, even when the correct documentation exists in the knowledge base. The model generates fluent but factually incorrect responses.

What is the MOST appropriate first action to address this problem?

A) Fine-tune the foundation model using Amazon Bedrock fine-tuning with the company's maintenance documentation.

B) Optimize the chunking strategy and embedding model in Amazon Bedrock Knowledge Bases to improve retrieval relevance.

C) Increase the temperature setting on the model to generate more creative responses.

D) Use prompt engineering to instruct the model to "be more accurate" about equipment specifications.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Option Analysis

  • B is Correct: The symptom described—fluent but factually incorrect answers that exist in the knowledge base—indicates a retrieval problem. The passages being retrieved are not the relevant ones containing the correct information. Optimizing chunking (breaking documents into appropriately sized segments) and selecting a better embedding model directly improves retrieval relevance, which then gives the generator correct context.

Distractor Analysis

  • A is Incorrect: Fine-tuning the model does not fix retrieval issues. The model is generating fluent text but receiving wrong context. No amount of fine-tuning can produce correct answers without correct retrieved passages.
  • C is Incorrect: Increasing temperature increases randomness and would likely generate MORE hallucinated or incorrect content, making the factual accuracy problem worse.
  • D is Incorrect: Prompt instructions cannot fix missing or incorrect context. Even with perfect instructions, the model cannot access information that wasn't retrieved from the knowledge base.

Key Takeaway: In RAG systems, always first diagnose whether the problem is retrieval (wrong passages) or generation (wrong response from correct passages)—they require different fixes.

Question 9 - Reinforcement Fine-Tuning vs. Supervised Fine-Tuning

A logistics company wants to customize a foundation model on Amazon Bedrock to generate route optimization recommendations. The team has two types of data:

  • 5,000 historical route plans written by human dispatchers (input-output pairs)
  • A simulator that can evaluate any generated route plan and return a score based on delivery time, fuel efficiency, and on-time delivery rate

The team wants to maximize the quality of route recommendations as measured by the simulator score, while also leveraging the historical human expertise.

Which TWO customization approaches should the team use in sequence for optimal results? (Select TWO)

A) First apply supervised fine-tuning using the 5,000 historical route plans to teach the model to generate reasonable baseline plans.

B) Skip supervised fine-tuning and go directly to reinforcement fine-tuning using the simulator as the reward signal.

C) After supervised fine-tuning, apply reinforcement fine-tuning using the simulator-based reward function to further optimize route quality.

D) Use prompt engineering with the simulator score included as a system prompt instruction.

E) Use model distillation from a larger model that generates route plans, ignoring the simulator entirely.

Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: Supervised fine-tuning on historical data teaches the model the fundamental patterns of route planning, providing a solid baseline. This leverages the human expertise embedded in the 5,000 historical plans.
  • C is Correct: After establishing baseline performance through SFT, reinforcement fine-tuning using the simulator reward function allows the model to explore and optimize beyond human-level performance. This sequential approach leverages both data types optimally.

Distractor Analysis

  • B is Incorrect: Skipping supervised fine-tuning ignores the valuable historical data and may result in slower convergence during RFT, as the model lacks basic route planning patterns.
  • D is Incorrect: Simulator scores cannot be passed as prompt instructions during inference—the model cannot process external evaluation tools through prompts alone. This is a fundamental misuse of the prompt channel.
  • E is Incorrect: Distillation from a larger model doesn't leverage the simulator's evaluation capability. Additionally, no larger model for this specific task is mentioned as available.

Key Takeaway: Supervised fine-tuning followed by reinforcement fine-tuning leverages both historical examples and simulation-based quality signals for maximum optimization.

Question 10 - Fine-Tuning Data Quality and Validation

A travel booking company is preparing data to fine-tune a foundation model on Amazon Bedrock for generating personalized travel itineraries. The team has collected 50,000 customer interaction logs that include:

  • Questions and answers from chats with travel agents
  • Internal system commands embedded in the agent responses
  • Duplicated conversations from multiple logging systems
  • Some incomplete interactions that were abandoned mid-conversation

Which TWO data preparation steps are MOST critical for successful fine-tuning? (Select TWO)

A) Keep all 50,000 interactions intact to maximize training data volume.

B) Remove internal system commands from the agent responses to create clean prompt-completion pairs.

C) Deduplicate conversations that appear multiple times in the logging systems.

D) Convert all interactions to PDF format for efficient S3 storage and Bedrock processing.

E) Include abandoned interactions to help the model learn what NOT to do.

Click to reveal Answer & Explanation

Correct Answers: B and C

Correct Options Analysis

  • B is Correct: Internal system commands are noise in the training data that would teach the model to generate non-useful content. Cleaning these from the agent responses ensures the model learns proper customer-facing language without system artifacts.
  • C is Correct: Deduplication prevents the model from over-emphasizing repeated examples and artificially biasing the training distribution. Duplicates also waste training compute without adding new information.

Distractor Analysis

  • A is Incorrect: Keeping all data without cleaning introduces noise (system commands, duplicates, incomplete interactions) that degrades model quality. Data quality often matters more than raw volume, especially when 50,000 examples is already substantial.
  • D is Incorrect: Bedrock fine-tuning requires JSONL format in S3, not PDF. Converting to PDF would make the data unusable for the fine-tuning pipeline.
  • E is Incorrect: Abandoned interactions represent incomplete prompt-completion pairs without a proper target output. They teach the model nothing useful about generating complete itineraries and add noise to the training signal.

Key Takeaway: Data quality (cleaning, deduplication, completeness) is as critical as data volume for fine-tuning success; always prepare clean prompt-completion pairs in JSONL format.