Skip to content

Skill 1.2.9: Prepare data for FM fine-tuning, continuous pre-training, and model distillation

Task 1.2: Data Preparation for ML and AI — Perform data transformation, feature engineering, and pre-processing

Objective: Produce the correct artifact for the chosen Bedrock model-customization path. The exam tests artifact selection far more than any single transform — the three paths consume different data and answer different business needs.

The Three Customization Paths

Path Artifact you prepare Supervision What it changes Use when
Fine-tuning (SFT) Paired prompt → response examples Labeled Behavior: tone, format, task adherence, structured output; weights updated Stable behavior, instruction following, domain tasks with hundreds+ good examples
Continued (continuous) pre-training Large corpus of unlabeled domain text Unlabeled / self-supervised Vocabulary, jargon, writing patterns of a field Domain adaptation (legal, medical, financial); base model lacks domain language
Model distillation Prompt set covering the use case (teacher responses are generated for you) Auto-labeled by teacher Smaller/cheaper student mimics larger teacher Cost + latency reduction while retaining teacher quality

Hard rule: Not every model supports every customization type. Confirm model support before designing the pipeline — a schema-perfect dataset for an unsupported model still fails. Bedrock distillation is offered for Amazon Nova (large teacher → small student); Titan/Llama/Mistral/Cohere families expose fine-tuning with provider-specific schemas.

Artifact Schemas & Formats

Path customizationType Record shape (per line)
Fine-tuning FINE_TUNING Provider schema: Titan inputText/outputText; Llama prompt/completion; Nova schemaVersion + messages[] with user/assistant roles
Continued pre-training CONTINUED_PRE_TRAINING Single unlabeled text field per record; very large corpus
Distillation DISTILLATION Prompt only (+ optional teacher prompt template / generation config); Bedrock invokes the teacher, captures responses, then fine-tunes the student
  • Format: JSONL (.jsonl or .jsonl.gz), UTF-8, one valid JSON object per line, no trailing commas, no BOM.
  • Split: Disjoint trainingDataConfig and validationDataConfig; never sample validation from training.
  • Failure feedback: Failed records are written to a validation error file (recordNumber + errorMessage). Fix and resubmit — Bedrock fails the job fast on schema violations rather than skipping rows.

Job Configuration & Guardrails

  • Key parameters: customizationType, baseModelIdentifier, customModelName, roleArn, trainingDataConfig, validationDataConfig, outputDataConfig.s3Uri, hyperParameters (epochCount, batchSize, learningRate, learningRateWarmupSteps), vpcConfig, kmsKeyId.
  • Hyperparameter trade-offs: epochCount too high → overfitting and memorized training prompts; learningRate too high → divergence/failed job; batchSize is constrained by model and instance.
  • Region: Data, job, and output must live in the same Region; cross-Region S3 URIs are rejected.
  • VPC jobs: Need an S3 gateway endpoint — the job cannot reach the internet.
  • Serving: Custom models are not available on-demand; you must purchase Provisioned Throughput (the no-commitment option) to host them.
  • Evaluate before cutover: Use Bedrock model evaluation jobs (automatic metrics, LLM-as-a-judge, human) to compare base vs. custom model on a held-out set before shifting production traffic.

Decision Flow

Knowledge that changes often? ──────────────► RAG / Knowledge Bases (re-index, no retraining)
Stable behavior, tone, format? ─────────────► Prompting first → then FINE_TUNING
New domain vocabulary, lots of raw text? ───► CONTINUED_PRE_TRAINING
Same quality, smaller/cheaper model? ───────► DISTILLATION (prepare prompts, not answers)

Security, Privacy & Compliance

  • Redact before training, never after — anything left in the data can surface in model output. Use Comprehend (PII entity detection) for text, Macie to locate sensitive data across the S3 estate, DataBrew/Glue to apply masking recipes at ingestion.
  • Mask vs. redact vs. anonymize: mask hides but preserves shape; redaction removes; anonymization is irreversible so an individual can no longer be identified.
  • KMS: Custom model artifacts and job output are encrypted; a customer managed key requires kms:CreateGrant/Decrypt on the job role.
  • IAM: The customization job role needs S3 read on training/validation and S3 write on output; distillation additionally needs permission to invoke the teacher model.
  • Prompts submitted for distillation are stored — do not place secrets in them.

Exam gotcha: Labeling prompt/response pairs and then pointing that dataset at continued pre-training wastes the labeling effort — CPT does not use responses as targets.

Exam gotcha: Changing the prompt template at inference (whitespace, system text, role ordering) from the one used in training causes the same class of defect as refitting a scaler — training-serving skew. The template is part of the model contract.

Common Traps

Trap Why it fails Correct move
Fine-tuning to add frequently changing facts Weights go stale; re-train on every change RAG / Knowledge Bases
CPT to fix tone or instruction following Expensive and unreliable behavior change Prompting → SFT
Distillation to inject proprietary knowledge Distills capability, not your facts RAG (+ SFT if behavior needed)
Fine-tuning with a few dozen examples Overfits, no generalization More/better data, or prompting
One huge record per document Silent truncation at max token length Chunk; cap tokens per record
Duplicate/contradictory prompt pairs Model learns inconsistent outputs Dedupe and resolve conflicts first

Pre-Submission Quality Gates

Dedupe near-identical prompts · balance intent/category coverage · cap token length per record · verify label correctness and consistent response format · hold out a representative validation split · scan for PII/toxic content · confirm model supports the chosen customization type in that Region.