Fine-tuning, RLHF & Model Alignment

Parameter-Efficient Fine-Tuning Methods

Parameter-efficient fine-tuning (PEFT) is the practical middle ground between prompt engineering and full fine-tuning. The core idea: update 0.1–1% of parameters while keeping 99%+ frozen. The pre-trained model already encodes most of what you need; PEFT introduces a small number of new parameters that steer the frozen backbone toward target behaviour. Benefits include drastically reduced memory and compute, negligible risk of catastrophic forgetting, and the ability to maintain multiple adapters on a single shared base model.

LoRA & QLoRA

LoRA (Low-Rank Adaptation) is based on the observation that the weight updates produced by fine-tuning have low intrinsic rank — most of the useful adaptation can be captured in a much smaller subspace than the full weight matrix. Instead of updating the full weight matrix W ∈ ℝ^(d×k), LoRA represents the update as the product of two low-rank matrices: ΔW = BA, where B ∈ ℝ^(d×r) and A ∈ ℝ^(r×k), with r ≪ min(d, k). The original weights are frozen; only A and B are trained.

Practical implications: for a 7B model, a rank-16 LoRA targeting the attention layers reduces trainable parameters from 7 billion to roughly 7 million — a 1000x reduction. At inference time, the adapter can be merged into the base weights (W' = W + BA) producing zero additional latency. LoRA adapters are also tiny on disk (tens of megabytes vs. tens of gigabytes for the full model), enabling easy sharing and multi-tenant serving.

Key hyperparameters: The rank r controls the capacity of the adapter (higher = more expressive but more parameters). The scaling factor α (typically set to 2r) controls the effective learning rate of the adapter. Target modules are typically the query and value projections of attention (q_proj, v_proj), though extending to all linear layers improves quality at modest cost. LoRA dropout (0.05–0.1) provides regularisation for small datasets.

QLoRA extends LoRA to 4-bit quantised base models. The frozen base model weights are stored in NF4 (4-bit NormalFloat) format with double quantisation, reducing memory by ~8x compared to FP16. The LoRA adapters themselves are trained in BF16 with paged optimisers that move optimizer states to CPU RAM when they overflow GPU VRAM. QLoRA enables fine-tuning of 65B parameter models on a single 48GB GPU — a task that would otherwise require 8+ A100s.

Adapters & Prefix Tuning

Adapter layers insert small bottleneck MLP modules after each attention and feed-forward sub-layer. They project from the hidden dimension down to a smaller bottleneck (e.g., 64 units), apply a non-linearity, and project back up. Only adapter parameters are trained. The downside: unlike LoRA, adapters cannot be merged into base weights, so they add a small inference latency overhead.

Prefix tuning prepends a set of learnable "soft token" vectors to the key and value sequences in every attention layer. These vectors are never decoded; they act as a persistent context that steers attention patterns. Prefix tuning is highly parameter-efficient but is limited in expressiveness — it can shift style and format but struggles with tasks requiring new factual knowledge.

Prompt tuning operates only at the input embedding layer: a small set of learnable vectors is prepended to the input embeddings. At scale (11B+ models), prompt tuning approaches full fine-tuning quality; at 7B and below, it significantly underperforms LoRA. BitFit takes a different approach, training only the bias terms of the model — effectively free in terms of parameter count (biases are <0.1% of total parameters) but limited in expressiveness.

In practice, LoRA has become the default PEFT method due to its favourable accuracy-efficiency trade-off, zero inference overhead after merging, and broad library support (HuggingFace PEFT, Axolotl, LLaMA-Factory). The multi-task serving pattern — a shared frozen base with hot-swappable per-task adapters — is increasingly used in production to serve dozens of specialised models from a single GPU cluster.

Instruction Tuning

Instruction tuning (also called supervised fine-tuning, or SFT) is the stage that transforms a raw pre-trained model — which simply predicts the next token — into an assistant that follows natural language instructions. The model is trained on a dataset of (instruction, response) pairs using the standard next-token prediction loss, but only the response tokens contribute to the loss gradient. This is a qualitative behavioural shift: the model learns that when it sees an instruction, the expected behaviour is to fulfil it helpfully and completely.

The data format matters enormously. Most modern models use a structured chat template with explicit role boundaries. The ChatML format (<|im_start|>system\n...<|im_end|>) or the Llama 3 template (<|begin_of_text|><|start_header_id|>system<|end_header_id|>...<|eot_id|>) encode the system prompt, user turn, and assistant response as distinct segments. Mixing formats across training examples is a common failure mode that produces inconsistent output formatting.

Dataset Construction

The canonical open SFT datasets include FLAN (task-specific instruction collections across 60+ NLP benchmarks), Dolly 15K (15,000 human-written instruction-response pairs from Databricks employees), OpenAssistant Conversations (161,000 messages in 35 languages with ranked responses), and the ShareGPT corpus (millions of real ChatGPT conversations scraped from the web).

Synthetic data generation has become standard practice. The Self-Instruct method (Wang et al., 2022) uses a strong LLM to generate new instruction-response pairs from a seed set, bootstrapping the dataset with minimal human effort. Evol-Instruct (used in WizardLM and OpenHermes) applies an LLM to progressively rewrite instructions to be more complex, diverse, and specific. Orca-style data distils reasoning chains from GPT-4, training smaller models to produce step-by-step explanations rather than just final answers.

The critical finding from LIMA (Zhou et al., 2023) is that data quality dominates data quantity: 1,000 carefully curated, diverse, high-quality examples can match the SFT quality of tens of thousands of noisier examples. Deduplication, length calibration (mixing short and long responses), and instruction diversity (not just Q&A but also coding, creative writing, summarisation, classification) are more important than raw dataset size.

SFT Best Practices

Practical guidance for SFT that practitioners frequently discover the hard way:

• Format consistency above all: A single format mismatch in 1% of your data can cause systematic output formatting failures. Validate every example with a parser before training.
• Pack sequences efficiently: Concatenate multiple short examples into single training sequences (separated by EOS tokens) to maximise GPU utilisation. Without packing, a 2048-token context is typically 30–40% empty.
• Train on responses only: Mask loss on prompt tokens. Training on both prompt and response teaches the model to complete prompts, which is different from following them.
• Watch for domain imbalance: If 80% of your data is code Q&A, the model will regress on general instruction following. Maintain category diversity proportional to expected production traffic.
• Evaluate on held-out instructions: SFT benchmark suites like MT-Bench and AlpacaEval provide LLM-judged evaluation of instruction-following quality across eight categories (writing, roleplay, reasoning, math, coding, extraction, STEM, humanities).

RLHF & Alignment

Reinforcement Learning from Human Feedback (RLHF) is the alignment technique behind InstructGPT, ChatGPT, Claude, Gemini, and every major production AI assistant. The fundamental problem it addresses: a model can follow instructions while simultaneously being unhelpful, deceptive, or harmful. Instruction tuning teaches the format; alignment teaches the values. RLHF introduces a training signal derived from human preference judgements rather than ground-truth labels.

The canonical RLHF pipeline operates in three stages. First, supervised fine-tuning (SFT) on a curated instruction dataset produces the SFT baseline — a well-behaved starting point. Second, reward model training uses human preference data to learn a scoring function. Third, PPO optimisation updates the SFT model's policy to maximise reward model scores while staying close to the SFT baseline via a KL divergence penalty.

Reward Modeling

The reward model (RM) is a neural network that takes a (prompt, response) pair and outputs a scalar score representing how much a human would prefer that response. It is typically initialised from the SFT model with the final token prediction head replaced by a linear scalar head.

Training data consists of pairwise preference comparisons: for a given prompt, human annotators choose which of two model responses they prefer. The Bradley-Terry model converts these pairwise comparisons into a ranking objective: the RM is trained to assign higher scores to preferred responses than rejected ones, minimising a cross-entropy loss over preference pairs.

Reward hacking is the primary failure mode: the policy learns to game the reward model rather than genuinely improving quality. Common manifestations include verbosity (longer responses score higher even when unhelpful), sycophancy (the RM was trained on human preferences that reward agreement), and formatting artifacts (bullet points and headers score higher regardless of content quality). Mitigation strategies include using an ensemble of reward models, adding a KL penalty against the SFT reference, and periodic reward model re-training on updated preference data.

Constitutional AI (CAI) and RLAIF (RL from AI Feedback) reduce dependence on expensive human annotation by using a strong LLM (Claude, GPT-4) to generate preference labels according to a set of principles (the "constitution"). This scales preference data collection dramatically but introduces the biases and failure modes of the judge model.

PPO & the RLHF Pipeline

Proximal Policy Optimisation (PPO) is the RL algorithm used to optimise the policy against the learned reward model. At each training step: (1) the policy generates a batch of responses to prompts from the preference data distribution; (2) the RM scores each response; (3) PPO computes the advantage estimate and updates the policy weights to increase the probability of high-reward responses. The clipped surrogate objective prevents excessively large updates that could destabilise training.

The KL divergence penalty is critical: without it, PPO rapidly drives the policy away from the SFT initialisation toward reward-hacking degenerate outputs. The penalty is applied as: R_total = R_RM - β * KL(π_θ || π_ref), where π_ref is the frozen SFT reference model. β is a hyperparameter controlling the trade-off between alignment and staying close to SFT behaviour.

The practical challenge of RLHF-PPO is the four-model memory requirement: the active policy (trainable), the frozen SFT reference (for KL computation), the reward model, and a critic model (value function for PPO advantage estimation). For 7B+ models, this requires a multi-GPU setup even with quantisation. This memory overhead is a primary driver for the adoption of simpler alignment alternatives like DPO.

Direct Preference Optimization

Direct Preference Optimization (DPO, Rafailov et al., 2023) achieves comparable alignment quality to RLHF while eliminating both the reward model and the PPO optimiser. The mathematical insight: the optimal RLHF policy can be expressed as a closed-form function of preference data, which means we can directly optimise preference probabilities without ever training a separate reward model or running online rollouts.

The DPO loss directly trains the policy to assign higher likelihood to preferred responses and lower likelihood to rejected responses, relative to a reference (SFT) model. The reference model prevents the policy from collapsing — without it, the model would simply assign near-zero probability to rejected responses rather than genuinely improving chosen response quality. Practical advantages: trains as fast as SFT (one forward-backward pass per preference pair), stable without careful hyperparameter tuning, no four-model memory overhead, and no reward hacking from a separate RM.

DPO vs. RLHF

The two approaches differ on several dimensions relevant to practitioners:

• Training complexity: DPO is equivalent to SFT in implementation complexity. RLHF-PPO requires a custom training loop with four model copies, online generation, reward scoring, and GAE advantage estimation.
• Memory: DPO needs two model copies (policy + reference). RLHF needs four (policy + reference + RM + critic).
• Stability: DPO converges reliably without reward model hyperparameter sensitivity. PPO is notoriously sensitive to learning rate, KL coefficient, and batch size.
• Data: DPO is offline — trained on a fixed preference dataset. PPO-based RLHF can collect on-policy preference data as the model improves, which is theoretically superior for continuous alignment.
• Alignment quality: DPO and RLHF-PPO achieve similar MT-Bench scores at 7B scale. At 70B+ scale with large preference datasets and on-policy data collection, RLHF maintains a quality edge.

Practical Implementation

The DPO family has expanded significantly since the original paper. IPO (Identity Policy Optimisation) adds a regularisation term that makes training robust to small or noisy preference datasets. SimPO removes the reference model entirely, using response length-normalised likelihood as an implicit reward. KTO (Kahneman-Tversky Optimisation) trains on unpaired binary feedback (good/bad labels) rather than preference pairs, enabling use of simpler annotation schemes. ORPO merges the SFT and alignment training stages into a single pass, reducing total training time.

Practical guidance: start with DPO on 5,000–50,000 high-quality preference pairs; only move to PPO-based RLHF if DPO demonstrably underperforms on your evaluation suite and you have the infrastructure to support four-model training. Preference data quality matters more than quantity: a well-calibrated annotator team producing 10,000 pairs will outperform noisy LLM-generated preference labels at 100,000 pairs.

Fine-Tuning & Alignment Method Comparisons

Choosing the right fine-tuning and alignment method requires weighing multiple factors simultaneously. The following tables provide a structured comparison across the most important dimensions.

Fine-Tuning Methods Comparison

Model Alignment Techniques Comparison

Code: LoRA Fine-Tuning with PEFT

The following demonstrates a complete LoRA fine-tuning run using HuggingFace PEFT and TRL's SFTTrainer. This setup fine-tunes Mistral-7B on a medical QA dataset using only 0.18% of the model's parameters on a single GPU.

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer
import torch

# Load base model (4-bit quantized to fit on single GPU)
model_id = "mistralai/Mistral-7B-v0.1"
model = AutoModelForCausalLM.from_pretrained(
    model_id, load_in_4bit=True, device_map="auto",
    torch_dtype=torch.bfloat16
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# LoRA configuration: inject rank-16 adapters into attention layers only
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,                    # rank of update matrices (trade-off: quality vs params)
    lora_alpha=32,           # scaling factor (lora_alpha/r = effective learning rate)
    target_modules=["q_proj", "v_proj"],  # which attention matrices to adapt
    lora_dropout=0.05,
    bias="none"
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# → trainable params: 6,815,744 || all params: 3,758,071,808 || trainable%: 0.18%

# SFTTrainer: fine-tune on instruction-formatted dataset
trainer = SFTTrainer(
    model=model, tokenizer=tokenizer,
    train_dataset=train_dataset,  # {"text": "<s>[INST] {instruction} [/INST] {response}</s>"}
    max_seq_length=2048,
    args=TrainingArguments(
        output_dir="./lora-medical-qa",
        num_train_epochs=3,
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,  # effective batch = 16
        learning_rate=2e-4,
        lr_scheduler_type="cosine",
        warmup_ratio=0.05,
        save_steps=100, logging_steps=25
    )
)
trainer.train()
# Training cost: ~$5 on A100 for 1K examples vs $500+ full fine-tune

Code: DPO Training

DPO requires preference pairs — (prompt, chosen_response, rejected_response) triplets. Here we demonstrate the full DPO training loop using TRL's DPOTrainer, which handles the reference model computation and the DPO loss automatically.

from trl import DPOTrainer, DPOConfig
from datasets import Dataset

# DPO: directly optimize preferences without separate reward model
# Dataset format: (prompt, chosen_response, rejected_response) triplets
preference_data = Dataset.from_dict({
    "prompt": [
        "Explain photosynthesis to a 10-year-old.",
        "Write a professional email declining a meeting."
    ],
    "chosen": [
        "Plants make their own food using sunlight! Here's how it works: leaves capture sunlight, roots drink water, and together they make sugar the plant uses for energy. They also release oxygen — that's the air we breathe!",
        "Thank you for the invitation. Unfortunately, I have a prior commitment at that time and won't be able to attend. I'd welcome the opportunity to connect at a future date."
    ],
    "rejected": [
        "Photosynthesis is the process by which organisms containing the pigment chlorophyll convert light energy into chemical energy stored in glucose.",
        "I can't make it sorry. Maybe another time."
    ]
})

dpo_config = DPOConfig(
    beta=0.1,  # KL divergence penalty: higher = stay closer to reference model
    max_prompt_length=512,
    max_length=1024,
    num_train_epochs=1,
    learning_rate=5e-7,  # very small LR for preference alignment
    per_device_train_batch_size=2
)

trainer = DPOTrainer(
    model=model,
    ref_model=None,  # None = use model itself as reference (PEFT-compatible)
    args=dpo_config,
    train_dataset=preference_data,
    tokenizer=tokenizer
)
trainer.train()
# DPO advantage over RLHF: no separate reward model, more stable training, simpler

Code: Instruction Tuning Dataset Format

High-quality instruction tuning datasets use the multi-turn chat format with explicit role markers. The following shows the OpenAI-compatible messages format used by most modern SFT pipelines, with a domain-specific medical coding example.

[
  {
    "messages": [
      {"role": "system", "content": "You are a medical coding specialist following ICD-11 standards."},
      {"role": "user",   "content": "Code: Patient presents with moderate persistent asthma without complications."},
      {"role": "assistant", "content": "CA23.1 — Asthma, moderate persistent, without complications\nRationale: CA23 = Asthma; .1 = moderate persistent severity; no mention of complications or status asthmaticus."}
    ]
  },
  {
    "messages": [
      {"role": "system", "content": "You are a medical coding specialist following ICD-11 standards."},
      {"role": "user",   "content": "Code: Type 2 diabetes with stage 3 chronic kidney disease."},
      {"role": "assistant", "content": "5A11 + MF56.2 — Type 2 diabetes mellitus (5A11) with Chronic kidney disease, stage 3 (MF56.2)\nRationale: ICD-11 uses extension codes for comorbidities. Diabetes and CKD are coded separately and linked via extension."}
    ]
  }
]

Advanced Alignment Topics

Beyond the core SFT-RLHF-DPO pipeline, several advanced alignment research directions are shaping the future of safe AI development. Practitioners building production systems benefit from understanding these techniques even if they are not yet in their immediate stack.

Scalable Oversight

Scalable oversight addresses the fundamental challenge of AI alignment: how do you align a model that is more capable than the humans supervising it? Standard RLHF assumes humans can correctly identify the better of two model responses — but for highly technical domains (advanced mathematics, cutting-edge research, long-horizon planning), this assumption breaks down. The model may produce a plausible-sounding but incorrect response that human evaluators cannot detect.

Proposed solutions include debate (Irving et al., 2018): two AI agents debate the correct answer while a human judge arbitrates, with the insight that it is easier to identify a flaw in an argument than to verify a complex claim from scratch. Recursive reward modelling trains assistants to help humans give more informed feedback, improving the quality of the reward signal. Iterated amplification bootstraps oversight by decomposing complex tasks into simpler sub-tasks the human can evaluate directly.

Process Reward Models

Process Reward Models (PRMs) score the quality of each reasoning step in a chain-of-thought, rather than only the final answer. Standard reward models are outcome-based — they score final responses — which can incentivise the model to arrive at correct answers through flawed reasoning. PRMs provide denser training signal and better generalisation for multi-step reasoning tasks. OpenAI's Let's Verify Step by Step paper demonstrated that process supervision significantly outperforms outcome supervision on mathematical problem solving, and PRMs are now standard in frontier math reasoning models (o1, o3, DeepSeek-R1).

Constitutional AI in Depth

Anthropic's Constitutional AI (CAI) uses a two-phase approach. In the supervised learning from AI feedback (SL-CAF) phase, the model generates responses to harmful prompts, then critiques and revises them according to a set of principles (the "constitution"), and is trained on the revised responses. In the RL from AI feedback (RLAIF) phase, the AI uses the constitution to provide preference labels for pairs of responses, replacing expensive human annotation. The constitution can articulate nuanced ethical principles that are difficult to convey through preference data alone.

Evaluation and Red-Teaming

Alignment evaluation requires systematic red-teaming — actively attempting to elicit harmful, deceptive, or unsafe behaviour from aligned models. Red-teaming approaches include: manual red-teaming by domain experts (the most effective but expensive); automated red-teaming using adversarial LLMs to generate jailbreaking prompts at scale; constitutional red-teaming where the model critiques its own responses; and benchmark suites like TruthfulQA (accuracy on misleading questions), HarmBench (harmfulness elicitation), and SafetyBench (multilingual safety).

Multi-Task LoRA Serving Architecture

LoRA adapters can be combined using several techniques. Model merging (SLERP, TIES-Merging, DARE) linearly interpolates or combines the weights of multiple fine-tuned models, producing a merged model with averaged capabilities — useful for combining domain knowledge (e.g., a medical adapter + a coding adapter) without multi-task training. LoRA composition stacks multiple LoRA adapters during inference, applying each in sequence. Task vectors represent fine-tuning as a direction in weight space that can be added, subtracted, or scaled to compose capabilities.

The multi-task LoRA serving pattern is particularly powerful for organisations with many specialised use cases: a single large base model (e.g., Llama 3 70B) is kept in GPU VRAM, with a library of task-specific LoRA adapters that are hot-swapped per request. Each adapter adds only a few tens of megabytes of overhead. A request routing layer selects the appropriate adapter based on task type, user segment, or explicit selection. This architecture reduces total GPU footprint dramatically compared to serving multiple full fine-tuned models.

Evaluating Fine-tuned Models in Production

Production fine-tuning evaluation requires more than offline benchmarks. A robust evaluation suite for a domain-adapted LLM typically includes: held-out task accuracy on a representative sample of the target task; regression testing on general instruction following benchmarks (MT-Bench, MMLU) to detect capability degradation; latency and throughput profiling at production batch sizes; human preference evaluation comparing the fine-tuned model to the base model and to existing solutions; and adversarial input testing for the specific safety concerns of the domain.

A common production failure mode: fine-tuning improves task-specific accuracy while degrading general instruction following. This manifests as a model that performs well on the narrow target task but fails on adjacent requests from production users. The solution is diverse evaluation — include both task-specific and general evaluations, and treat regression on general benchmarks as a blocker for deployment.

Exercises

These exercises are designed to move you from conceptual understanding to hands-on implementation. Work through them in order — each builds on the previous.

Fine-tuned Model Card Generator

Model cards are the standard documentation artifact for any fine-tuned model — capturing intended use, training data, performance characteristics, and known limitations. Generate a complete model card for your fine-tuned model using the form below.

Conclusion & Next Steps

The four-layer adaptation stack covers the full journey from a raw pre-trained model to a production-ready AI assistant: full fine-tuning for large distribution shifts with abundant data, LoRA/QLoRA for efficient task adaptation with modest data budgets, instruction tuning (SFT) for teaching behavioural format and instruction following, and RLHF/DPO for aligning model values with human preferences and safety requirements.

A key insight running through this article: alignment is not a one-time checkbox. Production models require continuous evaluation, periodic red-teaming, and scheduled re-alignment as the world changes and the model's failure modes become better understood. The most sophisticated alignment pipeline in the world can be undone by data drift, adversarial users, or unanticipated edge cases in deployment.

The field is evolving rapidly. Constitutional AI and RLAIF are reducing the cost of alignment by replacing expensive human preference annotation with LLM-generated feedback. The DPO family (IPO, SimPO, KTO, ORPO) is simplifying the alignment pipeline further. Scalable oversight techniques (debate, process reward models, automated red-teaming) are being developed to align models that are more capable than the humans supervising them — the core challenge for the next decade of AI development.