Preference Optimization Without the Pain: DPO vs. RLHF in Production
Aligning a language model to produce helpful, harmless, and honest responses was, until 2022, a multi-stage process requiring a reward model, a reinforcement learning training loop, and careful management of KL divergence penalties to prevent the policy from collapsing. InstructGPT made RLHF work at scale, but the engineering complexity was substantial. DPO, published in 2023, replaced the reinforcement learning component with a supervised classification objective and achieved comparable alignment quality with dramatically lower infrastructure requirements. Whether to use RLHF, DPO, or one of several newer alternatives is now a genuine engineering decision rather than a foregone conclusion in favor of the simpler method.
RLHF mechanics in detail
The standard RLHF pipeline has three stages, each producing an artifact that feeds the next, and each introducing its own failure modes.
Stage 1: supervised fine-tuning (SFT). The base model is fine-tuned on a curated set of high-quality (prompt, response) pairs to produce a model that can follow instructions at all. Without SFT, the RL training starts from a distribution too diffuse to make progress. A base language model predicts the next token over the entire distribution of text that appeared during pre-training, including code, news articles, fiction, and web scrape. An RL reward signal for helpfulness cannot steer a model that does not yet know what helpful responses look like. SFT narrows the starting distribution to something the reward signal can work with.
Stage 2: reward model training. Human annotators compare pairs of model outputs for the same prompt and indicate which is better. These comparison pairs, structured as (prompt, chosen response, rejected response), are used to train a reward model that learns to score outputs. The reward model is typically initialized from the SFT model with the language modeling head replaced by a scalar output. Training uses the Bradley-Terry model for pairwise preferences:
P(a > b | prompt) = sigmoid(r(a) - r(b))
where r is the reward function the model is learning to approximate. The Bradley-Terry model is a standard approach for learning from pairwise comparisons. Its key assumption is that the probability that one response is preferred over another depends only on their respective reward scores and not on any other contextual factors. This assumption is clean enough to make the training tractable, but it breaks down for subjective tasks where the same annotator might prefer different responses on different days, or where different annotators have systematically different preferences.
Stage 3: PPO fine-tuning. The SFT model is used as the policy. The reward model scores the policy's outputs. Proximal Policy Optimization updates the policy weights to maximize expected reward, subject to a KL divergence penalty that prevents the policy from drifting too far from the SFT initialization. The complete objective is:
maximize E[r(x,y)] - β · KL[π(y|x) || π_ref(y|x)]
The KL penalty is the crucial component. Without it, the policy would maximize the reward model's score by producing adversarial outputs, responses that happen to score high on the reward model's learned function but are not actually good by the criteria human annotators were trying to express. The reward model is a compressed, approximate representation of human preferences. Any model trained to maximize an approximate objective while unconstrained will find ways to exploit the approximation. The KL penalty anchors the policy to the SFT distribution, limiting how far it can drift in the direction of reward model exploitation. The coefficient β controls this tradeoff: small β allows more aggressive optimization of the reward, at the cost of more reward hacking risk; large β keeps the policy close to the SFT baseline.
PPO itself introduces hyperparameter complexity. The clipping epsilon (typically 0.1 to 0.2) controls how large a policy update is allowed in a single step. The Generalized Advantage Estimation lambda (typically 0.95) controls the bias-variance tradeoff in advantage estimation. The value function learning rate needs to be calibrated relative to the policy learning rate. Getting these right for a new task is not straightforward, and the training signal is noisier than standard supervised learning because the rewards arrive per-sequence rather than per-token.
The memory requirement is daunting. Training requires simultaneously in memory: a policy model (the one being updated), a frozen reference policy (for computing KL divergence), a reward model, and a value function for PPO's advantage estimation. For a 7B model in fp16, each model copy is approximately 14 GB. Four copies represent 56 GB just for model weights, before gradients and optimizer states. Adam optimizer states for the policy alone add another 28 GB (first and second moment estimates, each the same size as the parameters). Total training memory for RLHF on a 7B model runs to approximately 160 GB on a well-tuned setup, which requires a multi-GPU node even before accounting for activation memory.
DPO: the key insight
DPO (Rafailov et al., 2023) showed that the RLHF objective can be rearranged to eliminate the explicit reward model and RL training loop entirely. The derivation is concise enough to follow in full.
The RLHF objective under a KL-constrained policy has a known closed-form solution for the optimal policy:
π*(y|x) ∝ π_ref(y|x) · exp(r(x,y) / β)
where π_ref is the reference (SFT) policy, r is the reward, and β is the KL penalty coefficient. This expression says: the optimal policy is proportional to the reference policy, weighted by how much the reward exceeds the typical reward. Responses the reward model scores highly relative to the reference policy get amplified; responses it scores poorly get suppressed.
Rearranging this to express the reward in terms of the policy:
r(x,y) = β · log(π*(y|x) / π_ref(y|x)) + β · log Z(x)
where Z(x) is a partition function that normalizes the distribution. The partition function is the same for any two responses to the same prompt, so when you plug this expression into the Bradley-Terry pairwise preference model, Z(x) cancels:
P(y_w > y_l | x) = sigmoid(r(x, y_w) - r(x, y_l))
= sigmoid(β · log(π*(y_w|x) / π_ref(y_w|x)) - β · log(π*(y_l|x) / π_ref(y_l|x)))
Since the optimal policy π* is what we are trying to learn, we substitute the current policy π for π* and turn this into a training objective:
L_DPO = -E[log σ(β · log(π(y_w|x)/π_ref(y_w|x)) - β · log(π(y_l|x)/π_ref(y_l|x)))]
where y_w is the preferred (winning) response and y_l is the rejected (losing) response.
This is a standard cross-entropy loss. Computing it requires one forward pass through the policy (to compute log π(y_w|x) and log π(y_l|x)) and one forward pass through the frozen reference (to compute log π_ref(y_w|x) and log π_ref(y_l|x)). No reward model. No value function. No PPO training loop. Two model copies instead of four.
The gradient of this loss has an interpretable form. Increasing the probability of the preferred response and decreasing the probability of the rejected response are both weighted by how much the current policy is miscalibrated relative to its implicit reward signal. If the policy already assigns much higher probability to the preferred response, the gradient update is small. If the policy currently prefers the rejected response (a failure case), the gradient update is large and corrective. This self-weighting behavior makes DPO more stable than PPO in practice, because updates are automatically scaled by how much the policy is currently wrong rather than requiring the learning rate to be hand-tuned relative to an external critic.
For a 7B model, DPO training memory drops to roughly 28 GB: policy weights (14 GB) plus reference weights (14 GB) plus optimizer states only for the policy (28 GB for Adam). The reference model is frozen, so it requires no optimizer state. This fits on a single A100-80GB. Running RLHF at the same model size requires a minimum of a 4-GPU node, and in practice 8 GPUs for comfortable headroom. The infrastructure difference is not just cost; it is the difference between fine-tuning being accessible to a small team with a single rented GPU and requiring a multi-node cluster.
Preference data requirements
Both RLHF and DPO require preference pairs in the form (prompt, chosen response, rejected response). The volume needed scales with task complexity, and the relationship between volume and task type is worth reasoning about carefully before starting annotation.
For style and tone adjustment, 5,000 to 10,000 pairs is often sufficient. Style preferences are consistent and generalizable from few examples. If you want the model to write more concisely or maintain a formal register, a few thousand examples that distinguish good from bad writing along these specific dimensions typically produce good results. The preference signal is coherent across examples: the annotator is applying a consistent criterion that generalizes well from a small training set.
For factual preference, specifically teaching the model to prefer accurate responses over plausible-sounding incorrect ones, the volume requirement climbs to 30,000 to 100,000 pairs. The space of factual claims is large and factual correctness requires domain-specific generalization that is harder to learn from few examples. The problem is that factual accuracy does not have a single uniform criterion the way writing style does. A response about immunology is evaluated on different dimensions than a response about tax law, and both are evaluated differently from a response about numerical reasoning. You need enough coverage across the factual domain that the preference model generalizes to novel claims, rather than memorizing which responses are preferred in training.
For multi-turn dialogue preferences, the challenge is not just volume but structure. The preference between two multi-turn conversations may depend on subtle coherence across turns, persona consistency, and long-range reference resolution. A response at turn 4 that contradicts something established at turn 1 is bad, but detecting this requires attending to the full conversational context, not just the local response quality. This is where DPO shows structural weakness: it was derived for single-turn preferences, and extending it to multi-turn correctly requires careful handling of which tokens are included in the log-ratio computation.
The annotation process itself introduces systematic biases that affect both RLHF and DPO. Annotators show a well-documented preference for longer responses, more confident-sounding responses, and responses that mention the annotator's own name or prior context. These preferences are partly genuine (longer responses often do contain more useful information) and partly artifacts of how preferences are collected (annotators reading two options side-by-side may favor the one that feels more thorough regardless of accuracy). Both RLHF and DPO will faithfully learn these biases if they are present in the training data. The garbage-in-garbage-out principle applies to preference learning just as it applies to supervised fine-tuning.
Where DPO falls short
DPO works well when the task can be framed as a binary preference between two responses to the same prompt. Several practical scenarios fall outside this framing in ways that cannot be fixed by collecting more data or tuning hyperparameters.
Multi-turn dialogue is the most common failure case in production. The preference between two conversation trajectories is not cleanly expressible as a single-step preference: the model's responses at turn 1 shape what happens at turn 2, and the quality of a conversation is determined by the interaction across turns, not by any individual response. A model trained with single-turn DPO on multi-turn conversations is optimizing a local objective when the true quality signal is global. Extensions like RPO (Robust Preference Optimization) and step-level DPO address this by computing the preference loss at each conversational step with appropriate conditioning, but they add complexity that erodes DPO's simplicity advantage and require more careful implementation.
Sparse rewards present a different challenge. RLHF with PPO can handle rewards that arrive infrequently or are computed per-episode rather than per-token, because PPO is designed to handle delayed reward signals through temporal credit assignment. DPO requires preference pairs, which implicitly assumes response-level evaluation. For tasks like code generation where the reward is a pass/fail from running the code against test cases, generating the preference pairs requires actually executing both the winning and losing candidate responses through an evaluator. This is feasible but adds a data preparation pipeline that can be expensive when the evaluation involves compilation, sandboxed execution, or calls to external services.
DPO can also produce over-optimization on the preference data in ways that are subtler than RLHF reward hacking. The objective directly minimizes the log-likelihood of rejected responses, which means the model is being trained to avoid certain patterns, not just to produce preferred ones. This can cause the model to become overly conservative, avoiding response structures that superficially resemble rejected training examples even when those structures are appropriate. The effect is most pronounced when the rejected responses in the training data are concentrated in a narrow part of the output space, because the model learns to broadly suppress that region rather than learning the more precise criterion that made specific instances within that region undesirable.
Finally, DPO inherits the Bradley-Terry assumption that preference can be summarized as a real-valued scalar reward. For tasks where the preference space is genuinely multi-dimensional (a response can be more helpful but less safe, or more accurate but less concise) and where the appropriate tradeoff between dimensions varies by context, the scalar-reward formulation is a compression that loses information. RLHF at least makes the reward model an explicit object that can be examined and corrected; DPO folds the reward implicitly into the policy, which makes it harder to audit what preference criterion the model has actually internalized.
The practical decision tree
The choice between RLHF, DPO, and the newer alternatives depends on infrastructure, data availability, and task structure. Getting this decision right at the start saves substantial compute and iteration time.
DPO is the right default for most alignment tasks in production. It requires fewer compute resources, produces comparable quality for single-turn preference tasks, and is easier to debug because the training objective is a standard classification loss with interpretable gradients. It trains stably without PPO hyperparameter tuning. For style and tone adjustment, safety filtering, format enforcement, and single-turn question-answering quality, start with DPO. The bar for moving to something more complex should be a specific, demonstrable failure mode, not an abstract concern about optimality.
RLHF is justified under three conditions: the reward signal is not pairwise-comparable (for example, the reward is a continuous score or arrives per-episode rather than as a response-level comparison), the task is complex enough that binary preference annotation cannot capture the full quality signal, or you have existing reward model infrastructure and preference data in a format that does not cleanly convert to (chosen, rejected) pairs. The canonical case for RLHF over DPO is multi-turn dialogue optimization with episode-level reward, where the reward signal evaluates entire conversations and where PPO's temporal credit assignment is actually necessary.
RLAIF, reinforcement learning from AI feedback (Bai et al., 2022), replaces human annotators with a strong AI model (typically Claude or GPT-4) to generate preference labels. The quality of RLAIF-generated preferences is surprisingly high for tasks where the AI judge can reliably distinguish better from worse responses, particularly for factual accuracy, formatting, and safety. The cost is a fraction of human annotation: generating 100,000 preference pairs with GPT-4 as judge costs on the order of hundreds of dollars, versus tens of thousands of dollars for equivalent human annotation. For organizations without annotation infrastructure, RLAIF is often the practical path to preference optimization. The limitation is that the AI judge's own biases and blind spots propagate into the preference data. An AI judge trained to be cautious may systematically prefer responses that are more hedged or longer, and fine-tuning on those preferences produces a model that inherits those biases.
Constitutional AI (Anthropic, 2022) generates both preference data and conducts alignment with a self-critique mechanism. The model critiques its own responses against a set of principles (the "constitution"), revises them, and uses the original versus revised pairs as preference data. This eliminates the need for any external preference annotation. The quality is good for harmlessness, where the constitutional principles can be stated clearly and the model can reason about violations with reasonable reliability. It is weaker for helpfulness nuance, where the relevant principles are harder to articulate as a constitution and where self-critique is not a reliable substitute for human judgment about what responses are genuinely useful.
For most production teams building on top of open-weight base models, the practical sequence is: collect or generate 10,000 to 50,000 preference pairs (using RLAIF if human annotation is not available), run DPO fine-tuning on a single A100 with the SFT checkpoint as reference, evaluate, and iterate on the preference data before considering a more complex method. The limiting factor is almost always preference data quality and quantity, not algorithm choice. Moving from DPO to RLHF with mediocre preference data will not improve results. Getting the preference data right and then running DPO is almost always the right first step.
References
Ouyang, L., Wu, J., Jiang, X., et al. "Training Language Models to Follow Instructions with Human Feedback." NeurIPS, 2022. https://arxiv.org/abs/2203.02155
Rafailov, R., Sharma, A., Mitchell, E., et al. "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." NeurIPS, 2023. https://arxiv.org/abs/2305.18290
Bai, Y., Jones, A., Ndousse, K., et al. "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback." Anthropic, 2022. https://arxiv.org/abs/2204.05862
Bai, Y., Kadavath, S., Kundu, S., et al. "Constitutional AI: Harmlessness from AI Feedback." Anthropic, 2022. https://arxiv.org/abs/2212.08073
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. "Proximal Policy Optimization Algorithms." arXiv, 2017. https://arxiv.org/abs/1707.06347
Azar, M. G., Guo, Z. D., Piot, B., et al. "A General Theoretical Paradigm to Understand Learning from Human Feedback." AISTATS, 2024. https://arxiv.org/abs/2310.12036