Avoiding Explicit Reward Models in DPO

1. Bradley-Terry Preference Model:

$$p^{\ast }(y_1>y_2|x)=\sigma (r^{\ast }(x,y_1)-r^{\ast }(x,y_2)),\text{\hspace{1em}(a)}$$

2. Optimal Policy in RLHF:

$${\pi }_r(y|x)=\frac{1}{Z(x)}{\pi }_{\text{ref}}(y|x)\exp \left(\frac{1}{\beta }r(x,y)\right),\text{\hspace{1em}(b)}$$

DPO Method

By taking the logarithm of both sides of equation (b), we can express the reward model $r$ in terms of the optimal policy ${\pi }_r$​, the reference policy ${\pi }_{\text{ref}}$​, and the partition function $Z(x)$:

$$r(x,y)=\beta \log \frac{{\pi }_r(y|x)}{{\pi }_{\text{ref}}(y|x)}+\beta \log Z(x),\text{\hspace{1em}(c)}$$

This derived $r$ from (c) can then be applied to the ground-truth reward function $r^{\ast }$ or optimal policy ${\pi }^{\ast }$. Fortunately, the Bradley-Terry (BT) model calculates preference using only $r^{\ast }$.

When we apply the result from (c) to equation (a), the partition function $Z(x)$ conveniently cancels out:

$$p^{\ast }(y_1>y_2|x)=\sigma \left(\beta \log \frac{{\pi }^{\ast }(y_1|x)}{{\pi }_{\text{ref}}(y_1|x)}-\beta \log \frac{{\pi }^{\ast }(y_2|x)}{{\pi }_{\text{ref}}(y_2|x)}\right)$$

Ultimately, this means that DPO allows us to directly derive the optimal policy that reflects preferences without the need for a separate reward function.

Bonus

This method is not limited to the Bradley-Terry model; it can be applied more generally to any Plackett-Luce model.

(Rafailov et al., 2024, pp. 4–5)

Reference

Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2024). Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. arXiv. https://doi.org/10.48550/arXiv.2305.18290