Understanding Lemma 1 in DPO

Lemma 1

Under the Plackett-Luce, and in particular the Bradley-Terry, preference framework, two reward functions from the same class induce the same preference distribution. — Rafailov et al. (2024), p. 5

The Plackett-Luce preference framework, including the Bradley-Terry model, is used to model the probability of preferring a specific item (e.g., a response $y$) among several options. This framework typically derives preference patterns based on the differences in reward function values. For instance, in the Bradley-Terry model, the probability $p(y_1>y_2\mid x)$ that response $y_1$ is preferred over $y_2$ for a given prompt $x$ is calculated as follows:

$$p(y_1>y_2|x)=\sigma (r(x,y_1)-r(x,y_2))$$

This equation shows that the greater the difference in reward values between two responses, $y_1$ and $y_2$, the higher the probability that $y_1$ will be preferred.

According to the Definition, if two reward functions $r(x,y)$ and $r^{\prime }(x,y)$ are equivalent, then $r^{\prime }(x,y)$ can be expressed as:

$$r^{\prime }(x,y)=r(x,y)+f(x)$$

When using two equivalent reward functions, $r$ and $r^{\prime }$, the difference in reward values between two responses, $y_1$ and $y_2$, is calculated as:

$$\begin{array}{rl}{r}^{\prime }(x,y_1)-r^{\prime }(x,y_2)& =(r(x,y_1)+f(x))-(r(x,y_2)+f(x))\\ & =r(x,y_1)-r(x,y_2)+f(x)-f(x)\\ & =r(x,y_1)-r(x,y_2)\end{array}$$

As the result shows, the reward difference between the two responses is the same whether calculated with $r$ or $r^{\prime }$.

Since the Plackett-Luce framework is based on this ‘reward difference’ to determine preference patterns, reward functions belonging to the same equivalence class will ultimately induce identical preference probability distributions.

See also

Rafailov et al. (2024), p. 17

Reference

Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2024). Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. arXiv. https://doi.org/10.48550/arXiv.2305.18290