Understanding The Equivalence Between Two Reward Models in DPO

Definition 1

We say that two reward functions $r(x,y)$ and $r^{\prime }(x,y)$ are equivalent iff $r(x,y)-r^{\prime }(x,y)=f(x)$ for some function $f$. — Rafailov et al. (2024), p. 5

According to the definition, two reward function $r_1$ and $r_2$, are considered equivalent if their difference depends only on the prompt $x$ and not on the response $y$. This difference is expressed as a function $f(x)$.

For example, if a given prompt $x$, there are five possible responses $y_1,y_2,\dots ,y_5$, the difference between two equivalent reward functions $r_1$ and $r_2$, will be the same for all possible responses, as shown below:

$$\begin{array}{rl}{r}_1(x,y_1)-r_2(x,y_1)& =f(x)\\ r_1(x,y_2)-r_2(x,y_2)& =f(x)\\ ⋮& \\ r_1(x,y_5)-r_2(x,y_5)& =f(x)\end{array}$$

This means that for the same prompt $x$, the difference between the values of the two reward functions will always remain a constant value, $f(x)$, regardless of which response is generated.

See also

Understanding Lemma 1 in DPO
Understanding Lemma 2 in DPO

Reference

Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2024). Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. arXiv. https://doi.org/10.48550/arXiv.2305.18290