Understanding Reward Modeling in RLHF

1. Generate Responses With a Fine-tuned Model

A fine-tuned model, ${\pi }^{\text{SFT}}$, is given a prompt $x$ to generate two distinct responses $y_1$ and $y_2$.

$$(y_1,y_2)\sim {\pi }^{\text{SFT}}(y|x)$$

2. Human Labeling (Preference Collection)

A human labeler evaluates the two responses $(y_1,y_2)$ and indicates which one is better, resulting in a preferred response $y_w$​ (winner) and a dispreferred response $y_l$​ (loser). This pattern of preferences will be used by the reward model to assign future preference scores.

3. Reward Model

There are various methods for modeling preferences, with the Bradley-Terry (BT) model being the most popular.

• Bradley-Terry Model:

$$p^{\ast }(y_1>y_2|x)=\frac{\exp (r^{\ast }(x,y_1))}{\exp (r^{\ast }(x,y_1))+\exp (r^{\ast }(x,y_2))}$$

This equation calculates the probability that $y_1$​ is preferred over $y_2$​ given prompt $x$, based on their respective reward scores $r^{\ast }$.

4. Maximum Likelihood Estimation (MLE)

1. MLE is used to train the reward model $r_{\varphi }$​
2. This can be framed as a binary classification problem (classifying which response is preferred).
3. The training involves minimizing the negative log-likelihood function: $${\mathcal{L}}_R(r_{\varphi },\mathcal{D})=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}[\log \sigma (r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l))]$$ This loss function aims to maximize the probability of assigning a higher reward to the preferred response ($y_w$) compared to the dispreferred response ($y_l$) based on the collected dataset $D$.

(Rafailov et al., 2024, p. 3)

Reference

Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2024). Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. arXiv. https://doi.org/10.48550/arXiv.2305.18290