1
00:00:03,000 --> 00:00:06,020
Evan: Welcome to Daily Paper Cast.

2
00:00:06,072 --> 00:00:15,032
Ashley: Today we’re covering a paper from Hugging Face's daily paper list of September 17, 2026, which has garnered 21 upvotes.

3
00:00:15,096 --> 00:00:21,276
Evan: The paper we’re discussing is titled 'A Zeroth-Order Paradigm for LLM Preference Alignment.'

4
00:00:21,366 --> 00:00:24,756
Ashley: Peter Chen and Xi Chen are the first two authors.

5
00:00:24,756 --> 00:00:33,456
The corresponding author, Peter Chen, is from the Department of Electrical Engineering and Computer Sciences at the University of California, Berkeley.

6
00:00:33,564 --> 00:00:36,214
Evan: All right, let’s dive into the Introduction.

7
00:00:36,214 --> 00:00:43,144
Background first: Generative AI has become a critical tool across academia, industry, and government.

8
00:00:43,200 --> 00:00:44,210
Ashley: Exactly.

9
00:00:44,210 --> 00:00:54,320
Large language models, or LLMs, are central to this progress, excelling in areas like data organization, retrieval, reasoning, and analysis.

10
00:00:54,384 --> 00:01:04,884
Evan: However, LLMs trained on diverse corpora often need further alignment with human preferences to ensure their responses are helpful, harmless, and reliable.

11
00:01:04,944 --> 00:01:05,854
Ashley: Correct.

12
00:01:05,854 --> 00:01:19,804
A common approach to achieve this alignment is reinforcement learning from human feedback, or RLHF, which involves learning a reward model from human preference pairs, followed by policy optimization using reinforcement learning.

13
00:01:19,848 --> 00:01:29,708
Evan: But RLHF is computationally expensive, requiring a multi-stage training pipeline involving significant memory and computation resources.

14
00:01:29,760 --> 00:01:33,800
Ashley: Yes, which is why direct alignment methods have become popular.

15
00:01:33,800 --> 00:01:44,460
These methods, like direct preference optimization or DPO, directly optimize policy using preference pairs, bypassing the need to train a separate reward model.

16
00:01:44,520 --> 00:01:47,780
Evan: However, even these methods face issues.

17
00:01:47,780 --> 00:01:56,480
One major challenge is 'likelihood displacement,' where the training inadvertently shifts the probability mass away from preferred responses.

18
00:01:56,544 --> 00:01:57,544
Ashley: That’s right.

19
00:01:57,544 --> 00:02:09,004
Likelihood displacement can cause the absolute probability of the preferred responses to decrease, leading to unintended unalignment, where the model could produce unsafe or unhelpful responses.

20
00:02:09,048 --> 00:02:18,308
Evan: The researchers noted that this displacement often arises from preference pairs with small likelihood margins, deemed 'noisy preference pairs.'

21
00:02:18,360 --> 00:02:19,470
Ashley: Indeed.

22
00:02:19,470 --> 00:02:28,940
Unlike approaches that filter out these noisy pairs, Peter Chen and team proposed a novel method called Comparison-based Preference Optimization, or ComPO.

23
00:02:28,940 --> 00:02:34,500
This method leverages the comparative information from these noisy pairs instead of discarding them.

24
00:02:34,560 --> 00:02:39,680
Evan: Ashley, how does ComPO differ from traditional optimization methods?

25
00:02:39,744 --> 00:02:41,064
Ashley: Great question.

26
00:02:41,064 --> 00:02:47,244
ComPO is a zeroth-order alignment method that doesn't directly optimize a differentiable preference loss.

27
00:02:47,244 --> 00:02:53,444
Instead, it uses comparison oracles to extract directional information from preference pairs.

28
00:02:53,496 --> 00:03:07,216
Evan: So, rather than optimizing based on a fixed loss function, ComPO perturbs the current policy and evaluates whether each perturbation improves the likelihood of preferred responses over dispreferred ones.

29
00:03:07,272 --> 00:03:08,092
Ashley: Correct.

30
00:03:08,092 --> 00:03:17,872
This allows ComPO to utilize the information in low-margin preference pairs without risking the adverse effects associated with directly optimizing noisy pairs.

31
00:03:17,928 --> 00:03:27,448
Evan: The researchers provide a convergence guarantee for ComPO under certain conditions, which is always reassuring for those concerned with theoretical foundations.

32
00:03:27,504 --> 00:03:38,544
Ashley: And they didn’t stop there; they also introduced an online version of ComPO, which integrates unlabeled policy generations for controlling deviations from a reference policy.

33
00:03:38,592 --> 00:03:39,832
Evan: Interesting.

34
00:03:39,832 --> 00:03:47,992
It appears they framed this through a coverage perspective, ensuring local coverage and in-distribution pairwise reward accuracy.

35
00:03:48,048 --> 00:03:49,228
Ashley: Exactly.

36
00:03:49,228 --> 00:04:01,088
They experimented with several models like Mistral, Llama, and Gemma-3 and found that ComPO outperforms existing methods, providing valuable insights into mitigating likelihood displacement.

37
00:04:01,212 --> 00:04:04,672
Evan: And with that, we’ve covered the Introduction section.

38
00:04:05,990 --> 00:04:11,090
Evan: Ashley, let's delve into the methodologies proposed by the authors for ComPO.

39
00:04:11,138 --> 00:04:12,238
Ashley: Sure, Evan.

40
00:04:12,238 --> 00:04:24,378
The proposed method, Comparison-based Preference Optimization or ComPO, is designed to address the limitations of existing direct alignment methods, particularly the issue of likelihood displacement.

41
00:04:24,434 --> 00:04:28,754
Evan: Could you break down the key components of the ComPO method for us?

42
00:04:28,832 --> 00:04:29,912
Ashley: Of course.

43
00:04:29,912 --> 00:04:43,662
The core idea behind ComPO is to use comparison oracles to extract directional information from noisy preference pairs, which are pairs where the preferred and dispreferred responses have similar likelihoods from the model's perspective.

44
00:04:43,706 --> 00:04:49,726
Evan: So, rather than ignoring these noisy pairs, ComPO utilizes them constructively?

45
00:04:49,778 --> 00:04:50,898
Ashley: Exactly.

46
00:04:50,898 --> 00:04:59,478
To put this into practice, ComPO employs a process that involves policy perturbation and evaluation based on preference comparison oracles.

47
00:04:59,522 --> 00:05:05,642
Evan: Can you elaborate on these comparison oracles and how they function within ComPO?

48
00:05:05,690 --> 00:05:06,720
Ashley: Certainly.

49
00:05:06,720 --> 00:05:16,510
A comparison oracle is a mechanism that takes two policies and evaluates which of them is better based on the likelihoods of the preferred versus dispreferred responses.

50
00:05:16,510 --> 00:05:23,070
Specifically, it assigns a 'better' or 'worse' label to each perturbed policy based on this evaluation.

51
00:05:23,114 --> 00:05:28,374
Evan: Ah, so ComPO relies on these labels to guide its optimization process?

52
00:05:28,418 --> 00:05:29,238
Ashley: Correct.

53
00:05:29,238 --> 00:05:35,158
Here's a step-by-step breakdown of how it works: First, ComPO starts with an initial policy.

54
00:05:35,158 --> 00:05:40,778
It then generates multiple perturbed versions of this policy by applying random perturbations.

55
00:05:40,826 --> 00:05:44,326
Evan: And how are these perturbations evaluated?

56
00:05:44,378 --> 00:05:53,988
Ashley: For each perturbed policy, a comparison oracle evaluates whether the likelihood of the preferred response has increased and the dispreferred response has decreased.

57
00:05:53,988 --> 00:05:58,558
Based on this, the oracle assigns either a positive or negative signal.

58
00:05:58,610 --> 00:05:59,530
Evan: I see.

59
00:05:59,530 --> 00:06:03,950
And then these signals are aggregated to update the policy, right?

60
00:06:04,010 --> 00:06:05,100
Ashley: Precisely.

61
00:06:05,100 --> 00:06:09,410
The aggregated signals help estimate a normalized update direction.

62
00:06:09,410 --> 00:06:17,930
This direction informs how the policy should be adjusted to align better with human preferences without directly optimizing a noisy loss function.

63
00:06:17,978 --> 00:06:19,218
Evan: Interesting.

64
00:06:19,218 --> 00:06:23,458
But what ensures that this method converges to an optimal policy?

65
00:06:23,552 --> 00:06:24,582
Ashley: Good point.

66
00:06:24,582 --> 00:06:38,802
The authors prove a convergence guarantee for ComPO's offline scheme, stating that under conditions of smoothness and gradient sparsity, the method converges to a policy where the gradient of the underlying alignment objective is small.

67
00:06:38,918 --> 00:06:40,658
Evan: That sounds promising.

68
00:06:40,658 --> 00:06:43,818
Did they also introduce an online version of ComPO?

69
00:06:43,874 --> 00:06:45,224
Ashley: Yes, they did.

70
00:06:45,224 --> 00:06:52,684
The online version of ComPO retains the comparison mechanism but includes additional components like reverse-KL control.

71
00:06:52,684 --> 00:06:57,814
This constrains the policy divergence from a reference policy to a prescribed neighborhood.

72
00:06:57,866 --> 00:07:01,966
Evan: How does this reverse-KL control work in practice?

73
00:07:02,018 --> 00:07:07,978
Ashley: In practice, the reverse-KL control is estimated using generations from the current policy.

74
00:07:07,978 --> 00:07:16,298
It translates the deviation into a step-size adjustment, ensuring that the updates maintain performance without deviating too much from the reference.

75
00:07:16,406 --> 00:07:20,036
Evan: And what about the practical implementation of ComPO?

76
00:07:20,036 --> 00:07:23,206
Did they manage to simplify it for real-world use?

77
00:07:23,258 --> 00:07:24,178
Ashley: Indeed.

78
00:07:24,178 --> 00:07:31,988
The practical implementation applies perturbations only to the output layer of the model, significantly reducing computational costs.

79
00:07:31,988 --> 00:07:39,318
Additionally, it uses a sparse gradient estimation technique where only significant gradient entries are updated.

80
00:07:39,362 --> 00:07:41,162
Evan: This sounds very efficient.

81
00:07:41,162 --> 00:07:45,782
Were there any experimental setups or benchmarks discussed in the methodology?

82
00:07:45,842 --> 00:07:55,572
Ashley: Yes, the authors evaluated ComPO on various models, including Mistral-7B, Llama-3, Gemma-2, Qwen3, and Gemma-3.

83
00:07:55,572 --> 00:08:06,782
They optimized ComPO over clean and noisy preference pairs across these models and used the evaluation metrics like length-controlled win rates and raw win rates to benchmark its performance.

84
00:08:06,842 --> 00:08:11,702
Evan: Were there any unique results or findings in these evaluations?

85
00:08:11,762 --> 00:08:13,622
Ashley: Several findings stood out.

86
00:08:13,622 --> 00:08:23,192
For instance, ComPO improved length-controlled win rates, meaning it was able to maintain or improve performance while controlling the verbosity of responses.

87
00:08:23,192 --> 00:08:28,582
This indicates better efficacy in managing the trade-off between response length and quality.

88
00:08:28,634 --> 00:08:33,784
Evan: That's quite a feat, considering verbosity often complicates model responses.

89
00:08:33,784 --> 00:08:38,554
Did they have any specific metrics or diagnostic tools to support these findings?

90
00:08:38,618 --> 00:08:39,478
Ashley: Certainly.

91
00:08:39,478 --> 00:08:44,998
The authors used pair-level likelihood diagnostics as an additional measure to support their findings.

92
00:08:44,998 --> 00:08:53,658
These diagnostics provided evidence that ComPO effectively mitigated likelihood displacement, showing substantial improvements across various metrics.

93
00:08:53,714 --> 00:08:54,994
Evan: Fascinating.

94
00:08:54,994 --> 00:09:06,474
So, in summary, ComPO presents a novel and efficient method for aligning LLMs with human preferences by leveraging comparison oracles and addressing noisy preference pairs.

95
00:09:06,530 --> 00:09:07,660
Ashley: Exactly.

96
00:09:07,660 --> 00:09:18,270
It provides a solid alternative to traditional methods with theoretical guarantees and practical efficiencies, making it a noteworthy advancement in the field of AI preference alignment.

97
00:09:18,374 --> 00:09:22,734
Evan: And that wraps up our detailed look at the Method section of this paper.

98
00:09:23,979 --> 00:09:29,289
Evan: Ashley, let’s dive into the Experimental setup and Results discussed by the authors.

99
00:09:29,289 --> 00:09:34,469
From what I understand, they conducted multiple experiments to evaluate ComPO's performance.

100
00:09:34,469 --> 00:09:37,539
Can you start by telling us about their experimental setup?

101
00:09:37,587 --> 00:09:48,977
Ashley: The authors conducted experiments on various models, including Mistral-7B, Llama-3-8B, Gemma-2-9B-it, Qwen3-4B, and Gemma-3-4B-it.

102
00:09:48,977 --> 00:09:54,667
They used different configurations, such as base and instruction-tuned versions of these models.

103
00:09:54,723 --> 00:09:58,403
Evan: And how did they segment the data for these experiments?

104
00:09:58,467 --> 00:10:05,367
Ashley: They identified and separated preference pairs into clean and noisy subsets using a margin criterion.

105
00:10:05,367 --> 00:10:12,467
Essentially, they used pairs with high confidence levels as clean and those with small likelihood margins as noisy.

106
00:10:12,531 --> 00:10:14,021
Evan: That makes sense.

107
00:10:14,021 --> 00:10:18,291
What specific benchmarks or evaluation metrics did they use?

108
00:10:18,339 --> 00:10:25,389
Ashley: They utilized several benchmarks, including AlpacaEval 2, Arena-Hard, and MT-Bench.

109
00:10:25,389 --> 00:10:35,799
For AlpacaEval 2, they reported both raw win rate (WR) and length-controlled win rate (LC), which adjusts preferences based on response length.

110
00:10:35,919 --> 00:10:37,079
Evan: Interesting.

111
00:10:37,079 --> 00:10:39,819
What were some of their primary findings?

112
00:10:39,867 --> 00:10:41,887
Ashley: The results were quite promising.

113
00:10:41,887 --> 00:10:50,877
For instance, applying ComPO to DPOclean, the model trained on only clean pairs, improved the length-controlled win rate significantly.

114
00:10:50,877 --> 00:10:56,887
In some instances, they observed up to a 2.5% increase in length-controlled win rate.

115
00:10:56,961 --> 00:11:01,671
Evan: Did they compare ComPO directly with any baseline methods?

116
00:11:01,731 --> 00:11:03,141
Ashley: Yes, they did.

117
00:11:03,141 --> 00:11:10,661
They compared DPO with ComPO directly and also applied ComPO as an augmentation to existing SimPO checkpoints.

118
00:11:10,661 --> 00:11:17,471
Additionally, they performed ablation studies to understand the effect of different perturbation numbers and gradient thresholds.

119
00:11:17,523 --> 00:11:21,963
Evan: Were there any notable comparisons or findings from these studies?

120
00:11:22,011 --> 00:11:23,171
Ashley: Certainly.

121
00:11:23,171 --> 00:11:27,371
One notable comparison was between offline ComPO and SimPO.

122
00:11:27,371 --> 00:11:33,111
The incorporation of ComPO into SimPO showed consistent improvements across various benchmarks.

123
00:11:33,111 --> 00:11:41,411
For example, SimPO+ComPO surpassed SimPO in AlpacaEval 2’s length-controlled metrics by about 1.5%.

124
00:11:41,475 --> 00:11:42,885
Evan: That’s impressive.

125
00:11:42,885 --> 00:11:46,935
And how about the specific metrics and diagnostic tools they used?

126
00:11:46,995 --> 00:11:52,255
Ashley: They used pair-level likelihood diagnostics to measure the effectiveness of ComPO.

127
00:11:52,255 --> 00:11:58,415
This helped them understand how well ComPO mitigated likelihood displacement and improved model alignment.

128
00:11:58,415 --> 00:12:02,295
These diagnostics provided concrete evidence of its success.

129
00:12:02,355 --> 00:12:07,015
Evan: How efficient is ComPO in practical, real-world applications?

130
00:12:07,059 --> 00:12:10,779
Ashley: The practical implementation was designed to be highly efficient.

131
00:12:10,779 --> 00:12:16,719
They applied perturbations only to the output layer, significantly reducing computational overhead.

132
00:12:16,719 --> 00:12:24,179
For example, perturbing only 1% of the output layer’s parameters was often sufficient to achieve significant gains.

133
00:12:24,243 --> 00:12:29,743
Evan: That’s a game-changer for models that require frequent updates and real-time performance.

134
00:12:29,787 --> 00:12:30,807
Ashley: Indeed.

135
00:12:30,807 --> 00:12:40,667
Moreover, they experimented with online ComPO, which incorporates damping mechanisms such as reverse-KL control using unlabeled policy generations.

136
00:12:40,761 --> 00:12:45,351
Evan: And how did the online experiments pan out compared to the offline ones?

137
00:12:45,411 --> 00:12:51,011
Ashley: In their experiments, adding online damping and replay resulted in even better performance.

138
00:12:51,011 --> 00:13:02,031
For example, online ComPO showed improvements in AlpacaEval 2 and Arena-Hard benchmarks, corroborating the effectiveness of incorporating unlabeled policy generations.

139
00:13:02,091 --> 00:13:07,231
Evan: Did they mention any specific challenges or limitations during their experiments?

140
00:13:07,275 --> 00:13:12,685
Ashley: One challenge they noted was the computational cost associated with larger perturbation counts.

141
00:13:12,685 --> 00:13:18,775
Although necessary for high precision, it increased computation time, which is a trade-off for accuracy.

142
00:13:18,879 --> 00:13:24,359
Evan: Were there any particular configurations or models where ComPO particularly excelled?

143
00:13:24,411 --> 00:13:34,121
Ashley: Yes, the Mistral-7B-Instruct and Llama-3-8B-Instruct models showed considerable enhancements in performance benchmarks when augmented with ComPO.

144
00:13:34,121 --> 00:13:38,051
Especially in length-controlled scenarios, their performance stood out.

145
00:13:38,115 --> 00:13:43,715
Evan: So, what’s the takeaway from the experiment section about the practical use of ComPO?

146
00:13:43,779 --> 00:13:53,049
Ashley: The experiments clearly demonstrate that ComPO not only mitigates likelihood displacement but also improves overall model performance efficiently.

147
00:13:53,049 --> 00:13:59,839
By leveraging noisy preference pairs, it offers a robust alternative to traditional direct alignment methods.

148
00:13:59,883 --> 00:14:03,863
Evan: That wraps up our discussion on the Experiment section of the paper.

149
00:14:05,177 --> 00:14:11,757
Evan: Ashley, it's time to look at how this research fits into the broader landscape of AI preference alignment.

150
00:14:11,757 --> 00:14:14,357
Let's discuss the Related Work section.

151
00:14:14,405 --> 00:14:15,775
Ashley: Certainly, Evan.

152
00:14:15,775 --> 00:14:25,865
The authors of this paper position their work against a rich backdrop of previous research in preference alignment, zeroth-order optimization, and related methodologies.

153
00:14:25,925 --> 00:14:28,815
Evan: Let's start with preference alignment methods.

154
00:14:28,815 --> 00:14:30,305
What did they highlight here?

155
00:14:30,365 --> 00:14:43,285
Ashley: They discuss a variety of direct preference alignment methods, beginning with Direct Preference Optimization, or DPO, which optimizes policies directly using preference pairs without the need for learning a reward model.

156
00:14:43,409 --> 00:14:48,589
Evan: And what are some of the limitations of DPO that the authors pointed out?

157
00:14:48,683 --> 00:14:49,913
Ashley: Great question.

158
00:14:49,913 --> 00:15:00,713
DPO tends to suffer from likelihood displacement, where the model inadvertently shifts probability mass away from preferred responses, potentially resulting in unsafe behavior.

159
00:15:00,773 --> 00:15:02,733
Evan: That sounds problematic.

160
00:15:02,733 --> 00:15:05,813
How do other works attempt to address these limitations?

161
00:15:05,861 --> 00:15:08,231
Ashley: Several approaches have been proposed.

162
00:15:08,231 --> 00:15:17,571
For instance, Azar et al. (2024) extended DPO to a wider class of problems without explicitly introducing a reward function.

163
00:15:17,571 --> 00:15:26,421
Another approach, Rafailov et al. (2024b), modified DPO to handle token-level transitions for more granular control.

164
00:15:26,477 --> 00:15:27,677
Evan: Interesting.

165
00:15:27,677 --> 00:15:30,997
And how does ComPO fit within this context?

166
00:15:31,061 --> 00:15:39,591
Ashley: ComPO is designed to address the issue of noisy preference pairs in a novel way by using comparison oracles rather than filtering them out.

167
00:15:39,591 --> 00:15:45,321
This makes it more robust against the likelihood displacement that plagues traditional DPO methods.

168
00:15:45,365 --> 00:15:46,655
Evan: Fascinating.

169
00:15:46,655 --> 00:15:49,555
What about zeroth-order optimization methods?

170
00:15:49,555 --> 00:15:51,245
Were they discussed as well?

171
00:15:51,293 --> 00:15:52,653
Ashley: Yes, indeed.

172
00:15:52,653 --> 00:15:59,963
Zeroth-order optimization methods have been explored extensively in literature for settings where the gradients aren't directly accessible.

173
00:15:59,963 --> 00:16:09,053
Specifically, the authors reference works like Flaxman et al. (2005) and Duchi et al. (2015) for foundational techniques.

174
00:16:09,101 --> 00:16:10,071
Evan: I see.

175
00:16:10,071 --> 00:16:15,961
And did they mention any specific zeroth-order methods that are particularly relevant to their work?

176
00:16:16,013 --> 00:16:23,363
Ashley: The paper builds on one-bit, comparison-based estimators, as discussed in Cai et al. (2022a).

177
00:16:23,363 --> 00:16:30,573
These approaches are useful for high-dimensional settings and sparse gradients, aligning well with the problem ComPO seeks to solve.

178
00:16:30,629 --> 00:16:34,249
Evan: It's amazing to see how interconnected these methods are.

179
00:16:34,249 --> 00:16:37,749
Were there any other areas of related work they touched upon?

180
00:16:37,805 --> 00:16:41,965
Ashley: They also delved into research on learning from noisy preference data.

181
00:16:41,965 --> 00:16:55,285
For instance, Razin et al. (2025) explored filtering out problematic preference pairs, a method ComPO contrasts by leveraging information from these noisy pairs rather than discarding them.

182
00:16:55,419 --> 00:17:01,049
Evan: How does ComPO's approach compare to other methods addressing noisy data directly?

183
00:17:01,109 --> 00:17:07,269
Ashley: ComPO stands out because it treats low-margin preference pairs as valuable sources of comparative signals.

184
00:17:07,269 --> 00:17:11,579
This is different from other methods that might exclude or down-weight such data.

185
00:17:11,579 --> 00:17:21,809
For example, Amini et al. (2024) worked on methods to handle corrupted preference labels, but not quite in the same comparative manner that ComPO employs.

186
00:17:21,869 --> 00:17:22,729
Evan: Got it.

187
00:17:22,729 --> 00:17:26,609
It's clear that ComPO brings a unique angle to the table.

188
00:17:26,609 --> 00:17:31,469
Any notable mentions on the practical challenges or theoretical guarantees?

189
00:17:31,517 --> 00:17:32,607
Ashley: Indeed.

190
00:17:32,607 --> 00:17:41,527
The paper mentions the practical challenge of computational overhead, which they address through output-layer perturbations and sparse gradient estimations.

191
00:17:41,527 --> 00:17:49,217
The theoretical framework they provide, including the convergence guarantees, also sets their work apart from more heuristic approaches.

192
00:17:49,277 --> 00:17:54,017
Evan: That should provide a lot of confidence for practitioners looking to adopt this method.

193
00:17:54,017 --> 00:17:57,377
Was there anything else noteworthy in the Related Work section?

194
00:17:57,437 --> 00:18:06,267
Ashley: One last point is their relation to preference fine-tuning methods like Song et al. (2024b), which emphasize dataset coverage.

195
00:18:06,267 --> 00:18:14,637
ComPO complements this by ensuring local coverage through comparisons without needing explicit reward modeling, adding another layer of robustness.

196
00:18:14,693 --> 00:18:18,893
Evan: That's a comprehensive wrap-up of the Related Work section.

197
00:18:20,142 --> 00:18:26,642
Evan: Ashley, let's wrap up our discussion by summarizing the key contributions and takeaways from this paper.

198
00:18:26,694 --> 00:18:40,994
Ashley: The primary contribution of this paper is the introduction of Comparison-based Preference Optimization, or ComPO, as a novel zeroth-order alignment method that effectively addresses the issue of likelihood displacement in large language models.

199
00:18:41,106 --> 00:18:53,786
Evan: By leveraging noisy preference pairs through comparison oracles, ComPO can utilize comparative signals that are otherwise discarded by traditional methods, leading to improved alignment with human preferences.

200
00:18:53,838 --> 00:18:54,788
Ashley: Right.

201
00:18:54,788 --> 00:19:01,578
What stands out is that ComPO achieves this without directly optimizing a noisy differentiable preference loss.

202
00:19:01,578 --> 00:19:08,638
It updates the policy in a more stable and robust manner by evaluating directional signals from perturbed policies.

203
00:19:08,694 --> 00:19:10,364
Evan: And they didn't stop there.

204
00:19:10,364 --> 00:19:22,474
The paper also introduces an online version of ComPO, which integrates reverse-KL control to ensure policy updates don’t deviate too much from a reference policy, thus maintaining local coverage.

205
00:19:22,518 --> 00:19:30,768
Ashley: This online component is particularly useful for real-time applications, where maintaining alignment without extensive retraining is crucial.

206
00:19:30,768 --> 00:19:36,898
Their experiments demonstrated significant performance improvements across several models and benchmarks.

207
00:19:36,992 --> 00:19:48,062
Evan: Overall, ComPO presents a robust alternative to traditional direct alignment methods, providing a practical and theoretically sound solution to mitigate likelihood displacement.

208
00:19:48,126 --> 00:19:57,906
Ashley: Indeed, it’s a noteworthy advancement in AI preference alignment, offering efficient and effective alignment while addressing the shortcomings of previous methods.

209
00:19:57,966 --> 00:20:01,086
Evan: That brings us to the end of today's episode.

210
00:20:01,134 --> 00:20:08,834
Ashley: We hope you enjoyed our deep dive into the paper 'A Zeroth-Order Paradigm for LLM Preference Alignment.'

211
00:20:08,946 --> 00:20:11,326
Evan: As always, thank you for listening.

212
00:20:11,326 --> 00:20:22,646
Don't forget to subscribe to Daily Paper Cast and stay tuned for more insights and discussions on the latest research in AI, NLP, computer vision, and related fields.

213
00:20:22,710 --> 00:20:25,940
Ashley: Join us again tomorrow for another exciting episode.

214
00:20:25,940 --> 00:20:29,390
Until then, keep exploring and stay curious.

215
00:20:29,514 --> 00:20:32,254
Evan: Take care and see you next time!