1 00:00:03,100 --> 00:00:05,720 Evan: Welcome to Daily Paper Cast. 2 00:00:05,784 --> 00:00:14,194 Ashley: Today's paper comes from the Hugging Face daily paper list of October 2, 2026, and has received 112 upvotes. 3 00:00:14,194 --> 00:00:24,524 The title is 'Adaptive Reward Routing: Dynamic Multi-Reward Optimization for Joint Audio-Video Diffusion via Forward-Process RL.' 4 00:00:24,576 --> 00:00:25,396 Evan: Great. 5 00:00:25,396 --> 00:00:31,726 The first two authors are Songlin Yang and Xiaotong Zhao, and the corresponding author is Jiacheng Zhang. 6 00:00:31,726 --> 00:00:36,836 The organizations involved are MMLab at HKUST and Tencent Video. 7 00:00:36,888 --> 00:00:39,928 Ashley: Alright, let's dive into the Introduction section. 8 00:00:39,928 --> 00:00:48,468 Recent advances in joint audio-video diffusion models are enabling the generation of visual and audio content from text prompts simultaneously. 9 00:00:48,468 --> 00:01:01,588 This capability hinges on meeting several objectives: ensuring high visual and audio quality specific to each modality, aligning semantic content across modalities, and achieving precise temporal synchronization. 10 00:01:01,632 --> 00:01:07,422 Evan: It sounds like capturing all these elements in a single model is quite challenging. 11 00:01:07,422 --> 00:01:11,412 How have approaches like Reinforcement Learning helped in this regard? 12 00:01:11,472 --> 00:01:12,672 Ashley: Indeed, Evan. 13 00:01:12,672 --> 00:01:22,432 Reward-guided RL, especially in the context of diffusion models, offers a promising paradigm by defining multiple reward signals that cater to each of these objectives. 14 00:01:22,432 --> 00:01:30,572 Methods such as GRPO-based strategies and DiffusionNFT optimize models by expressing requirements through these diverse rewards. 15 00:01:30,624 --> 00:01:37,974 Evan: So, if I understand correctly, these methods rely on fixed reward routes and weights. 16 00:01:37,974 --> 00:01:42,444 Does this paper address any limitations of these existing approaches? 17 00:01:42,504 --> 00:01:43,494 Ashley: Precisely. 18 00:01:43,494 --> 00:01:51,264 Existing methods generally use static routing and reward weights, which can become outdated as the model functions evolve during training. 19 00:01:51,264 --> 00:01:55,754 The paper introduces Adaptive Reward Routing to tackle these limitations. 20 00:01:55,754 --> 00:02:03,984 It adapts both the locations of reward-driven updates and the coordination of multiple, possibly competing rewards through the ongoing training. 21 00:02:04,032 --> 00:02:09,392 Evan: So, Adaptive Reward Routing adjusts to the model’s changing needs during training. 22 00:02:09,392 --> 00:02:12,452 What exactly are the components of this proposed system? 23 00:02:12,504 --> 00:02:15,214 Ashley: The system comprises two main components. 24 00:02:15,214 --> 00:02:26,194 First, there's Cross-Modal Influence-Guided Routing, which uses bidirectional cross-attention responses to efficiently localize where updates should occur across the model layers and tokens. 25 00:02:26,194 --> 00:02:33,224 This dynamic weighting helps in influencing the gradients during cross-modal interactions without additional interventions. 26 00:02:33,288 --> 00:02:34,448 Evan: Interesting. 27 00:02:34,448 --> 00:02:36,208 And what's the second component? 28 00:02:36,264 --> 00:02:41,354 Ashley: The second component is Preference-Preserving Modality-Aware Reweighting. 29 00:02:41,354 --> 00:02:48,844 It resolves conflicts between rewards by using predefined preference priors and branch-specific reward-gradient interactions. 30 00:02:48,844 --> 00:02:55,484 Essentially, this ensures that even less dominant but essential objectives are not overshadowed by the stronger ones. 31 00:02:55,536 --> 00:03:00,956 Evan: How do these components work together to improve the model's performance? 32 00:03:01,008 --> 00:03:09,408 Ashley: Together, these components allow the model to adapt in terms of reward coordination and update localization dynamically throughout training. 33 00:03:09,408 --> 00:03:19,288 Extensive experiments have shown consistent improvements in modalities' quality, semantic consistency, and synchronization when compared to strong RL baselines. 34 00:03:19,394 --> 00:03:30,264 Evan: Essentially, the paper proposes refining both the reward adaptation and routing process for joint diffusion models, providing more stable and effective post-training outcomes. 35 00:03:30,312 --> 00:03:31,382 Ashley: Exactly. 36 00:03:31,382 --> 00:03:38,392 That concludes the Introduction section, where we've discussed the background, objectives, and contributions of this paper. 37 00:03:39,710 --> 00:03:45,950 Evan: Let's dive into the Method section to understand how the authors have structured their approach. 38 00:03:46,010 --> 00:03:47,390 Ashley: Sure, Evan. 39 00:03:47,390 --> 00:03:56,460 The paper introduces a framework called Adaptive Reward Routing for forward-process RL in joint audio-video diffusion models. 40 00:03:56,460 --> 00:04:05,110 This framework consists of two main components: Cross-Modal Influence-Guided Routing and Preference-Preserving Modality-Aware Reweighting. 41 00:04:05,232 --> 00:04:07,982 Evan: Right, we briefly mentioned these earlier. 42 00:04:07,982 --> 00:04:11,582 Can you unpack Cross-Modal Influence-Guided Routing? 43 00:04:11,642 --> 00:04:19,232 Ashley: Cross-Modal Influence-Guided Routing dynamically determines where reward-driven updates should act in the model. 44 00:04:19,232 --> 00:04:27,332 Essentially, it uses bidirectional cross-attention responses as proxies for evolving cross-modal influence. 45 00:04:27,332 --> 00:04:33,002 This process involves two steps: token-level routing and layer-level routing. 46 00:04:33,080 --> 00:04:36,070 Evan: What does token-level routing involve? 47 00:04:36,122 --> 00:04:42,662 Ashley: Token-level routing assigns weights to individual tokens based on their influence in cross-modal interactions. 48 00:04:42,662 --> 00:04:51,882 Specifically, it averages the pre-gate cross-attention responses over selected timesteps and normalizes these scores to form positive loss weights. 49 00:04:51,882 --> 00:04:56,202 These weights help emphasize more influential tokens during model updates. 50 00:04:56,318 --> 00:04:58,758 Evan: And how about layer-level routing? 51 00:04:58,802 --> 00:05:02,672 Ashley: Layer-level routing adapts the gradient flow across different layers. 52 00:05:02,672 --> 00:05:10,422 It takes the same pre-gate cross-attention responses and averages them over tokens and selected timesteps to derive a layer score. 53 00:05:10,422 --> 00:05:16,152 This score is then used to create a detachment coefficient that scales the gradients in backpropagation. 54 00:05:16,152 --> 00:05:23,282 Essentially, layers that have stronger cross-modal influence retain more gradient while weaker ones are progressively detached. 55 00:05:23,400 --> 00:05:28,980 Evan: So, the framework adapts weights at both the token and layer levels. 56 00:05:28,980 --> 00:05:34,850 What role does Preference-Preserving Modality-Aware Reweighting play in this setup? 57 00:05:34,898 --> 00:05:41,008 Ashley: Preference-Preserving Modality-Aware Reweighting deals with how multiple rewards are coordinated. 58 00:05:41,008 --> 00:05:46,018 It resolves conflicts between rewards while preserving user-defined preferences. 59 00:05:46,018 --> 00:05:53,318 This component incorporates three sub-strategies: branch-aware balancing, residual corrections, and warm-up. 60 00:05:53,438 --> 00:05:54,788 Evan: Interesting. 61 00:05:54,788 --> 00:05:57,758 Can you break down these sub-strategies for us? 62 00:05:57,818 --> 00:05:58,988 Ashley: Certainly. 63 00:05:58,988 --> 00:06:08,358 Branch-aware balancing assigns reward interactions to the respective modality branches that they supervise, such as video and audio branches. 64 00:06:08,358 --> 00:06:17,018 Residual corrections then adjust predefined reward weights by incorporating coefficients derived from gradient-based conflict estimation. 65 00:06:17,018 --> 00:06:22,638 This ensures that objectives do not get entirely overridden by early noisy gradients. 66 00:06:22,638 --> 00:06:32,798 Lastly, warm-up stabilizes the reward reweighting process by delaying dynamic adjustments until the relationships between gradients become reliable. 67 00:06:32,898 --> 00:06:34,078 Evan: Got it. 68 00:06:34,078 --> 00:06:40,868 It sounds like these methods together make the model more adaptable to the changing dynamics during training. 69 00:06:40,868 --> 00:06:44,378 How is this reflected in the actual training process? 70 00:06:44,426 --> 00:06:45,426 Ashley: Exactly. 71 00:06:45,426 --> 00:06:48,986 The complete optimization process involves several steps. 72 00:06:48,986 --> 00:06:53,156 First, reward reweighting decides the contribution of each objective. 73 00:06:53,156 --> 00:06:57,366 Then, branch routing assigns these objectives to target modalities. 74 00:06:57,366 --> 00:07:03,106 After that, token-level routing determines where within each modality loss should be emphasized. 75 00:07:03,106 --> 00:07:08,946 Finally, layer-level routing controls how the gradient flows across modality boundaries. 76 00:07:09,062 --> 00:07:16,182 Evan: How does the framework ensure that these weight changes truly reflect the evolving needs of the model? 77 00:07:16,226 --> 00:07:17,696 Ashley: Great question. 78 00:07:17,696 --> 00:07:24,896 The framework uses the pre-gate cross-attention responses collected during sampling to track influence shifts. 79 00:07:24,896 --> 00:07:35,286 For instance, certain layers and tokens might become more influential as the model fine-tunes, and the adaptive routing ensures these shifts are accounted for. 80 00:07:35,286 --> 00:07:44,586 Additionally, evaluation of reward signals helps in computing advantages and optimality probabilities that further guide the training updates. 81 00:07:44,642 --> 00:07:46,552 Evan: That makes a lot of sense. 82 00:07:46,552 --> 00:08:02,482 To sum it up, the Adaptive Reward Routing framework adapts both the importance and location of updates based on ongoing training dynamics, making it a sophisticated approach to multi-reward optimization in joint audio-video diffusion models. 83 00:08:02,546 --> 00:08:03,916 Ashley: Precisely, Evan. 84 00:08:03,916 --> 00:08:06,826 That concludes the discussion on the Method section. 85 00:08:06,826 --> 00:08:12,026 Next, we'll move into the Experiments section to see how these methods perform in practice. 86 00:08:13,275 --> 00:08:22,115 Evan: Alright, let's dive into the Experiment and Results section to see how this Adaptive Reward Routing framework performs in practice. 87 00:08:22,179 --> 00:08:23,369 Ashley: Sure, Evan. 88 00:08:23,369 --> 00:08:34,929 The authors conducted extensive experiments to evaluate their proposed framework on joint audio-video diffusion models, using two main backbones: LTX-2 and LTX-2.3. 89 00:08:34,929 --> 00:08:43,119 They trained these models with 19,487 audio-video prompts collected from a VGGSound-derived corpus. 90 00:08:43,179 --> 00:08:48,659 Evan: What type of rewards and baselines did they use for these evaluations? 91 00:08:48,723 --> 00:09:02,723 Ashley: They used five complementary reward signals: Video Quality with VideoAlign and HPSv3, Audio Quality with AudioBox Aesthetics, Text-Audio Alignment with CLAP, and Audio-Video Synchronization with DeSync. 92 00:09:02,723 --> 00:09:12,343 For baselines, they compared their method against four settings: the pretrained backbone with no post-training, GDPO, MARBLE, and OmniNFT. 93 00:09:12,387 --> 00:09:17,707 Evan: What metrics did they use to evaluate the performance? 94 00:09:17,763 --> 00:09:31,593 Ashley: They used JavisBench, encompassing diverse audio-video generation scenarios, and evaluated metrics across four groups: AV-Quality, Text-Consistency, AV-Consistency, and AV-Synchrony. 95 00:09:31,593 --> 00:09:49,063 Metrics like Visual Quality (VQ), Audio Quality (AQ), Text-Video and Text-Audio ImageBind similarity (TV-IB and TA-IB), CLIP, CLAP scores, AV-IB, AVHScore, JavisScore, and DeSync were measured. 96 00:09:49,107 --> 00:09:51,987 Evan: That's quite a comprehensive evaluation. 97 00:09:51,987 --> 00:09:56,707 How did the Adaptive Reward Routing framework perform compared to the baselines? 98 00:09:56,763 --> 00:09:58,433 Ashley: Remarkably well. 99 00:09:58,433 --> 00:10:07,363 For both LTX-2 and LTX-2.3 backbones, the proposed method achieved the best results on nine out of ten metrics. 100 00:10:07,363 --> 00:10:21,043 For example, in Visual Quality, they reported scores of 3.336 for LTX-2 and 3.599 for LTX-2.3, outperforming other baselines like OmniNFT. 101 00:10:21,099 --> 00:10:22,649 Evan: That's impressive. 102 00:10:22,649 --> 00:10:25,659 How did the training dynamics look for these models? 103 00:10:25,737 --> 00:10:37,047 Ashley: The training dynamics indicated that the Adaptive Reward Routing reached the highest average reward and maintained favorable trajectories across all five component rewards. 104 00:10:37,047 --> 00:10:41,757 In contrast, baselines made uneven progress across rewards. 105 00:10:41,757 --> 00:10:53,647 For example, AV-DeSync showed that Adaptive Reward Routing achieved a lower value around 0.341, indicating better synchronization performance compared to other models. 106 00:10:53,791 --> 00:11:01,591 Evan: How did the authors validate these improvements were due to their proposed methods and not just better hyperparameters or architectures? 107 00:11:01,665 --> 00:11:03,395 Ashley: Good question, Evan. 108 00:11:03,395 --> 00:11:10,425 The authors conducted comprehensive ablation studies to isolate the contributions of each component. 109 00:11:10,425 --> 00:11:21,305 They showed that the most significant gains came from incorporating both Cross-Modal Influence-Guided Routing and Preference-Preserving Modality-Aware Reweighting. 110 00:11:21,305 --> 00:11:31,055 Ablations confirmed that each added component yielded progressive gains, especially in metrics like AV-Quality and synchronization. 111 00:11:31,107 --> 00:11:35,787 Evan: Can you give an example of one of these ablation studies? 112 00:11:35,835 --> 00:11:48,905 Ashley: For instance, when they evaluated token weighting and layer scaling independently on GDPO, token weighting alone improved metrics like Text-Consistency, while layer scaling significantly boosted AV-Synchrony. 113 00:11:48,905 --> 00:11:52,575 Combining both resulted in the strongest performance overall. 114 00:11:52,665 --> 00:12:01,935 Evan: Did they also investigate the effectiveness of their response proxy for influencing gradients and routing decisions? 115 00:12:01,995 --> 00:12:08,985 Ashley: Yes, they validated the response proxy's accuracy in identifying influential cross-modal layers and tokens. 116 00:12:08,985 --> 00:12:18,175 For example, blocking the cross-modal responses of top-scoring tokens resulted in a larger prediction change, proving these tokens were most influential. 117 00:12:18,175 --> 00:12:23,135 This proxy maintained high accuracy throughout training, unlike a static proxy. 118 00:12:23,187 --> 00:12:24,867 Evan: That's compelling evidence. 119 00:12:24,867 --> 00:12:36,547 So, the Experiment section clearly shows that Adaptive Reward Routing dynamically optimizes joint audio-video diffusion models by effectively coordinating and routing rewards. 120 00:12:36,547 --> 00:12:41,727 It seems like a well-rounded approach that addresses the evolving needs during model training. 121 00:12:41,787 --> 00:12:43,107 Ashley: Exactly, Evan. 122 00:12:43,107 --> 00:12:51,767 This concludes the Experiment section where we've explored how well the proposed framework performs and the detailed methods behind these impressive results. 123 00:12:53,021 --> 00:13:01,261 Evan: Let's move on to the Related Work section to understand how this paper builds on and contrasts with previous research. 124 00:13:01,325 --> 00:13:02,605 Ashley: Sure, Evan. 125 00:13:02,605 --> 00:13:14,165 The Related Work section primarily addresses three areas: joint audio-video generation, reinforcement learning for diffusion models, and multi-reward optimization. 126 00:13:14,213 --> 00:13:18,013 Evan: Alright, let's start with joint audio-video generation. 127 00:13:18,013 --> 00:13:20,833 What have been the significant advancements in this field? 128 00:13:20,915 --> 00:13:30,515 Ashley: Joint audio-video generation has seen notable progress from earlier image diffusion models integrated with temporal modules, as well as large diffusion Transformers. 129 00:13:30,515 --> 00:13:41,025 For example, methods like those by Blattmann et al., 2023 and Guo et al., 2024 employ these strategies to enhance video generation effectively. 130 00:13:41,129 --> 00:13:41,919 Evan: I see. 131 00:13:41,919 --> 00:13:45,849 And how do joint audio-video systems function in this regard? 132 00:13:45,893 --> 00:14:01,363 Ashley: Joint audio-video systems often connect pretrained experts through cross-modal projections or use unified diffusion Transformers, as seen in works by Wang et al., 2025 and Liu et al., 2026. 133 00:14:01,363 --> 00:14:11,353 Alternatively, they couple separate streams via bidirectional cross-attention mechanisms, a method highlighted in HaCohen et al., 2026. 134 00:14:11,405 --> 00:14:20,885 Evan: So, while these strategies enable mutual conditioning across modalities, what challenges do they face? 135 00:14:20,933 --> 00:14:31,873 Ashley: The main challenge lies in the difficulty of defining reward responsibility and gradient routing within these heterogeneous branches, which is less straightforward compared to single-stream models. 136 00:14:31,925 --> 00:14:33,225 Evan: That's insightful. 137 00:14:33,225 --> 00:14:36,955 Now, moving on to reinforcement learning for diffusion models. 138 00:14:36,955 --> 00:14:39,245 What have been some pivotal advancements here? 139 00:14:39,293 --> 00:14:54,333 Ashley: In reinforcement learning for diffusion models, GRPO, or Group Reward-Decoupled Normalization Policy Optimization, has been a significant advancement, as mentioned by Guo et al., 2025. 140 00:14:54,333 --> 00:14:58,733 This method estimates relative advantages without using a critic. 141 00:14:58,733 --> 00:15:10,233 Subsequent methods like Flow-GRPO and DanceGRPO extend online optimization to flow-based generation by incorporating stochastic sampling techniques. 142 00:15:10,277 --> 00:15:11,227 Evan: Interesting. 143 00:15:11,227 --> 00:15:13,997 How about the DiffusionNFT approach? 144 00:15:14,045 --> 00:15:23,825 Ashley: DiffusionNFT, proposed by Zheng et al., 2025, optimizes the forward process using implicit positive and negative policies. 145 00:15:23,825 --> 00:15:32,385 This approach focuses on sampled timesteps rather than reverse-process likelihood estimation, which aligns closely with our paper's methodology. 146 00:15:32,489 --> 00:15:38,389 Evan: It seems like OmniNFT also plays a role in joint audio-video generation. 147 00:15:38,389 --> 00:15:39,789 Can you elaborate? 148 00:15:39,845 --> 00:15:41,165 Ashley: Certainly. 149 00:15:41,165 --> 00:15:51,135 OmniNFT, as described by Zhang et al., 2026, adds modality-wise credit assignment for joint audio-video generation. 150 00:15:51,135 --> 00:16:01,525 It employs layer-wise gradient adjustments and region-wise reweighting but relies on fixed routing rules, which our paper aims to improve upon. 151 00:16:01,619 --> 00:16:02,489 Evan: Got it. 152 00:16:02,489 --> 00:16:06,669 And how does multi-reward optimization fit into the picture? 153 00:16:06,725 --> 00:16:13,725 Ashley: Multi-reward optimization methods address the integration and balance of multiple conflicting objectives. 154 00:16:13,725 --> 00:16:18,565 Fixed scalarization techniques struggle to adapt to changing conflicts. 155 00:16:18,565 --> 00:16:27,625 GDPO preserves reward-specific signals through normalization but still uses predefined aggregation weights, which can be rigid. 156 00:16:27,625 --> 00:16:38,665 Multi-task approaches like those by Desideri, 2012, and Sener & Koltun, 2018, aim for common descent directions and optimize local agreement. 157 00:16:38,717 --> 00:16:39,797 Evan: That's right. 158 00:16:39,797 --> 00:16:45,317 How do methods like MARBLE adapt multi-reward optimization to diffusion RL? 159 00:16:45,365 --> 00:16:58,195 Ashley: MARBLE, discussed by Zhao et al., 2026, adapts multi-task optimization ideas to diffusion RL by dynamically adjusting reward weights through gradient geometry. 160 00:16:58,195 --> 00:17:09,925 However, this approach still faces limitations as it primarily focuses on gradient compatibility rather than objective importance or specific modality responsibility. 161 00:17:10,059 --> 00:17:13,629 Evan: How does the work in our paper build on these prior approaches? 162 00:17:13,685 --> 00:17:29,645 Ashley: Our paper unifies the strengths of these strategies by introducing adaptive routing mechanisms that dynamically localize updates based on evolving cross-modal influence and reward coordination techniques that resolve conflicts while preserving user-defined priorities. 163 00:17:29,645 --> 00:17:36,585 This dual approach offers a more comprehensive and responsive system for joint audio-video diffusion models. 164 00:17:36,629 --> 00:17:52,329 Evan: So essentially, integrating adaptive update routing with preference-preserving reward coordination creates a robust framework that addresses the dynamic needs of joint audio-video generation models, building significantly on past work. 165 00:17:52,373 --> 00:17:53,713 Ashley: Exactly, Evan. 166 00:17:53,713 --> 00:17:56,273 That wraps up the Related Work section. 167 00:17:57,534 --> 00:17:58,774 Evan: Alright, Ashley. 168 00:17:58,774 --> 00:18:03,624 We've covered the Introduction, Method, Experiments, and Related Work. 169 00:18:03,624 --> 00:18:07,634 Let's summarize the key contributions and takeaways of this paper. 170 00:18:07,686 --> 00:18:08,706 Ashley: Sure, Evan. 171 00:18:08,706 --> 00:18:19,586 To wrap it up, this paper presents Adaptive Reward Routing, a dynamic multi-reward optimization framework for joint audio-video diffusion via forward-process reinforcement learning. 172 00:18:19,586 --> 00:18:27,146 The authors propose two main components: Cross-Modal Influence-Guided Routing and Preference-Preserving Modality-Aware Reweighting. 173 00:18:27,198 --> 00:18:33,338 Evan: Right, these components adapt to the model's evolving needs during training. 174 00:18:33,338 --> 00:18:36,778 What are the key benefits observed from this approach? 175 00:18:36,822 --> 00:18:47,002 Ashley: The key benefits include consistent improvements in modality-specific quality, semantic alignment, and audio-video synchronization. 176 00:18:47,002 --> 00:18:59,962 By dynamically localizing updates and intelligently coordinating multiple rewards, the framework ensures that even weak but essential objectives are not suppressed by dominant rewards. 177 00:19:00,066 --> 00:19:17,666 Evan: Experiments showed that Adaptive Reward Routing outperformed strong RL baselines like GDPO, MARBLE, and OmniNFT, achieving the best results across nine out of ten metrics on both LTX-2 and LTX-2.3 backbones. 178 00:19:17,718 --> 00:19:18,748 Ashley: Exactly. 179 00:19:18,748 --> 00:19:28,458 The framework leverages cross-attention responses to track influence shifts and accurately updates weights, ensuring the model remains responsive to its training dynamics. 180 00:19:28,458 --> 00:19:34,398 Ablation studies further validated the complementary benefits of the routing and reweighting components. 181 00:19:34,486 --> 00:19:46,406 Evan: To sum up, Adaptive Reward Routing provides a sophisticated method for optimizing joint audio-video diffusion models by dynamically adapting to the evolving needs of the model during training. 182 00:19:46,406 --> 00:19:55,446 It's a significant contribution that builds on previous research in the areas of joint generation, reinforcement learning, and multi-reward optimization. 183 00:19:55,494 --> 00:19:58,124 Ashley: That's all for today’s episode. 184 00:19:58,124 --> 00:20:00,824 Thank you for tuning in to Daily Paper Cast. 185 00:20:00,824 --> 00:20:03,634 We hope you found our discussion insightful. 186 00:20:03,678 --> 00:20:09,498 Evan: Be sure to join us again tomorrow for more in-depth looks at fascinating new research papers. 187 00:20:09,498 --> 00:20:13,178 Until then, keep exploring and stay curious! 188 00:20:13,260 --> 00:20:14,930 Ashley: Goodbye everyone! 189 00:20:14,982 --> 00:20:15,982 Evan: Goodbye!