1
00:00:03,000 --> 00:00:05,880
Evan: Welcome to Daily Paper Cast.

2
00:00:05,928 --> 00:00:12,868
Ashley: Today's paper is from the Hugging Face daily paper list of September 9, 2026, with 29 upvotes.

3
00:00:12,972 --> 00:00:18,772
Evan: The paper is titled 'Miles v0.1: Production-Level Post-Training'.

4
00:00:18,816 --> 00:00:22,206
Ashley: The first two authors are Tom Chen and Mao Cheng.

5
00:00:22,206 --> 00:00:25,796
The corresponding author is Shi Dong from RadixArk.

6
00:00:25,898 --> 00:00:28,728
Evan: Let's dive into the Introduction section.

7
00:00:28,776 --> 00:00:32,666
Ashley: Post-training turns a pretrained language model into a useful one.

8
00:00:32,666 --> 00:00:37,676
At frontier scale, post-training poses substantial challenges for training systems.

9
00:00:37,788 --> 00:00:42,768
Evan: What makes post-training at frontier scale particularly challenging?

10
00:00:42,816 --> 00:00:51,896
Ashley: For example, reinforcement learning, or RL, for large language models no longer follows a simple generate-then-update loop over short completions.

11
00:00:51,896 --> 00:00:57,596
Rollouts span multiple turns, use tools, and let the model act in an external environment.

12
00:00:57,596 --> 00:01:02,496
The models producing the rollouts are often trillion-parameter mixtures of experts.

13
00:01:02,544 --> 00:01:04,344
Evan: That sounds complex.

14
00:01:04,344 --> 00:01:07,504
What are some specific challenges these systems face?

15
00:01:07,560 --> 00:01:11,650
Ashley: Sustaining high end-to-end hardware utilization is difficult.

16
00:01:11,650 --> 00:01:18,440
The system must juggle latency-sensitive rollouts with throughput-oriented training, introducing bubbles and idle time.

17
00:01:18,440 --> 00:01:26,280
Additionally, the numerical gap between the rollout engines and the trainer can become large enough to invalidate the objective outright.

18
00:01:26,378 --> 00:01:27,748
Evan: Interesting.

19
00:01:27,748 --> 00:01:31,428
So, how does Miles aim to tackle these problems?

20
00:01:31,488 --> 00:01:37,408
Ashley: Miles v0.1 is a full-stack, production-ready system for frontier post-training.

21
00:01:37,408 --> 00:01:44,828
It builds on the clean design of slime and centers on one principle: components should be verified, clean, and customizable.

22
00:01:44,880 --> 00:01:49,540
Evan: Could you elaborate on the core design principles of Miles?

23
00:01:49,614 --> 00:01:50,444
Ashley: Of course.

24
00:01:50,444 --> 00:01:56,354
The core principles of Miles are accuracy, efficiency, reliability, and scalability.

25
00:01:56,354 --> 00:02:01,844
The system aims to make frontier-scale RL accessible to researchers and enterprises alike.

26
00:02:01,966 --> 00:02:06,196
Evan: And what are the key components of Miles v0.1?

27
00:02:06,240 --> 00:02:22,020
Ashley: Miles encompasses several key components: rollout engines built on SGLang, a trainer with a choice of two backends—NVIDIA Megatron-LM and PyTorch FSDP—and three weight-synchronization transports for different deployment topologies.

28
00:02:22,140 --> 00:02:24,840
Evan: That sounds comprehensive.

29
00:02:24,840 --> 00:02:29,620
Does Miles support other types of training beyond full-parameter RL?

30
00:02:29,664 --> 00:02:31,124
Ashley: Yes, it does.

31
00:02:31,124 --> 00:02:43,964
Beyond full-parameter RL, Miles also supports Low-Rank Adaptation or LoRA RL, on-policy distillation, supervised fine-tuning, and true-on-policy rollout-training alignment.

32
00:02:43,964 --> 00:02:47,564
It even extends the same architecture to diffusion models.

33
00:02:47,616 --> 00:02:52,956
Evan: Can you provide a specific example demonstrating Miles' capabilities?

34
00:02:53,016 --> 00:02:54,156
Ashley: Certainly.

35
00:02:54,156 --> 00:03:15,776
The paper closes with an end-to-end case study: a fully asynchronous agentic RL on a GLM-5.2 744B-A40B model over terminal-use coding tasks, running on 64 NVIDIA GB300 GPUs with a median step time of 263 seconds over the first 30 measured steps.

36
00:03:15,840 --> 00:03:17,480
Evan: That's impressive.

37
00:03:17,480 --> 00:03:26,160
It sounds like Miles v0.1 is a robust system designed to handle the complexities of frontier-scale RL.

38
00:03:26,208 --> 00:03:27,278
Ashley: Indeed.

39
00:03:27,278 --> 00:03:30,768
And that wraps up the Introduction section of today's paper.

40
00:03:32,018 --> 00:03:38,638
Evan: Let's move on to the method section and delve into how Miles v0.1 operates in detail.

41
00:03:38,690 --> 00:03:42,170
Ashley: We'll start with the core RL loop that Miles employs.

42
00:03:42,170 --> 00:03:49,710
A reinforcement learning training job in Miles cycles through three key stages: rollout, training, and weight update.

43
00:03:49,834 --> 00:03:53,334
Evan: Can you break down each of these stages for us?

44
00:03:53,378 --> 00:03:54,218
Ashley: Sure.

45
00:03:54,218 --> 00:03:56,958
First, we have the Rollout stage.

46
00:03:56,958 --> 00:03:59,618
SGLang engines generate trajectories.

47
00:03:59,618 --> 00:04:10,238
In agentic RL, where the model acts across multiple turns, each rollout session interacts with its own isolated environment, which executes actions and provides rewards.

48
00:04:10,358 --> 00:04:15,098
Evan: So, SGLang is crucial for generating these multi-turn rollouts.

49
00:04:15,098 --> 00:04:16,478
What happens after that?

50
00:04:16,538 --> 00:04:19,238
Ashley: Next, we move to the Training stage.

51
00:04:19,238 --> 00:04:25,888
Here, the trainer consumes completed trajectory groups, computes the RL loss, and updates the policy.

52
00:04:25,888 --> 00:04:32,238
Miles supports two backends for this task: NVIDIA Megatron-LM and PyTorch FSDP.

53
00:04:32,362 --> 00:04:35,612
Evan: And finally, we have the weight update stage.

54
00:04:35,612 --> 00:04:36,902
How does that work?

55
00:04:36,962 --> 00:04:44,152
Ashley: After each training step, Miles updates the RL policy by synchronizing the new weights with the rollout engines.

56
00:04:44,152 --> 00:04:49,782
The goal is to minimize interruptions to in-flight rollouts, ensuring seamless progress.

57
00:04:49,826 --> 00:04:53,486
Evan: I've heard that synchronization can be a major bottleneck.

58
00:04:53,486 --> 00:04:55,486
How does Miles handle this?

59
00:04:55,568 --> 00:04:56,778
Ashley: Good question.

60
00:04:56,778 --> 00:05:09,018
To mitigate this, Miles offers different weight-synchronization transports based on the deployment topology: NCCL broadcast, peer-to-peer RDMA, or CUDA IPC when colocated.

61
00:05:09,074 --> 00:05:11,304
Evan: That's quite a variety.

62
00:05:11,304 --> 00:05:15,424
But I've also heard about fully asynchronous RL in Miles.

63
00:05:15,424 --> 00:05:16,794
How does that work?

64
00:05:16,850 --> 00:05:22,380
Ashley: Fully asynchronous RL allows rollout generation and training to progress concurrently.

65
00:05:22,380 --> 00:05:31,350
The rollout engines generate trajectories continuously while the trainer consumes whichever trajectories are ready to keep both stages busy simultaneously.

66
00:05:31,394 --> 00:05:33,754
Evan: So, they don’t wait for each other.

67
00:05:33,754 --> 00:05:35,194
That sounds efficient.

68
00:05:35,194 --> 00:05:38,654
What about throughput and fidelity during rollout generation?

69
00:05:38,714 --> 00:05:41,794
Ashley: Indeed, throughput and fidelity are crucial.

70
00:05:41,794 --> 00:05:48,474
Miles uses SGLang for request routing to preserve cache locality and decouple generation from training.

71
00:05:48,474 --> 00:05:54,314
This ensures that the trainer's view of trajectories remains consistent with what the policy actually sampled.

72
00:05:54,432 --> 00:05:57,042
Evan: Consistency must be tricky, right?

73
00:05:57,042 --> 00:06:01,542
How does Miles ensure token fidelity in multi-turn scenarios?

74
00:06:01,586 --> 00:06:09,396
Ashley: Miles employs a Token-In-Token-Out, or TITO, session server to maintain the exact tokens sampled by the policy.

75
00:06:09,396 --> 00:06:17,666
The server controls tokenization, preserving rollout log-probabilities, and ensures that training sequences match the generated tokens exactly.

76
00:06:17,774 --> 00:06:23,454
Evan: Can you explain how Miles maintains the balance between different components during training?

77
00:06:23,498 --> 00:06:30,028
Ashley: Miles keeps the rollout engines busy by replenishing rollout generation capacity as trajectories finish.

78
00:06:30,028 --> 00:06:36,308
A background worker maintains a consistent number of trajectories in generation to minimize trainer wait time.

79
00:06:36,308 --> 00:06:42,578
The data buffer also decouples generation and training, allowing each to operate at its own pace.

80
00:06:42,626 --> 00:06:44,756
Evan: That sounds well-coordinated.

81
00:06:44,756 --> 00:06:48,026
What about handling longer rollout contexts?

82
00:06:48,074 --> 00:06:59,414
Ashley: To accommodate longer contexts, Miles uses session-aware routing to maintain affinity for multi-turn episodes, ensuring that data-parallel ranks leverage cached prefix tokens efficiently.

83
00:06:59,414 --> 00:07:05,914
Additionally, there’s a fallback in the form of session server to track session identity and supply routing keys.

84
00:07:05,978 --> 00:07:11,058
Evan: With all these components, how does Miles manage asynchronous evaluation?

85
00:07:11,114 --> 00:07:14,424
Ashley: Asynchronous evaluation runs alongside training.

86
00:07:14,424 --> 00:07:21,404
Miles offers three evaluation modes: shared engines, a dedicated GPU fleet, or an external service.

87
00:07:21,404 --> 00:07:33,554
In shared mode, rollout engines pause new generation while evaluating, while the other modes keep evaluation off the critical training path, using snapshots to run evaluations without hindering training.

88
00:07:33,602 --> 00:07:40,322
Evan: How does it ensure that the evaluations are accurate and reflective of the policy versions?

89
00:07:40,400 --> 00:07:45,060
Ashley: Each evaluation score is tied to the specific policy version that produced it.

90
00:07:45,060 --> 00:07:49,910
During evaluation, weights are verified, ensuring consistency across the fleet.

91
00:07:49,910 --> 00:07:55,210
Even when an evaluation fails, training continues, with reasons logged appropriately.

92
00:07:55,274 --> 00:07:59,394
Evan: What types of environments can Miles handle for agentic RL?

93
00:07:59,450 --> 00:08:06,450
Ashley: Miles is versatile in accommodating various environments by organizing integration as three nested plug-in layers.

94
00:08:06,450 --> 00:08:14,870
It supports environments that provide different scopes of control, ranging from simple episode loops to complex batching and rewards management.

95
00:08:14,930 --> 00:08:16,730
Evan: That’s comprehensive.

96
00:08:16,730 --> 00:08:19,110
What else does the method section cover?

97
00:08:19,154 --> 00:08:29,074
Ashley: It touches on RL router handling for mixture-of-experts models, ensuring expert routing fidelity with R3, termed as Efficient Rollout Routing Replay.

98
00:08:29,074 --> 00:08:37,354
For mixture-of-experts models, the router holds routing tensors to replay expert assignments, which is vital for maintaining training stability.

99
00:08:37,418 --> 00:08:40,528
Evan: You mentioned memory efficiency earlier.

100
00:08:40,528 --> 00:08:42,218
Can you elaborate on that?

101
00:08:42,266 --> 00:08:43,456
Ashley: Certainly.

102
00:08:43,456 --> 00:08:51,856
Miles employs memory-efficient techniques like evicting paused actors and streaming optimizer state to reduce GPU memory consumption.

103
00:08:51,856 --> 00:09:01,126
Offloading optimizer states to CPU or NVMe further ensures that the training actor fits within the memory limits during active computation stages.

104
00:09:01,178 --> 00:09:03,858
Evan: What about low-precision training?

105
00:09:03,858 --> 00:09:05,578
Does Miles support that?

106
00:09:05,642 --> 00:09:13,692
Ashley: Yes, lower-precision formats like FP8 and NVFP4 are supported, improving computational efficiency.

107
00:09:13,692 --> 00:09:20,282
Miles ensures that both rollout and training stages use the same quantization logic to avoid discrepancies.

108
00:09:20,330 --> 00:09:21,580
Evan: Fascinating.

109
00:09:21,580 --> 00:09:28,670
This method section is indeed thorough in covering the operational aspects of Miles v0.1.

110
00:09:28,730 --> 00:09:29,800
Ashley: Indeed.

111
00:09:29,800 --> 00:09:32,890
And that brings us to the end of the Method section.

112
00:09:34,155 --> 00:09:40,935
Evan: Moving on, let's discuss the experiments and results presented in the Miles v0.1 paper.

113
00:09:40,995 --> 00:09:41,765
Ashley: Sure.

114
00:09:41,765 --> 00:09:54,155
The paper presents a case study where Miles v0.1 was used for fully asynchronous agentic RL on the GLM-5.2 744B-A40B model over terminal-use coding tasks.

115
00:09:54,155 --> 00:09:58,275
This was run on 64 NVIDIA GB300 GPUs.

116
00:09:58,323 --> 00:10:00,703
Evan: 64 GPUs?

117
00:10:00,703 --> 00:10:02,263
That's impressive.

118
00:10:02,263 --> 00:10:04,783
What did the experimental setup look like?

119
00:10:04,827 --> 00:10:11,987
Ashley: The setup divided 64 GPUs evenly between generation and training, with 32 GPUs for each.

120
00:10:11,987 --> 00:10:20,467
Rollout engines generated trajectories using SGLang, while the trainer updated the RL policy using the NVIDIA Megatron-LM framework.

121
00:10:20,523 --> 00:10:24,443
Evan: How long was each trajectory in this setup?

122
00:10:24,537 --> 00:10:31,397
Ashley: Each trajectory was a multi-turn interaction comprising up to 30 turns or one hour of wall-clock time.

123
00:10:31,397 --> 00:10:40,087
The maximum sequence length per session was 65,536 tokens, budgeting for the agent’s entire multi-turn session.

124
00:10:40,131 --> 00:10:41,911
Evan: And the evaluation?

125
00:10:41,911 --> 00:10:44,651
How did that integrate into the experiment?

126
00:10:44,715 --> 00:10:50,875
Ashley: The run paused generation to evaluate on a disjoint held-out set of tasks every ten steps.

127
00:10:50,875 --> 00:10:57,415
Evaluation used the same set of GPUs, pausing new generations while assessing current performance.

128
00:10:57,459 --> 00:10:58,519
Evan: Got it.

129
00:10:58,519 --> 00:11:02,339
So what were the key results from this experiment?

130
00:11:02,403 --> 00:11:04,533
Ashley: Several key metrics were tracked.

131
00:11:04,533 --> 00:11:13,643
First, the median step time for training was recorded at 263 seconds over the first 30 measured steps, excluding a warm-up period.

132
00:11:13,797 --> 00:11:19,027
Evan: 263 seconds per step is quite efficient given the scale.

133
00:11:19,027 --> 00:11:23,447
What about the divergence between rollout and trainer log-probabilities?

134
00:11:23,499 --> 00:11:33,789
Ashley: The average divergence between the rollout engine's and trainer’s log-probabilities for sampled tokens was measured to be 0.0369 over the 30 steps.

135
00:11:33,789 --> 00:11:39,899
This indicates close alignment, thanks to the measures implemented for token fidelity and routing replay.

136
00:11:40,023 --> 00:11:43,943
Evan: How did the model performance evolve over the training steps?

137
00:11:43,995 --> 00:11:50,505
Ashley: Model performance, indicated by the raw task reward, improved throughout the 100-step experiment.

138
00:11:50,505 --> 00:11:57,675
The nine-step moving average of raw task reward increased from 0.438 to 0.556.

139
00:11:57,723 --> 00:12:03,803
Evan: We usually see improvements in rewards, but how significant are these results?

140
00:12:03,867 --> 00:12:15,827
Ashley: While the rise in the average raw task reward suggests improved performance, it's important to note that a single run on a specific task distribution cannot conclusively determine overall effectiveness.

141
00:12:15,827 --> 00:12:19,367
Variability between runs can influence these numbers.

142
00:12:19,479 --> 00:12:20,599
Evan: Understood.

143
00:12:20,599 --> 00:12:24,579
Were there any other significant observations or metrics reported?

144
00:12:24,627 --> 00:12:29,517
Ashley: Yes, another crucial metric was the overlap between generation and training.

145
00:12:29,517 --> 00:12:41,167
Throughout the experiment, the number of active requests generating trajectories remained close to the limit of 128, thanks to efficiently managing the rollout capacity and cache locality.

146
00:12:41,271 --> 00:12:46,891
Evan: It seems like the architecture was quite effective in maintaining high hardware utilization.

147
00:12:46,891 --> 00:12:51,051
Did they mention any limitations or areas for future improvement?

148
00:12:51,099 --> 00:13:00,979
Ashley: While the paper doesn't delve deeply into limitations in this section, it acknowledges that ongoing work involves tuning the setup for even better performance and consistency.

149
00:13:00,979 --> 00:13:05,039
Real-world applications may reveal further areas for enhancement.

150
00:13:05,091 --> 00:13:08,331
Evan: That's often the case with large-scale ML systems.

151
00:13:08,331 --> 00:13:14,771
It sounds like Miles v0.1 made significant strides in efficient and scalable post-training.

152
00:13:14,835 --> 00:13:15,925
Ashley: Indeed.

153
00:13:15,925 --> 00:13:19,015
And that brings us to the end of the Experiment section.

154
00:13:20,261 --> 00:13:26,141
Evan: Next, let's delve into the Related Work section of the Miles v0.1 paper.

155
00:13:26,189 --> 00:13:38,789
Ashley: The Related Work section places Miles v0.1 in the context of existing research and systems that address similar challenges in reinforcement learning and post-training for large language models.

156
00:13:38,837 --> 00:13:44,917
Evan: Which systems or frameworks does the paper specifically compare Miles to?

157
00:13:44,981 --> 00:13:53,881
Ashley: The paper builds on the clean design principles of an earlier system called Slime, which was a framework aimed at scaling RL for large language models.

158
00:13:53,881 --> 00:14:01,061
Slime focused on the modularity and verification of components, a philosophy that Miles adheres to and extends.

159
00:14:01,109 --> 00:14:03,999
Evan: So, Slime provided a foundation.

160
00:14:03,999 --> 00:14:06,809
What about other prominent systems in this space?

161
00:14:06,869 --> 00:14:07,889
Ashley: Exactly.

162
00:14:07,889 --> 00:14:14,139
Another comparison is with SGLang, which efficiently executes structured language model programs.

163
00:14:14,139 --> 00:14:20,869
SGLang is central to Miles' rollout engines, ensuring high throughput and fidelity for multi-turn rollouts.

164
00:14:21,013 --> 00:14:23,243
Evan: SGLang seems quite integral.

165
00:14:23,243 --> 00:14:24,983
What about training frameworks?

166
00:14:24,983 --> 00:14:26,553
Any comparisons there?

167
00:14:26,597 --> 00:14:28,087
Ashley: Yes, definitely.

168
00:14:28,087 --> 00:14:34,277
Miles employs NVIDIA Megatron-LM and PyTorch FSDP as its training backends.

169
00:14:34,277 --> 00:14:41,767
Megatron-LM is notable for its extensive parallelism options, which are key for scaling models across many GPUs.

170
00:14:41,767 --> 00:14:47,797
PyTorch FSDP, on the other hand, offers simplicity and direct Hugging Face model loading.

171
00:14:47,921 --> 00:14:54,061
Evan: Both Megatron-LM and PyTorch FSDP are well-regarded in the community.

172
00:14:54,061 --> 00:14:58,101
How does Miles differentiate itself further in the context of RL?

173
00:14:58,157 --> 00:15:08,267
Ashley: Miles extends beyond typical RL loops by incorporating elements like Low-Rank Adaptation (LoRA), on-policy distillation, and true-on-policy alignment.

174
00:15:08,267 --> 00:15:16,157
These elements are compared to similar adaptations in other frameworks, which often don't provide the same level of integration and flexibility.

175
00:15:16,255 --> 00:15:17,445
Evan: Interesting.

176
00:15:17,445 --> 00:15:22,365
Does the paper mention any particular advantages over existing systems?

177
00:15:22,421 --> 00:15:23,771
Ashley: Yes, it does.

178
00:15:23,771 --> 00:15:32,441
One standout aspect is the ability to handle fully asynchronous training and evaluation, which isn't as robustly supported by other frameworks.

179
00:15:32,441 --> 00:15:40,561
This capability allows for efficient, scalable RL training that minimizes idle times and better utilizes hardware resources.

180
00:15:40,613 --> 00:15:44,063
Evan: So, asynchronous training is a big deal here.

181
00:15:44,063 --> 00:15:49,173
How does Miles ensure compatibility and maintain support for new models and hardware?

182
00:15:49,229 --> 00:15:54,189
Ashley: Miles emphasizes verified coverage and the principle of easy extensibility.

183
00:15:54,189 --> 00:16:03,619
It incorporates small, typed interfaces at key points, ensuring that user-specific logic and new models can be integrated without overhauling the core system.

184
00:16:03,619 --> 00:16:06,729
This modularity is a significant advantage.

185
00:16:06,773 --> 00:16:09,393
Evan: That extensibility sounds practical.

186
00:16:09,393 --> 00:16:13,633
How does Miles handle the intricacies of multi-vendor GPU support?

187
00:16:13,715 --> 00:16:14,725
Ashley: Good point.

188
00:16:14,725 --> 00:16:26,665
Miles runs natively on both NVIDIA and AMD hardware, sidestepping the need for translation layers, and ensures that precision formats and model scalability are compatible across these platforms.

189
00:16:26,717 --> 00:16:28,737
Evan: It seems comprehensive.

190
00:16:28,737 --> 00:16:33,597
What about the numerical fidelity and efficient weight updates you mentioned earlier?

191
00:16:33,653 --> 00:16:42,723
Ashley: Miles' approach to numerical fidelity includes robust tokenization and tight integration between rollout and training stages to prevent discrepancies.

192
00:16:42,723 --> 00:16:52,933
As for weight updates, it offers flexible transport mechanisms such as NCCL broadcast, RDMA, and delta updates to cater to various deployment scenarios.

193
00:16:52,997 --> 00:16:58,497
Evan: So, it really covers a lot of ground in terms of ensuring accuracy and efficiency.

194
00:16:58,497 --> 00:17:03,817
Are there any key references or predecessors mentioned in the paper that influenced its design?

195
00:17:03,869 --> 00:17:04,909
Ashley: Definitely.

196
00:17:04,909 --> 00:17:20,459
The paper cites several foundational works, including the original Megatron-LM paper for model parallelism, NVIDIA's PyTorch FSDP for sharded data parallelism, and the introduction of Low-Rank Adaptations for fine-tuning large models efficiently.

197
00:17:20,459 --> 00:17:25,229
Each of these has inspired and informed the design of Miles in meaningful ways.

198
00:17:25,277 --> 00:17:29,237
Evan: It's clear how these references contributed to shaping Miles.

199
00:17:29,237 --> 00:17:31,817
Anything else noteworthy about the related work?

200
00:17:31,877 --> 00:17:41,117
Ashley: The Related Work section also emphasizes the novelty of combining multiple advanced techniques into a single, cohesive system, which is something that stands out in Miles.

201
00:17:41,117 --> 00:17:49,577
It brings together the best practices and methodologies into a unified framework that addresses the challenges of frontier-scale post-training comprehensively.

202
00:17:49,637 --> 00:17:52,087
Evan: That's a significant achievement.

203
00:17:52,087 --> 00:17:57,697
It seems like Miles has synthesized a wide array of best practices into its design.

204
00:17:57,749 --> 00:17:58,959
Ashley: Indeed.

205
00:17:58,959 --> 00:18:02,569
And that brings us to the end of the Related Work section.

206
00:18:03,822 --> 00:18:10,422
Evan: Now, let's summarize the key contributions and takeaways from the Miles v0.1 paper.

207
00:18:10,470 --> 00:18:18,970
Ashley: First, Miles v0.1 introduces a full-stack, production-ready system for frontier post-training of large language models.

208
00:18:18,970 --> 00:18:27,730
It capitalizes on the foundation established by prior frameworks like Slime, emphasizing components that are verified, clean, and customizable.

209
00:18:27,774 --> 00:18:42,074
Evan: Miles supports various RL stages with different deployment topologies by utilizing rollout engines built on SGLang and offering a choice between NVIDIA Megatron-LM and PyTorch FSDP as training backends.

210
00:18:42,126 --> 00:18:59,846
Ashley: Moreover, Miles addresses key challenges in maintaining high hardware utilization and reducing numerical discrepancies between rollout and training stages through token fidelity and specialized routing mechanisms like the Token-In-Token-Out session server and Efficient Rollout Routing Replay.

211
00:18:59,910 --> 00:19:12,760
Evan: The system also extends beyond core RL to support Low-Rank Adaptation, on-policy distillation, supervised fine-tuning, and true-on-policy alignment, as well as diffusion models.

212
00:19:12,760 --> 00:19:18,450
This flexibility makes Miles capable of handling a variety of post-training goals effectively.

213
00:19:18,510 --> 00:19:31,090
Ashley: Experimentally, Miles demonstrated its capabilities in a large-scale case study involving asynchronous agentic RL on a GLM-5.2 model with significant computational resources.

214
00:19:31,090 --> 00:19:34,530
This setup showcased its efficiency and robustness.

215
00:19:34,650 --> 00:19:50,490
Evan: Through cross-references with related works, Miles is positioned as a comprehensive solution that integrates multiple advanced techniques, offering substantial improvements in scalability, efficiency, and flexibility for reinforcement learning at frontier scales.

216
00:19:50,550 --> 00:20:03,890
Ashley: Indeed, Miles represents a significant step forward in post-training methodologies for large language models, promising to make frontier-scale RL more accessible and effective for both researchers and enterprises.

217
00:20:04,002 --> 00:20:07,272
Evan: That's it for today's episode of Daily Paper Cast.

218
00:20:07,272 --> 00:20:10,862
We hope you found our discussion as insightful as we did.

219
00:20:10,926 --> 00:20:20,706
Ashley: Be sure to join us again for future episodes, where we'll continue to bring you the latest and most exciting developments in AI, NLP, and computer vision research.

220
00:20:20,766 --> 00:20:24,546
Evan: Thanks for listening and see you next time!