Miles v0.1: Production-Level Post-Training

🤗 Upvotes: 29 | cs.LG, cs.CL

Authors:
RadixArk, Tom Chen, Mao Cheng, Shi Dong, Kangrui Du, Yanbin Jiang, Jiajun Li, Yiming Li, Tao Lin, Yusheng Su, Andy Ye, Yueming Yuan, Zhichen Zeng

Title:
Miles v0.1: Production-Level Post-Training

Arxiv:
http://arxiv.org/abs/2609.08368v1

Abstract:
We present Miles v0.1, a full-stack, production-ready system for frontier post-training. Building upon the clean design of slime, Miles designs each stage of the reinforcement-learning (RL) training loop around a single principle: components should be verified, clean, and customizable. With accuracy, efficiency, reliability, and scalability as first-class goals, Miles aims to make frontier-scale RL accessible to researchers and enterprises alike. This report walks through the system end to end: rollout engines built on SGLang, a trainer with a choice of two backends (NVIDIA Megatron-LM and PyTorch FSDP), and three weight-synchronization transports for different deployment topologies. Beyond full-parameter RL, Miles also supports LoRA RL, on-policy distillation, supervised fine-tuning, and true-on-policy rollout-training alignment, and extends the same architecture to diffusion models. We close with an end-to-end case study: fully asynchronous agentic RL on a GLM-5.2 744B-A40B model over terminal-use coding tasks, running on 64 NVIDIA GB300 GPUs with a median step time of 263 seconds over the first 30 measured steps. Miles is open-sourced at https://github.com/radixark/miles, with the project website at https://miles.radixark.com.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast.

Ashley: Today's paper is from the Hugging Face daily paper list of September 9, 2026, with 29 upvotes.

Evan: The paper is titled 'Miles v0.1: Production-Level Post-Training'.

Ashley: The first two authors are Tom Chen and Mao Cheng.

The corresponding author is Shi Dong from RadixArk.

Evan: Let's dive into the Introduction section.

Ashley: Post-training turns a pretrained language model into a useful one.

At frontier scale, post-training poses substantial challenges for training systems.

Evan: What makes post-training at frontier scale particularly challenging?

Ashley: For example, reinforcement learning, or RL, for large language models no longer follows a simple generate-then-update loop over short completions.

Rollouts span multiple turns, use tools, and let the model act in an external environment.

The models producing the rollouts are often trillion-parameter mixtures of experts.

Evan: That sounds complex.

What are some specific challenges these systems face?

Ashley: Sustaining high end-to-end hardware utilization is difficult.

The system must juggle latency-sensitive rollouts with throughput-oriented training, introducing bubbles and idle time.

Additionally, the numerical gap between the rollout engines and the trainer can become large enough to invalidate the objective outright.

Evan: Interesting.

So, how does Miles aim to tackle these problems?

Ashley: Miles v0.1 is a full-stack, production-ready system for frontier post-training.

It builds on the clean design of slime and centers on one principle: components should be verified, clean, and customizable.

Evan: Could you elaborate on the core design principles of Miles?

Ashley: Of course.

The core principles of Miles are accuracy, efficiency, reliability, and scalability.

The system aims to make frontier-scale RL accessible to researchers and enterprises alike.

Evan: And what are the key components of Miles v0.1?

Ashley: Miles encompasses several key components: rollout engines built on SGLang, a trainer with a choice of two backends—NVIDIA Megatron-LM and PyTorch FSDP—and three weight-synchronization transports for different deployment topologies.

Evan: That sounds comprehensive.

Does Miles support other types of training beyond full-parameter RL?

Ashley: Yes, it does.

Beyond full-parameter RL, Miles also supports Low-Rank Adaptation or LoRA RL, on-policy distillation, supervised fine-tuning, and true-on-policy rollout-training alignment.

It even extends the same architecture to diffusion models.

Evan: Can you provide a specific example demonstrating Miles' capabilities?

Ashley: Certainly.

The paper closes with an end-to-end case study: a fully asynchronous agentic RL on a GLM-5.2 744B-A40B model over terminal-use coding tasks, running on 64 NVIDIA GB300 GPUs with a median step time of 263 seconds over the first 30 measured steps.

Evan: That's impressive.

It sounds like Miles v0.1 is a robust system designed to handle the complexities of frontier-scale RL.

Ashley: Indeed.

And that wraps up the Introduction section of today's paper.

Evan: Let's move on to the method section and delve into how Miles v0.1 operates in detail.

Ashley: We'll start with the core RL loop that Miles employs.

A reinforcement learning training job in Miles cycles through three key stages: rollout, training, and weight update.

Evan: Can you break down each of these stages for us?

Ashley: Sure.

First, we have the Rollout stage.

SGLang engines generate trajectories.

In agentic RL, where the model acts across multiple turns, each rollout session interacts with its own isolated environment, which executes actions and provides rewards.

Evan: So, SGLang is crucial for generating these multi-turn rollouts.

What happens after that?

Ashley: Next, we move to the Training stage.

Here, the trainer consumes completed trajectory groups, computes the RL loss, and updates the policy.

Miles supports two backends for this task: NVIDIA Megatron-LM and PyTorch FSDP.

Evan: And finally, we have the weight update stage.

How does that work?

Ashley: After each training step, Miles updates the RL policy by synchronizing the new weights with the rollout engines.

The goal is to minimize interruptions to in-flight rollouts, ensuring seamless progress.

Evan: I've heard that synchronization can be a major bottleneck.

How does Miles handle this?

Ashley: Good question.

To mitigate this, Miles offers different weight-synchronization transports based on the deployment topology: NCCL broadcast, peer-to-peer RDMA, or CUDA IPC when colocated.

Evan: That's quite a variety.

But I've also heard about fully asynchronous RL in Miles.

How does that work?

Ashley: Fully asynchronous RL allows rollout generation and training to progress concurrently.

The rollout engines generate trajectories continuously while the trainer consumes whichever trajectories are ready to keep both stages busy simultaneously.

Evan: So, they don’t wait for each other.

That sounds efficient.

What about throughput and fidelity during rollout generation?

Ashley: Indeed, throughput and fidelity are crucial.

Miles uses SGLang for request routing to preserve cache locality and decouple generation from training.

This ensures that the trainer's view of trajectories remains consistent with what the policy actually sampled.

Evan: Consistency must be tricky, right?

How does Miles ensure token fidelity in multi-turn scenarios?

Ashley: Miles employs a Token-In-Token-Out, or TITO, session server to maintain the exact tokens sampled by the policy.

The server controls tokenization, preserving rollout log-probabilities, and ensures that training sequences match the generated tokens exactly.

Evan: Can you explain how Miles maintains the balance between different components during training?

Ashley: Miles keeps the rollout engines busy by replenishing rollout generation capacity as trajectories finish.

A background worker maintains a consistent number of trajectories in generation to minimize trainer wait time.

The data buffer also decouples generation and training, allowing each to operate at its own pace.

Evan: That sounds well-coordinated.

What about handling longer rollout contexts?

Ashley: To accommodate longer contexts, Miles uses session-aware routing to maintain affinity for multi-turn episodes, ensuring that data-parallel ranks leverage cached prefix tokens efficiently.

Additionally, there’s a fallback in the form of session server to track session identity and supply routing keys.

Evan: With all these components, how does Miles manage asynchronous evaluation?

Ashley: Asynchronous evaluation runs alongside training.

Miles offers three evaluation modes: shared engines, a dedicated GPU fleet, or an external service.

In shared mode, rollout engines pause new generation while evaluating, while the other modes keep evaluation off the critical training path, using snapshots to run evaluations without hindering training.

Evan: How does it ensure that the evaluations are accurate and reflective of the policy versions?

Ashley: Each evaluation score is tied to the specific policy version that produced it.

During evaluation, weights are verified, ensuring consistency across the fleet.

Even when an evaluation fails, training continues, with reasons logged appropriately.

Evan: What types of environments can Miles handle for agentic RL?

Ashley: Miles is versatile in accommodating various environments by organizing integration as three nested plug-in layers.

It supports environments that provide different scopes of control, ranging from simple episode loops to complex batching and rewards management.

Evan: That’s comprehensive.

What else does the method section cover?

Ashley: It touches on RL router handling for mixture-of-experts models, ensuring expert routing fidelity with R3, termed as Efficient Rollout Routing Replay.

For mixture-of-experts models, the router holds routing tensors to replay expert assignments, which is vital for maintaining training stability.

Evan: You mentioned memory efficiency earlier.

Can you elaborate on that?

Ashley: Certainly.

Miles employs memory-efficient techniques like evicting paused actors and streaming optimizer state to reduce GPU memory consumption.

Offloading optimizer states to CPU or NVMe further ensures that the training actor fits within the memory limits during active computation stages.

Evan: What about low-precision training?

Does Miles support that?

Ashley: Yes, lower-precision formats like FP8 and NVFP4 are supported, improving computational efficiency.

Miles ensures that both rollout and training stages use the same quantization logic to avoid discrepancies.

Evan: Fascinating.

This method section is indeed thorough in covering the operational aspects of Miles v0.1.

Ashley: Indeed.

And that brings us to the end of the Method section.

Evan: Moving on, let's discuss the experiments and results presented in the Miles v0.1 paper.

Ashley: Sure.

The paper presents a case study where Miles v0.1 was used for fully asynchronous agentic RL on the GLM-5.2 744B-A40B model over terminal-use coding tasks.

This was run on 64 NVIDIA GB300 GPUs.

Evan: 64 GPUs?

That's impressive.

What did the experimental setup look like?

Ashley: The setup divided 64 GPUs evenly between generation and training, with 32 GPUs for each.

Rollout engines generated trajectories using SGLang, while the trainer updated the RL policy using the NVIDIA Megatron-LM framework.

Evan: How long was each trajectory in this setup?

Ashley: Each trajectory was a multi-turn interaction comprising up to 30 turns or one hour of wall-clock time.

The maximum sequence length per session was 65,536 tokens, budgeting for the agent’s entire multi-turn session.

Evan: And the evaluation?

How did that integrate into the experiment?

Ashley: The run paused generation to evaluate on a disjoint held-out set of tasks every ten steps.

Evaluation used the same set of GPUs, pausing new generations while assessing current performance.

Evan: Got it.

So what were the key results from this experiment?

Ashley: Several key metrics were tracked.

First, the median step time for training was recorded at 263 seconds over the first 30 measured steps, excluding a warm-up period.

Evan: 263 seconds per step is quite efficient given the scale.

What about the divergence between rollout and trainer log-probabilities?

Ashley: The average divergence between the rollout engine's and trainer’s log-probabilities for sampled tokens was measured to be 0.0369 over the 30 steps.

This indicates close alignment, thanks to the measures implemented for token fidelity and routing replay.

Evan: How did the model performance evolve over the training steps?

Ashley: Model performance, indicated by the raw task reward, improved throughout the 100-step experiment.

The nine-step moving average of raw task reward increased from 0.438 to 0.556.

Evan: We usually see improvements in rewards, but how significant are these results?

Ashley: While the rise in the average raw task reward suggests improved performance, it's important to note that a single run on a specific task distribution cannot conclusively determine overall effectiveness.

Variability between runs can influence these numbers.

Evan: Understood.

Were there any other significant observations or metrics reported?

Ashley: Yes, another crucial metric was the overlap between generation and training.

Throughout the experiment, the number of active requests generating trajectories remained close to the limit of 128, thanks to efficiently managing the rollout capacity and cache locality.

Evan: It seems like the architecture was quite effective in maintaining high hardware utilization.

Did they mention any limitations or areas for future improvement?

Ashley: While the paper doesn't delve deeply into limitations in this section, it acknowledges that ongoing work involves tuning the setup for even better performance and consistency.

Real-world applications may reveal further areas for enhancement.

Evan: That's often the case with large-scale ML systems.

It sounds like Miles v0.1 made significant strides in efficient and scalable post-training.

Ashley: Indeed.

And that brings us to the end of the Experiment section.

Evan: Next, let's delve into the Related Work section of the Miles v0.1 paper.

Ashley: The Related Work section places Miles v0.1 in the context of existing research and systems that address similar challenges in reinforcement learning and post-training for large language models.

Evan: Which systems or frameworks does the paper specifically compare Miles to?

Ashley: The paper builds on the clean design principles of an earlier system called Slime, which was a framework aimed at scaling RL for large language models.

Slime focused on the modularity and verification of components, a philosophy that Miles adheres to and extends.

Evan: So, Slime provided a foundation.

What about other prominent systems in this space?

Ashley: Exactly.

Another comparison is with SGLang, which efficiently executes structured language model programs.

SGLang is central to Miles' rollout engines, ensuring high throughput and fidelity for multi-turn rollouts.

Evan: SGLang seems quite integral.

What about training frameworks?

Any comparisons there?

Ashley: Yes, definitely.

Miles employs NVIDIA Megatron-LM and PyTorch FSDP as its training backends.

Megatron-LM is notable for its extensive parallelism options, which are key for scaling models across many GPUs.

PyTorch FSDP, on the other hand, offers simplicity and direct Hugging Face model loading.

Evan: Both Megatron-LM and PyTorch FSDP are well-regarded in the community.

How does Miles differentiate itself further in the context of RL?

Ashley: Miles extends beyond typical RL loops by incorporating elements like Low-Rank Adaptation (LoRA), on-policy distillation, and true-on-policy alignment.

These elements are compared to similar adaptations in other frameworks, which often don't provide the same level of integration and flexibility.

Evan: Interesting.

Does the paper mention any particular advantages over existing systems?

Ashley: Yes, it does.

One standout aspect is the ability to handle fully asynchronous training and evaluation, which isn't as robustly supported by other frameworks.

This capability allows for efficient, scalable RL training that minimizes idle times and better utilizes hardware resources.

Evan: So, asynchronous training is a big deal here.

How does Miles ensure compatibility and maintain support for new models and hardware?

Ashley: Miles emphasizes verified coverage and the principle of easy extensibility.

It incorporates small, typed interfaces at key points, ensuring that user-specific logic and new models can be integrated without overhauling the core system.

This modularity is a significant advantage.

Evan: That extensibility sounds practical.

How does Miles handle the intricacies of multi-vendor GPU support?

Ashley: Good point.

Miles runs natively on both NVIDIA and AMD hardware, sidestepping the need for translation layers, and ensures that precision formats and model scalability are compatible across these platforms.

Evan: It seems comprehensive.

What about the numerical fidelity and efficient weight updates you mentioned earlier?

Ashley: Miles' approach to numerical fidelity includes robust tokenization and tight integration between rollout and training stages to prevent discrepancies.

As for weight updates, it offers flexible transport mechanisms such as NCCL broadcast, RDMA, and delta updates to cater to various deployment scenarios.

Evan: So, it really covers a lot of ground in terms of ensuring accuracy and efficiency.

Are there any key references or predecessors mentioned in the paper that influenced its design?

Ashley: Definitely.

The paper cites several foundational works, including the original Megatron-LM paper for model parallelism, NVIDIA's PyTorch FSDP for sharded data parallelism, and the introduction of Low-Rank Adaptations for fine-tuning large models efficiently.

Each of these has inspired and informed the design of Miles in meaningful ways.

Evan: It's clear how these references contributed to shaping Miles.

Anything else noteworthy about the related work?

Ashley: The Related Work section also emphasizes the novelty of combining multiple advanced techniques into a single, cohesive system, which is something that stands out in Miles.

It brings together the best practices and methodologies into a unified framework that addresses the challenges of frontier-scale post-training comprehensively.

Evan: That's a significant achievement.

It seems like Miles has synthesized a wide array of best practices into its design.

Ashley: Indeed.

And that brings us to the end of the Related Work section.

Evan: Now, let's summarize the key contributions and takeaways from the Miles v0.1 paper.

Ashley: First, Miles v0.1 introduces a full-stack, production-ready system for frontier post-training of large language models.

It capitalizes on the foundation established by prior frameworks like Slime, emphasizing components that are verified, clean, and customizable.

Evan: Miles supports various RL stages with different deployment topologies by utilizing rollout engines built on SGLang and offering a choice between NVIDIA Megatron-LM and PyTorch FSDP as training backends.

Ashley: Moreover, Miles addresses key challenges in maintaining high hardware utilization and reducing numerical discrepancies between rollout and training stages through token fidelity and specialized routing mechanisms like the Token-In-Token-Out session server and Efficient Rollout Routing Replay.

Evan: The system also extends beyond core RL to support Low-Rank Adaptation, on-policy distillation, supervised fine-tuning, and true-on-policy alignment, as well as diffusion models.

This flexibility makes Miles capable of handling a variety of post-training goals effectively.

Ashley: Experimentally, Miles demonstrated its capabilities in a large-scale case study involving asynchronous agentic RL on a GLM-5.2 model with significant computational resources.

This setup showcased its efficiency and robustness.

Evan: Through cross-references with related works, Miles is positioned as a comprehensive solution that integrates multiple advanced techniques, offering substantial improvements in scalability, efficiency, and flexibility for reinforcement learning at frontier scales.

Ashley: Indeed, Miles represents a significant step forward in post-training methodologies for large language models, promising to make frontier-scale RL more accessible and effective for both researchers and enterprises.

Evan: That's it for today's episode of Daily Paper Cast.

We hope you found our discussion as insightful as we did.

Ashley: Be sure to join us again for future episodes, where we'll continue to bring you the latest and most exciting developments in AI, NLP, and computer vision research.

Evan: Thanks for listening and see you next time!