Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning

🤗 Upvotes: 31 | cs.AI

Authors:
Kai Chen, Jifeng Ding, Ning Ding, Jiaye Ge, Lixin Gu, Yicheng Gu, Qipeng Guo, Ermo Hua, Haian Huang, Haozheng Hou, Jie Hou, Xiangyu Hong, Che Jiang, Minxi Jin, Cheng Liang, Dahua Lin, Dawei Liu, Kuikun Liu, Chengqi Lv, Haijun Lv, Han Lv, Ningsheng Ma, Biqing Qi, Jianmin Qian, Shiya Su, Youbang Sun, Huanze Tang, Zhongbo Tian, Hanjing Wang, Rui Wang, Ting Wang, Yi Wang, Baiting Wu, Jun Xu, Bowen Yang, Hui Wang, Weida Wang, Haochen Ye, Jiashuo Yu, Shan Yu, Xiaoyi Yu, Qirui Zeng, Qi Zhang, Ming Zhang, Wenwei Zhang, Bowen Zhou, Xinyu Zhou

Title:
Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning

Arxiv:
http://arxiv.org/abs/2608.14290v1

Abstract:
We introduce Mobius-v0, an architecture that comprises a globally shared Memory (FFN) that stores knowledge vectors and multiple Reasoners (Self-Attn) that iteratively achieve compositional reasoning. Using hidden states as cache and carrier, reasoners repeatedly query memory for required knowledge-vectors, while the knowledge is transmitted back to reasoning operators. Through this knowledge-reasoning-separation architecture, Mobius achieves better knowledge compression and reasoning efficiency. Built upon Mobius-v0 architecture: 1) Our 7B model trained-from-scratch achieves similar downstream score as a 7B Transformer baseline with 62.6% of baseline's training data. 2) Our Intern-S2-Mobius, continually-pretrained from Qwen3.5-35B, achieves similar downstream score while delivering nearly 4x end-to-end inference speedup.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast, everyone.

Ashley: Today, we're diving into a fascinating paper from the Hugging Face daily paper list dated August 18, 2026.

This paper has garnered 31 upvotes and is quite a hot topic.

Evan: The title of the paper is 'Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning.'

Ashley: The first two authors are Ermo Hua and Xiangyu Hong, with the corresponding author Dahua Lin from the Shanghai AI Laboratory.

Evan: Alright, Ashley, let's get into the core of this paper.

Start us off with the background and the motivation behind this work.

Ashley: Sure, Evan.

The paper addresses a key challenge in the development of foundation models, largely focusing on how these models handle knowledge and reasoning.

Traditionally, models like Transformers, which are heavily used in various domains including language, vision, video, and scientific computing, face two main developmental bottlenecks.

Evan: And what are these bottlenecks exactly?

Ashley: The first issue revolves around scalability.

Essentially, enhancing model capabilities usually involves scaling up parameters, increasing training datasets, and extending reasoning chains.

Although this enhances the model's performance, it hits a plateau due to diminishing returns — more resources don't necessarily mean leaps in effectiveness.

Evan: That sounds familiar.

And the second bottleneck?

Ashley: The second bottleneck is tied to efficiency.

As models increase in complexity, the computational demand for training and inference also skyrockets, making large-scale deployment impractical.

Despite advancements like reducing the complexity of self-attention mechanisms, the foundational architecture still poses a significant inefficiency challenge.

Evan: So, what does this paper propose to tackle these issues?

Ashley: The authors introduce 'Mobius,' an architecture designed to decouple knowledge storage from the reasoning process.

To break this down: in this structure, knowledge vectors are stored globally in a shared Memory component, whereas reasoning is handled by Self-Attention modules called Reasoners.

This separation allows for better knowledge compression and more efficient reasoning.

Evan: That's intriguing.

Does this mean Mobius operates entirely differently from traditional Transformers?

Ashley: Yes, indeed.

Transformers handle both knowledge and reasoning within coupled modules, making every layer's processing interdependent.

Mobius, on the other hand, uses Reasoners to query the Memory for required knowledge vectors actively.

This not only conserves resources but also speeds up the entire reasoning process quite significantly.

Evan: How does this new approach impact the model's performance and efficiency?

Ashley: Great question.

The paper presents some remarkable results.

Their 7-billion parameter model, Mobius-v0, achieved similar downstream scores as a comparable 7-billion parameter Transformer model, but with only 62.6% of the training data required.

More impressively, their continually pre-trained Intern-S2-Mobius model not only matched but significantly improved inference speeds by almost four times compared to the baseline Transformer.

Evan: That’s quite a leap forward.

So, how does Mobius achieve this efficiency?

Ashley: The efficiency gains primarily come from two mechanisms.

First, Mobius introduces a more flexible activation path during inference, which optimally uses both shallow and deep hidden states thanks to its backward residual connections.

This allows higher layers to access lower-layer knowledge and vice versa.

Secondly, Mobius employs dynamic latent reasoning.

Traditional Transformers need to traverse all layers for each token during inference, but Mobius can refine its latent states through fewer layers by interacting dynamically with the entire knowledge repository.

Evan: Fascinating.

Dynamic latent reasoning sounds sophisticated.

How does it work in practice?

Ashley: In practice, Mobius maps complex reasoning processes into continuous vector space rather than processing long chains of discrete tokens step-by-step.

This reduces token redundancy and allows more nuanced and condensed information processing.

Thus, instead of generating verbose outputs for complex problems, Mobius provides efficient and dense reasoning outputs, which obviously leads to much faster processing times.

Evan: This sounds like a promising shift in AI architectures, particularly for large-scale systems.

Ashley: It really is.

By decoupling knowledge storage and reasoning, Mobius efficiently leverages computational resources, potentially setting new standards for AI model architecture.

Evan: That wraps up the introduction to this groundbreaking model.

Next, we'll dive into the technical details and see how the architecture specifically implements these concepts.

Stay with us.

Evan: Welcome back.

Now that we have a good understanding of the motivation and objectives behind the Mobius architecture, let's dive deeper into the methodology.

Ashley, could you walk us through the core components and mechanisms?

Ashley: Of course, Evan.

The core of the Mobius architecture is its innovative decoupling of knowledge storage and reasoning, which is achieved through two primary components: the Memory and the Reasoners.

Let’s break it down.

Evan: Great, let’s start with the Memory component.

What is it and how does it function?

Ashley: The Memory component is essentially a globally shared database of knowledge vectors.

Unlike traditional Transformer models where the Feed-Forward Network, or FFN, handles both knowledge storage and computation, Mobius separates these roles.

In Mobius, the FFN is dedicated solely to storing knowledge vectors in a highly efficient, compressed form.

Evan: And how does the reasoning part work then?

What exactly are the Reasoners tasked with?

Ashley: The Reasoners, built using self-attention mechanisms, are responsible for the iterative process of compositional reasoning.

They query the Memory for necessary knowledge vectors, use these vectors to perform computations, and then transmit the results back into the Memory.

This iterative querying and updating enable dynamic and efficient reasoning across multiple layers.

Evan: So, essentially, the Reasoners and Memory are continuously interacting?

Ashley: Exactly.

Each Reasoner functions somewhat like a specialized worker that repeatedly queries a central knowledge repository as it processes data, allowing it to efficiently synthesize and refine information.

This back-and-forth interaction between Reasoners and Memory enhances the model’s overall reasoning efficiency.

Evan: Interesting.

Now, the paper mentions something called 'Backward Residual Connection.' Can you explain how that fits into this methodology?

Ashley: Certainly.

Traditional transformers largely rely on forward residual connections, where processed information is relayed from shallow to deep layers.

Mobius introduces backward residual connections, which allow information to flow not just from lower layers to higher layers but also in the opposite direction.

This bidirectional flow is enabled by the shared Memory component.

Evan: So, how does this backward mechanism benefit the model?

Ashley: This mechanism allows lower-layer Reasoners to access high-layer knowledge and vice versa.

It enhances compositional generalization across different layers, speeding up the synthesis of critical information because shallower processes can directly utilize deeper insights without waiting for multiple processing cycles.

Evan: It seems like a huge advantage for efficiency.

What about the Dynamic Latent Reasoning — how does that work in Mobius?

Ashley: Dynamic Latent Reasoning in Mobius is a game-changer.

Instead of processing each token in isolation through the entire network, Mobius iteratively refines a continuous latent vector space.

This means the model can dynamically allocate computation efforts to different tokens based on their requirements, away from the rigid sequential token generation process.

Evan: So, in practice, how does Mobius handle this refinement process?

Ashley: In practice, Mobius repeatedly refines these continuous latent vectors by querying the Memory and performing necessary computations within fewer layers.

It uses fewer high-information-density tokens, reducing the number of passes required compared to traditional transformers, which improves both speed and efficiency.

Evan: Got it.

What about the training process?

How does Mobius approach this?

Ashley: The training process for Mobius includes both training-from-scratch and continual pre-training.

For the training-from-scratch experiments, they trained a 7-billion-parameter model on one terabyte of tokens.

For the continual pre-training, they used the Qwen3.5 as a starting checkpoint and performed further training on another one terabyte of tokens.

Evan: And what were the key findings from these training experiments?

Ashley: From the training-from-scratch experiments, Mobius achieved significantly higher data efficiency.

It achieved the same MMLU scores as a Transformer model but with only 62.6% of the training data.

This suggests a superior compression rate in the Mobius architecture.

Evan: And how about the continual pre-training?

Ashley: For the continual pre-training, Mobius not only preserved the reasoning capabilities of the base model but also enhanced them.

It delivered nearly four times the end-to-end inference speedup compared to the Transformer, thanks to the shorter chains of thought required for each task, which showcases its dynamic latent reasoning.

Evan: So, the models were both faster and more data-efficient.

That’s a solid improvement.

Now, metrics and evaluation methods are crucial.

How did the authors measure the model's performance?

Ashley: The evaluation was quite comprehensive.

They used standardized benchmarks for both general and scientific tasks.

For general tasks, they used benchmarks like MMLU Pro, GPQA Diamond, IMO Bench, AIME 2026, HMMT 2026, and others.

For scientific tasks, they specifically focused on biology and molecular sciences with benchmarks like Biology-Instructions, Mol-Instructions, and MolecularIQ.

Evan: Could you give us a sense of how Mobius performed on these benchmarks?

Ashley: Sure.

Mobius-35B, during continual pre-training, achieved higher or comparable scores to the baseline Qwen3.5-35B across almost all benchmarks.

For instance, in the MMLU Pro benchmark, Mobius scored 89.05 compared to Qwen's 85.31.

In the molecular and biology-specific benchmarks, the improvement was even more significant, showing Mobius's strong adaptation to scientific tasks.

Evan: What about the real-world application of Mobius?

How practical is it for deployment?

Ashley: The practicality of Mobius shines in its inference efficiency.

Given that it requires fewer tokens to achieve the same reasoning, its suitability for real-time applications is vastly improved.

This is particularly valuable in commercial deployments where both speed and computational cost are critical factors.

Evan: It sounds like Mobius offers considerable advantages in both performance and efficiency over traditional Transformer models.

Ashley: Indeed it does.

By decoupling knowledge and reasoning, and introducing mechanisms like backward residual connections and dynamic latent reasoning, Mobius sets a new standard for AI model architecture.

That wraps up our detailed look at the methodology behind this innovative model.

Evan: Stay tuned as we continue to delve deeper into the experiments and results in the next section.

Evan: Welcome back.

Now, let's delve into the experiments and results that validate the Mobius architecture.

Ashley, what do the authors share about their experimental setup?

Ashley: The authors conducted both training-from-scratch (TFS) and continual pre-training (CPT) experiments to evaluate the performance of the Mobius architecture.

For the TFS experiments, they trained a 7-billion-parameter model from scratch on one terabyte of tokens.

For the CPT experiments, they used Qwen3.5-35B as the starting model and continued pre-training it with another terabyte of tokens.

Evan: Interesting.

And what were the key findings from these experiments?

Ashley: From the TFS experiments, Mobius achieved significantly higher data efficiency compared to the Transformer model.

Specifically, Mobius reached the same MMLU scores as the Transformer model but required only 62.6% of the training data.

This suggests that Mobius is much more effective in leveraging training data for knowledge acquisition.

Evan: That's impressive.

How did it fare in terms of reasoning capabilities?

Ashley: In the CPT experiments, Mobius not only preserved the reasoning abilities of the baseline Qwen3.5 but enhanced them.

The model showed considerable improvements in end-to-end inference efficiency, achieving nearly 4 times the throughput of the Transformer model.

This efficiency mainly comes from Mobius's ability to resolve tasks with shorter chains of thought, thanks to its dynamic latent reasoning capabilities.

Evan: Got it.

How was this efficiency measured?

Ashley: The efficiency gains were quantified through benchmark tests.

They evaluated Mobius and the Transformer on multiple tasks and tracked metrics like request throughput and average output length.

In every benchmark – including MMLU Pro, GPQA Diamond, IMO Bench, AIME 2026, and HMMT 2026 – Mobius delivered higher throughput and shorter average output lengths.

Evan: Could you give us a specific example from these benchmarks?

Ashley: Sure.

Take the MMLU Pro benchmark: during this task, Mobius achieved a score of 89.05 while the Qwen3.5-35B model scored 85.31.

Similarly, on the GPQA Diamond benchmark, Mobius outperformed with a score of 80.81 compared to Qwen3.5’s 80.24.

These numbers illustrate Mobius’s superior performance across diverse evaluation criteria.

Evan: It seems that Mobius consistently outperforms the Transformer in various tasks.

What about the scientific benchmarks?

Ashley: On the scientific front, the evaluations were equally impressive.

For instance, in the Biology-Instructions benchmark, Mobius scored 51.40, whereas Qwen3.5 scored just 3.77.

In the Mol-Instructions benchmark, Mobius achieved 45.73 versus Qwen’s 21.70.

The most striking difference was seen in the MolecularIQ benchmark, where Mobius scored 59.29 against Qwen’s 29.13.

These results indicate that Mobius not only excels in general tasks but is also highly efficient in domain-specific scientific challenges.

Evan: These performance metrics are indeed noteworthy.

What about qualitative assessments?

How did Mobius handle the accuracy and brevity of reasoning traces?

Ashley: In qualitative assessments, Mobius consistently produced more concise and accurate reasoning traces.

For example, a step-aligned comparison on a linear algebra multiple-choice question showed that Mobius completed the reasoning with 516 tokens, whereas Qwen3.5 used 2,364 tokens to arrive at the same answer.

This reduction in token usage showcases Mobius's ability to optimize cognitive processes efficiently.

Evan: That’s a huge difference.

Any insights on how Mobius manages to be so efficient?

Ashley: Yes, the paper attributes this efficiency to Mobius’s inherent design features.

The use of backward residual connections enables lower layers to leverage higher-layer knowledge and vice versa, facilitating efficient information synthesis.

Moreover, the dynamic latent reasoning allows the system to optimally allocate computational resources, focusing more on complex tokens and less on simpler ones.

Evan: What about the cases where the problem might not be straightforward?

How does Mobius handle ambiguous or complex scenarios?

Ashley: The paper provides examples from both general tasks and scientific benchmarks to show how Mobius handles such cases.

In challenging scenarios, Mobius’s use of iterative latent refinement and its ability to synthesize knowledge efficiently come into play, leading to faster convergence and more accurate results.

For example, in a biology reasoning task, Mobius provided the correct answer with a much shorter reasoning trace, demonstrating both accuracy and efficiency.

Evan: That's quite compelling.

It seems like Mobius offers a significant step forward in model architecture, balancing both speed and accuracy effectively.

Ashley: Indeed, Evan.

By redesigning how knowledge and reasoning are handled within the architecture, Mobius significantly enhances efficiency without compromising on performance.

And that wraps up our deep dive into the experimental validation of Mobius’s capabilities.

Evan: Welcome back.

Now let's transition to examining the related work section, which places the Mobius architecture in the context of existing research.

Ashley, what kind of research does this paper build upon or contrast with?

Ashley: The authors draw on a wide body of research to contextualize Mobius.

They segment existing approaches into a few key categories: foundational transformer models, efficient attention mechanisms, recurrent neural networks, and latent reasoning models.

Let's address each of these areas.

Evan: That sounds like a solid structure.

Let’s start with transformer models then.

Ashley: Sure.

Transformers have arguably been the most significant advancement in deep learning over the past decade.

Introduced in the seminal paper 'Attention is All You Need' by Vaswani et al., transformers revolutionized the handling of sequential data by utilizing self-attention mechanisms.

This architecture has seen widespread application in various domains, including language, vision, and even scientific computing.

Evan: Right, transformers have truly been a game-changer.

What does the paper say about optimizing these models?

Ashley: The paper acknowledges efforts to optimize transformers primarily through scaling, such as increasing parameter counts and training on larger datasets.

However, as these models scale, they encounter efficiency bottlenecks.

To address this, previous research has explored various efficiency-improving techniques like sparse attention and linearization of self-attention mechanisms.

Yet, these approaches often sacrifice some degree of model capability for improved computational efficiency.

Evan: Understood.

How about recurrent neural networks and their role here?

Ashley: Recurrent Neural Networks, or RNNs, laid the groundwork for handling sequential and time-series data.

Albeit less powerful than transformers, they introduced essential concepts like sequence-to-sequence learning.

RNNs and their variants, such as LSTMs and GRUs, have been instrumental in developing early NLP models but faced issues like vanishing gradients and limited context windows.

Evan: Interesting.

And then we have the concept of latent reasoning.

What's that about?

Ashley: Latent reasoning, particularly in the context of language models, focuses on handling information in continuous latent space rather than through explicit symbols or tokens.

This approach promises more efficient and nuanced reasoning capabilities.

The authors highlight techniques like looped transformers and diffusion language models, which iterate over latent states to juice out more compact and high-information-density representations.

Evan: How does Mobius leverage these ideas differently from traditional models?

Ashley: Mobius builds on these ideas by separating the storage of knowledge from the process of reasoning, which enables dynamic and iterative reasoning within fewer layers.

The introduction of backward residual connections further allows information to flow bidirectionally within the model, a significant departure from traditional uni-directional information flow in RNNs and transformers.

Evan: So, it’s kind of an amalgamation of the best practices from different architectures?

Ashley: Exactly.

Mobius takes the strengths of various architectures and builds a refined, efficient model that addresses the inefficiencies and limitations identified in previous research.

By doing so, it also introduces novel concepts like shared Memory and Reasoners which have not been seen in such a capacity in earlier models.

Evan: What about any specific models or techniques that Mobius directly challenges or improves upon?

Ashley: The paper contrasts Mobius against several state-of-the-art models, primarily the standard Transformer architecture and its optimized variants.

For instance, it shows how Mobius, through continual pre-training from models like Qwen3.5, can significantly improve inference efficiency and reduce training data requirements while matching or exceeding the performance.

This juxtaposition illustrates Mobius’s potential to set new standards in the field.

Evan: This is fascinating.

How broad is the impact of Mobius, according to the paper?

Ashley: The impact is quite broad.

By decoupling knowledge and reasoning, Mobius not only improves performance metrics across various benchmarks but also introduces a more flexible and scalable architecture.

This positions it as a highly promising candidate for next-generation AI models, particularly in areas requiring efficient and scalable reasoning capabilities, such as scientific computing, language modeling, and real-time applications.

Evan: That neatly wraps up the related work section, showing how Mobius builds on and advances the current state of AI model architectures.

Evan: Welcome back for the final part of this episode.

Let's take a moment to summarize the key contributions and takeaways from the paper 'Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning.'

Ashley: To start with, the most significant contribution of the Mobius architecture is its decoupling of knowledge storage from reasoning operations.

By using a globally shared Memory for knowledge vectors and specialized Reasoners for compositional reasoning, Mobius achieves better knowledge compression and enhanced reasoning efficiency.

Evan: Definitely.

And this innovative design results in remarkable improvements, particularly in data efficiency and inference speed.

Mobius matches or exceeds the performance of traditional transformer models while requiring substantially less training data and delivering nearly four times the inference speedup.

Ashley: Additionally, key methodological advances include the introduction of backward residual connections, which allow for bidirectional information flow, and dynamic latent reasoning, enabling faster and more efficient token processing.

These features work in tandem to grant Mobius a significant edge over traditional architectures.

Evan: The paper also highlights impressive performance metrics across both general and specialized scientific tasks, showcasing Mobius's adaptability and efficiency in diverse applications.

This makes it a highly promising architecture for real-world deployment.

Ashley: In summary, Mobius represents a considerable leap forward in AI model architecture, offering a more efficient, scalable, and powerful approach to handling complex reasoning and knowledge storage tasks.

Its innovative design holds potential across various domains, from language processing to scientific computing.

Evan: That wraps up today's deep dive into the Intern-S2-Mobius architecture.

We hope you found this discussion informative and engaging.

If you enjoyed today's episode, make sure to tune in for future episodes of the Daily Paper Cast, where we bring you the latest and greatest in AI research.

Ashley: Don't forget to subscribe and leave us a review on your favorite podcast platform.

We always appreciate your feedback and support.

Thanks for listening, and we look forward to having you with us next time.

Evan: Until next time, stay curious and keep exploring the world of AI.

Goodbye, everyone!