ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search

🤗 Upvotes: 120 | cs.AI

Authors:
Jiyan He, Guang Liang, Hao Liu, Haoxiang Guan, Jinbo Sun, Junyi Guo, Wenjun Feng, Yantai Xie, Yifei Shen, Bin Shao, Chuyang Wei, Kai Chen, Kexin Zhou, Minghang Zhu, Shuxin Zheng, Tie-Yan Liu, Taine Zhao, Wenhui Zhu, Xueyin Xu, Xiaoqing Zhang, Yatao Li, Yuxuan Ren

Title:
ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search

Arxiv:
http://arxiv.org/abs/2609.13356v1

Abstract:
In this work, we present ZGCM-1, a fully open 7B dense foundation model trained from scratch with extreme data, system, and algorithmic efficiency. ZGCM-1 is founded on a core premise: compact models cannot passively memorize the open web, but can overcome parametric capacity limits by coupling deliberate internal thinking with active external tool use. To support this paradigm across a 256K context, we develop an end-to-end, high-efficiency open training recipe: Architecture & System Co-design: interleaved gated sliding-window and full attention, and a stable FP8 Muon optimizer; Progressive Curriculum & MDP Mid-Training: context scaling across 16K, 64K, and 256K, and the reformulation of interaction traces into Markov Decision Processes. Furthermore, we establish an AI-native R&D workflow where agent swarms autonomously manage cluster operations, data curation, and rapid diagnostic evaluation. Extensive evaluations show that ZGCM-1-7B is competitive across 7B model family on general benchmarks. On several challenging mathematical reasoning and agentic search suites, it remains competitive with frontier models orders of magnitude larger, such as Qwen3-235B-A22B and GLM-5.1. We also show that our pre-training design offers a ~4.2x efficiency improvement in 16K pre-training time-to-loss. Across the full development lifecycle, we distill eight actionable empirical findings-spanning architectural scaling, SFT quality pruning, long-context generalization, and agentic co-training dynamics. To facilitate community research, we open-source model weights from the pre-training, mid-training, and post-training stages, intermediate checkpoints, training code, per-stage data and data recipes, and W&B logs.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast!

Ashley: Today’s paper is from the Hugging Face daily paper list of September 15, 2026, with 120 upvotes.

Evan: The paper is titled 'ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search.'

Ashley: It’s authored by the ZGCM Team comprising members from Zhongguancun Academy and Zhongguancun Institute of Artificial Intelligence.

Evan: While foundation models continue to push the frontiers of mathematical reasoning and agentic problem solving, many researchers find themselves excluded due to prohibitive compute requirements and closed training recipes.

Ashley: ZGCM-1 addresses these barriers by presenting a fully open 7 billion parameter foundation model.

It’s trained from scratch, emphasizing extreme data, system, and algorithmic efficiency.

Evan: The core premise of ZGCM-1 is particularly intriguing.

Rather than passively memorizing the open web, it operates on the principle that compact models can transcend parametric capacity limits.

Ashley: How does it achieve that?

By coupling deliberate internal thinking with active external tool use, supporting this paradigm across a 256,000 token context.

Evan: That’s quite a large context window!

For this, they develop an end-to-end, high-efficiency open training recipe.

Ashley: Exactly.

They’ve incorporated several novel features, including interleaved gated sliding-window and full attention mechanisms, along with a stable FP8 Muon optimizer.

Evan: Their training approach sounds highly structured.

Can you break down their curriculum for us?

Ashley: Sure, Evan.

The curriculum involves progressive context scaling across 16,000, 64,000, and finally 256,000 tokens.

Interaction traces are reformulated into Markov Decision Processes to provide dense, step-level supervision.

Evan: I see they’ve also developed an AI-native R&D workflow.

What does that entail?

Ashley: In this setup, agent swarms autonomously manage cluster operations, data curation, and rapid diagnostic evaluation—basically speeding up the development cycle while maintaining efficiency.

Evan: So, how does ZGCM-1 compare to other models in terms of performance?

Ashley: The evaluations indicate that ZGCM-1-7B is competitive across the 7 billion parameter model family on general benchmarks.

On challenging mathematical reasoning and agentic search suites, it remains competitive with frontier models orders of magnitude larger, like Qwen3-235B-A22B and GLM-5.1.

Evan: Impressive!

They even show a 4.2x efficiency improvement in 16,000-token pre-training time-to-loss.

Ashley: Correct.

Across the entire development lifecycle, the team distills eight actionable empirical findings that span architectural scaling, fine-tuning quality pruning, long-context generalization, and agentic co-training dynamics.

Evan: And they’ve made all stages publicly accessible, right?

Ashley: They’ve open-sourced model weights from the pre-training, mid-training, and post-training stages, intermediate checkpoints, training code, per-stage data, and data recipes, along with W&B logs.

Evan: A truly comprehensive approach to fostering community research.

That's the end of the Introduction section of the paper.

Evan: Let’s dive into the methods used in the development of ZGCM-1.

Ashley: Sure, Evan.

The methodology behind ZGCM-1 is quite comprehensive, covering everything from architecture and system design to training techniques and evaluation protocols.

Evan: Great!

Let's start with the architecture.

What can you tell us about it?

Ashley: ZGCM-1 employs a decoder-only Transformer architecture, incorporating several state-of-the-art components like Grouped-Query Attention, RMS Normalization, SwiGLU activation, and Rotary Position Embedding.

Evan: That sounds sophisticated.

Can you break down these components for us?

Ashley: Grouped-Query Attention helps in better managing the attention heads, RMS Normalization stabilizes the training process, SwiGLU activation adds non-linearity, and Rotary Position Embedding enhances the model's capability to handle long sequences.

Evan: Got it.

And how is this architecture different from traditional ones?

Ashley: One unique aspect is its hybrid attention mechanism, which interleaves gated sliding-window attention with global attention at a 5:1 ratio.

The sliding-window attention reduces memory footprint significantly, while the global attention layers allow long-range dependencies to be captured.

Evan: Interesting!

How about the training process?

How did they make it so efficient?

Ashley: Efficiency is indeed a cornerstone of their approach.

They use an optimizer named Muon, which is paired with hybrid FP8 precision and TWEO outlier regularization.

This combination accelerates the pre-training process by approximately 4.2 times compared to traditional methods.

Evan: But how does this optimizer work?

Ashley: The Muon optimizer is designed for neural network hidden layers, combining momentum-based optimization with spectral scaling and Newton-Schulz steps.

This approach allows it to be faster and more stable than traditional optimizers like AdamW.

Evan: Fascinating!

And you mentioned a progressive curriculum earlier.

Can you elaborate on that?

Ashley: Of course.

The training curriculum scales the context length progressively from 16,000 tokens to 64,000 tokens, and ultimately to 256,000 tokens.

This scaling is accompanied by reformulating interaction traces into Markov Decision Processes, which provides dense, step-level supervision.

Evan: It sounds like a well-thought-out approach.

What datasets did they use for training?

Ashley: The training data is quite diverse, covering general web data, academic and OCR data, code data, and specialized mathematics data.

They also used reasoning-oriented material and LaTeX papers to enhance the model's capability in those specific areas.

Evan: Did they apply any special techniques to handle such a large and diverse dataset?

Ashley: Yes, they implemented a multi-stage data curation pipeline that combines deterministic heuristic rules with model-based quality scoring.

This ensures that only high-quality data is used for training, which is crucial for the model's performance.

Evan: And what about the fine-tuning process?

Ashley: For fine-tuning, they used a mixture of general and agentic datasets, focusing on both instruction following and deep research capabilities.

The fine-tuning process also involved rigorous tiered data pruning to maximize the model's capability coverage and minimize overfitting.

Evan: How did they evaluate the performance and efficiency of ZGCM-1?

Ashley: They used a wide array of benchmarks, covering general reasoning, mathematics, and agentic search tasks.

For instance, ZGCM-1-7B was evaluated on MATH-500, AIME 2024 to 2026, and the Harvard–MIT Mathematics Tournament.

They also ran benchmarks in agentic environments like WebWalkerQA and Binary Function Search.

Evan: And how did they fare in these benchmarks?

Ashley: ZGCM-1-7B performed impressively well, often matching or exceeding the performance of much larger models.

For example, it scored 97.1% on MATH-500, and 63.1% on WebWalkerQA.

It also showed a 4.2 times improvement in pre-training time-loss at 16,000 tokens.

Evan: It seems they have made significant advancements.

Anything else noteworthy about their methods?

Ashley: Yes, they also integrated AI-native R&D workflows, where LLM-based agents contribute to various stages of development—right from data cleaning and model design to experimentation and deployment.

Evan: That’s quite an autonomous setup.

How effective is this AI-native approach?

Ashley: It's highly effective, significantly reducing iteration cycles while maintaining high efficiency.

The team measured AI autonomy across various tasks, such as data cleaning, resource management, and model evaluation, and found that experimental and deployment engineering tasks reached high autonomy levels.

Evan: This integration of AI into the R&D lifecycle is quite fascinating.

So, to wrap it up, what are some of the key takeaways from their methodology?

Ashley: Key takeaways include the use of a hybrid attention mechanism, the Muon optimizer coupled with hybrid FP8 precision, and a progressive curriculum that enhances efficiency and performance across various benchmarks.

Their comprehensive approach, from pre-training to fine-tuning, has enabled ZGCM-1 to achieve impressive results.

Evan: Thank you, Ashley.

That wraps up our discussion on the Method section of the paper.

Evan: Now that we’ve covered the methodology, let’s move on to the experiments and results for ZGCM-1.

Ashley: Sure, Evan.

To evaluate the performance and efficiency of ZGCM-1, the team conducted extensive experiments across a range of benchmarks.

Evan: Great, let's start with the benchmarks they used.

What can you tell us about those?

Ashley: The benchmarks covered several key areas: mathematical reasoning, agentic search, general knowledge, and code generation.

For instance, they used MATH-500, AIME 2024 to 2026, the Harvard–MIT Mathematics Tournament, and WebWalkerQA, among others.

Evan: That’s quite a comprehensive set of benchmarks.

How did ZGCM-1-7B perform on these tests?

Ashley: ZGCM-1-7B performed impressively well.

In mathematical reasoning, for example, it scored 97.1% on MATH-500 and 75.0% on AIME 2026.

These scores are comparable to larger models, even those with hundreds of billions of parameters.

Evan: And what about the agentic search benchmarks?

Ashley: In agentic search, ZGCM-1 achieved 63.1% on WebWalkerQA and 19.4% on BrowseComp.

For Binary Function Search, it scored 62%, which is remarkable considering the complexity of these tasks.

Evan: Those are impressive numbers.

How did ZGCM-1 compare to other models in terms of efficiency?

Ashley: In terms of efficiency, ZGCM-1 demonstrated a 4.2 times improvement in pre-training time-to-loss for 16,000 tokens.

This was achieved through their innovative optimization techniques, including the Muon optimizer and hybrid FP8 precision.

Evan: Did they provide any details on the performance trends during training?

Ashley: Yes, they tracked several training metrics over time, showing continuous improvements across various checkpoints.

For example, MMLU scores improved from 26.37% at the start to 59.52% by the end of the general pre-training phase.

Evan: Did they analyze specific improvements in individual benchmarks?

Ashley: Benchmarks like HumanEval for code generation showed a rise from 13.66% to 43.54% over the training period, and reasoning benchmarks like ARC-Challenge improved from 24.96% to 75.90%.

Evan: It looks like they also compared ZGCM-1-7B to other models directly.

What were the results?

Ashley: Correct.

ZGCM-1-7B was compared against models like Qwen3-8B and MiniCPM4.1-8B.

Across 20 benchmarks, it consistently performed at the top or near the top.

For instance, on OlympiadBench, it scored 76.06%, second only to models with more parameters.

Evan: That’s impressive.

Did the results highlight any particular strengths or weaknesses?

Ashley: Yes.

The results highlighted that ZGCM-1-7B excels in mathematical reasoning and coding tasks.

However, it showed some room for improvement in instruction following and general knowledge tasks compared to its competitors.

Evan: How was the model’s performance in terms of agentic tasks compared to larger models?

Ashley: In agentic tasks, ZGCM-1-7B held its own against some of the largest models.

For instance, in WebWalkerQA, it scored 63.09%, which competes closely with Kimi-K2 and surpasses several other models.

Evan: Did they identify any areas where ZGCM-1-7B could improve?

Ashley: One area mentioned was the balance between instruction adherence and reasoning verbosity.

Heavy reasoning supervision can sometimes make the model verbose, affecting its strict instruction-following performance.

Evan: That’s a valuable insight.

Did they suggest any future directions based on these findings?

Ashley: They did.

Future directions include extending the architecture to sparse Mixture-of-Experts models to expand parametric capacity, transitioning towards multi-turn interactive reinforcement learning, and developing dynamic knowledge retrieval systems.

Evan: These directions sound promising.

Is there anything else noteworthy about the experiments?

Ashley: Yes, they also explored the effectiveness of their AI-native R&D workflow.

This involved AI agents managing cluster operations, data curation, and evaluation tasks autonomously, which significantly boosted their research efficiency.

Evan: It sounds like their integrated approach really paid off.

Thank you, Ashley.

That concludes our coverage of the Experiment section.

Evan: Alright, Ashley.

Let’s delve into the Related Work section.

How does ZGCM-1 build on or differ from previous research?

Ashley: Sure, Evan.

The Related Work section of the paper situates ZGCM-1 within the broader context of advancements in foundation models for mathematical reasoning and agentic search.

It references several key works in these fields to highlight ZGCM-1's contributions and unique aspects.

Evan: What kind of previous research does the paper reference?

Ashley: The paper starts by acknowledging the rapid advancements in foundation models that aim to tackle complex reasoning tasks.

It mentions notable models like DeepSeek-AI and Kimi Team’s projects, which have pushed the boundaries of long-horizon reasoning and tool-augmented agency.

Evan: Ah, I see.

How does ZGCM-1 compare with these previous foundation models?

Ashley: ZGCM-1 sets itself apart mainly through its compactness and efficiency.

Unlike models that require hundreds of billions of parameters to perform high-level reasoning tasks, ZGCM-1 aims to achieve similar outcomes with just 7.39 billion parameters.

This is achieved through a dual engine of internal thinking and external tool use.

Evan: Did the authors mention any specific techniques from past research that influenced their approach?

Ashley: Yes, they did.

They credit systems like Qwen3-235B-A22B and GLM-5.1 for demonstrating the effectiveness of extensive pre-training and sophisticated attention mechanisms.

These systems laid the groundwork for ZGCM-1’s hybrid attention approach, which interleaves sliding-window attention with global attention to manage long context lengths efficiently.

Evan: It sounds like ZGCM-1 builds on the successes of these previous models.

What about the optimization techniques?

Any influences there?

Ashley: The Muon optimizer, used in ZGCM-1, draws on earlier work that has shown the benefits of momentum-based optimization and spectral normalization in stabilizing and accelerating training.

Research by Jordan et al., 2024, and Liu et al., 2025, played a crucial role in this regard.

Evan: Interesting.

And what about the data strategy?

How does it compare with other models?

Ashley: The paper discusses the use of diverse data sources in training ZGCM-1, similar to strategies employed by models like SmolLM2 and Nemotron-Pretraining-Specialized-v1.

These references underline the importance of high-quality, varied data in enhancing a model’s reasoning capabilities.

Evan: So, ZGCM-1 integrates many best practices from these existing models but also introduces its own innovative techniques.

Were there any unique challenges they highlighted that were less addressed in prior works?

Ashley: Yes, one significant challenge was balancing the compact model size with the need for expansive reasoning capabilities.

The authors argue that previous models often relied heavily on sheer scale to cover for these needs.

ZGCM-1, however, demonstrates that efficiency in data and algorithm design can compensate for smaller parameter sizes.

Evan: And how does the AI-native R&D workflow fit into this picture?

Ashley: That’s another innovative element.

Previous research, like the works cited from OpenAI and Anthropic in 2026, often discussed the role of AI in automating parts of the research workflow.

However, ZGCM-1 takes this a step further by integrating LLM-based agents throughout the entire lifecycle, from data curation to model deployment, significantly improving efficiency and reducing iteration cycles.

Evan: It really seems like ZGCM-1 is a model that stands on the shoulders of giants while also carving out its own niche.

Any final thoughts on how this model positions itself in ongoing research?

Ashley: Indeed, Evan.

ZGCM-1 not only leverages the advancements in pre-training and optimization from past research but also pioneers new methods in training efficiency and AI-native R&D workflows.

It shows that compact models can be just as capable, if not more so, with the right innovations in place.

Evan: That wraps up our discussion on the related work section.

Evan: Alright, Ashley.

We’ve covered the introduction, methods, experiments, and related work.

Let’s wrap up with the key contributions and takeaways from the paper.

Ashley: Evan.

ZGCM-1 pushes the boundaries of what's possible with compact foundation models.

Their core contribution is demonstrating that advanced reasoning doesn't necessitate hundreds of billions of parameters.

Evan: Instead, it leverages a dual-engine approach, combining deliberate internal thinking with active external tool use.

This allows a 7.39 billion parameter model to operate effectively within a 256,000 token context.

Ashley: Precisely.

Some of their novel techniques include hybrid sliding-window and global attention mechanisms, the Muon optimizer paired with FP8 precision, and a progressive curriculum that scales context lengths dramatically.

Evan: And how did these techniques impact their results?

Ashley: Their techniques resulted in a 4.2 times improvement in pre-training time-to-loss for 16,000 tokens.

Additionally, ZGCM-1-7B performed impressively across various benchmarks, often matching or exceeding the capabilities of much larger models.

Evan: That’s remarkable, and their approach towards fostering open science is commendable, right?

Ashley: Correct.

They’ve made all stages of their model—pre-training, mid-training, and post-training—publicly accessible, including model weights, training code, data recipes, and evaluation protocols.

This sets a high standard for transparency and collaboration in the research community.

Evan: Incredible.

So, Ashley, what are the key takeaways for our listeners?

Ashley: Key takeaways include the effectiveness of compact models when paired with innovative attention mechanisms, the significance of data and algorithmic efficiency, and the potential of AI-native R&D workflows to accelerate research while maintaining high standards.

Evan: Eloquent as always, Ashley.

That brings us to the end of today’s episode on Daily Paper Cast.

Ashley: Thank you, Evan.

And thank you to our listeners for joining us.

We hope you found this discussion as enlightening as we did.

Evan: We’ll be back with more insights and discussions on the latest research papers tomorrow.

Don’t forget to subscribe and leave us a review.

Ashley: Have a great day, everyone!

Evan: Goodbye for now!