Questioning the Questions: Sustaining Self-Evolution in Reasoning Models

🤗 Upvotes: 57 | cs.LG, cs.AI, cs.CL

Authors:
Jinyuan Li, Chengsong Huang, Langlin Huang, Donghong Cai, Shiping Gao, Yuyi Yang, Jiaxin Huang

Title:
Questioning the Questions: Sustaining Self-Evolution in Reasoning Models

Arxiv:
http://arxiv.org/abs/2610.04299v2

Abstract:
Self-evolving reasoning models learn from their own generated questions, yet repeated self-training can lead to performance collapse. In this paper, we investigate why performance deteriorates over successive rounds and how to sustain self-evolution. Our analysis identifies two recurring quality problems in self-generated questions: invalid questions and repeated variants of the same mathematical questions. First, invalid questions become more prevalent across rounds, and answer-consistency filtering further increases their proportion in training data. Second, existing question diversity controls based on lexical similarity can miss mathematically equivalent questions expressed in different ways, which leads to question diversity collapse in later training rounds. Building on these findings, we introduce R-Quest, which uses question validity and novelty feedback to guide self-evolution. We first train the solver to recognize and reject invalid questions, then use its judgments to guide questioner rewards and filter solver training data. To avoid question repetition, we use a frozen base model to compare sampled question pairs and provide novelty feedback. Empirically, our method consistently achieves the highest average performance on 12 benchmarks in mathematical reasoning, general-domain reasoning, and code generation across two model families. Additionally, R-Quest maintains stable performance gains over ten rounds of self-evolution, peaking in the final round and outperforming R-Zero by 17.32 points.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast.

Evan: Today's paper is from the Hugging Face daily paper list of October 8, 2026, with 57 upvotes.

Ashley: The title is ‘Questioning the Questions: Sustaining Self-Evolution in Reasoning Models.’

Evan: The first two authors are Jinyuan Li and Chengsong Huang.

The corresponding author is Jiaxin Huang from Washington University in St.

Louis.

Ashley: Alright, let's dive into the introduction.

Self-evolving reasoning models represent an advanced approach where models learn by generating and solving their own training tasks.

Evan: That sounds intriguing, but what exactly are the goals of these models?

Ashley: The main goals are to reduce the dependence on human-curated data and to continuously improve the model's performance through self-training.

However, this process faces significant challenges as repeated self-training can lead to performance degradation.

Evan: Performance degradation?

Why does that happen?

Ashley: The paper identifies two key issues.

First, the prevalence of invalid questions increases across successive rounds of self-training.

Second, there’s a reduction in diversity due to the repeated generation of variants of the same mathematical tasks.

Evan: Interesting.

So, the focus is on understanding and solving these problems.

What did the authors propose to tackle this?

Ashley: To address these issues, they introduced R-Quest, a method that incorporates question validity and novelty feedback into the self-evolution process.

R-Quest first trains the solver to recognize and reject invalid questions.

Then, it uses a frozen base model to compare sampled questions and provide novelty feedback.

Evan: How does R-Quest ensure sustained performance improvement?

Ashley: R-Quest includes mechanisms to filter out invalid questions and minimize repetition, which helps maintain a high diversity of training tasks.

Empirically, this method consistently achieved the highest average performance on 12 benchmarks, outperforming existing methods.

Evan: That sounds promising.

How does R-Quest manage question repetition specifically?

Ashley: R-Quest uses a frozen base model to compare sampled question pairs and provide novelty feedback, ensuring that repeated tasks expressed in different ways are detected and minimized.

Evan: So, how does it guide the questioner roles in the process?

Ashley: The questioner role is guided through rewards based on question validity and novelty.

Valid and novel questions earn rewards, encouraging the generation of high-quality and diverse questions.

Evan: And how do these mechanisms impact the solvers?

Ashley: By filtering out invalid questions and promoting diversity, the solver is trained on a varied and high-quality set of tasks, which supports sustained self-improvement.

The method maintains stable performance gains over ten rounds of self-evolution, peaking in the final round and outperforming previous methods by a significant margin.

Evan: Was there any comparison mentioned to highlight R-Quest’s performance?

Ashley: Yes.

R-Quest showed improvements across mathematical reasoning, general-domain reasoning, and code generation.

Specifically, it achieved the highest average performance on 12 benchmarks, and it managed to maintain stable gains over ten rounds, peaking in the final round with a lead of 17.32 points over R-Zero.

Evan: That's quite impressive.

Does the introduction mention anything else regarding the practical implications of R-Quest?

Ashley: It emphasizes that assessing the validity and task novelty of self-generated questions provides a practical route to sustained self-improvement.

This is crucial for enhancing the robustness and effectiveness of self-evolving reasoning models.

Evan: And that wraps up the introduction section of this paper on sustaining self-evolution in reasoning models.

Let's move on to the next part.

Evan: Alright, Ashley, let's dive into the Method section of the paper.

How do the authors approach sustaining self-evolution in reasoning models?

Ashley: The authors have identified key issues related to self-evolution in reasoning models.

To address these, they propose R-Quest, which introduces two primary forms of feedback: question validity and task novelty.

Evan: So, what's the first step in R-Quest?

Ashley: The first step involves validity-aware solver initialization.

They construct a dataset from the solver training sets of previous iterations, specifically from the first five rounds of R-Zero.

This dataset is carefully annotated to include valid and invalid questions, and is used to train a solver to recognize and reject invalid questions.

Evan: And how is this validity feedback incorporated?

Ashley: During self-evolution, the solver uses its trained ability to detect invalid questions.

For each generated question, the solver assesses its validity while attempting a solution.

If the question is found invalid, it returns 'INVALID' as the answer instead of proceeding with a solution.

This way, the questioner is discouraged from generating invalid questions as they will not be used for training.

Evan: That makes sense.

What about the second form of feedback—task novelty?

Ashley: Task novelty feedback is used to address question repetition.

The frozen base model compares pairs of questions within each batch to identify if they are variations of the same underlying task.

If a repeated task is detected, the question is rejected.

They reduce the computational cost by sampling a limited number of comparisons per candidate question instead of comparing all question pairs exhaustively.

Evan: How do they balance between suppressing repeated tasks and encouraging novel questions?

Ashley: The novelty feedback is controlled by a parameter that dictates the number of random within-batch comparisons.

The authors found that setting this parameter appropriately balances suppression of repeated tasks while leaving room for challenging variants within recurring patterns, thus aiding sustained self-evolution.

Evan: Interesting.

How are these feedback signals used during self-evolution?

Ashley: The validity-aware solver first assesses the validity of each question.

If more than half of its responses are 'INVALID,' the question is classified as invalid and rejected.

For valid questions, the solver then provides consistency judgments by sampling multiple mathematical solutions.

Novelty feedback further filters out repeated tasks through pairwise comparisons.

This dual-filtering guides questioner rewards during alternating questioner-solver updates.

Evan: Can you explain the reward mechanism in detail?

Ashley: Certainly.

The questioner reward is based on two factors: validity and uncertainty.

Invalid questions receive a negative reward.

For questions that pass the validity check, uncertainty is used as a reward.

A question with moderate solver agreement, not too high or too low, maximizes the uncertainty reward, which encourages the generation of challenging problems.

Novelty feedback adjusts the reward based on if the question is deemed novel.

Therefore, only questions classified as both valid and novel earn high rewards.

Evan: How does the solver training proceed with these mechanisms in place?

Ashley: Once the questioner generates candidate questions, they are evaluated using the same criteria.

Valid questions with satisfactory answer consistency are retained for training the solver.

The solver is then updated using Group Relative Policy Optimization, or GRPO, incorporating this filtered dataset and a small fixed proportion of initial training data to maintain its validation capabilities.

Evan: Very detailed process!

How do they set the parameters and controls during training?

Ashley: For question evaluation, the authors use nine validity responses and ten mathematical solutions per question.

They employ a sampling temperature of 1.0 and a maximum response length of 4,096 tokens.

GRPO training employs feedback-driven rollouts, with specific settings for the number of steps, batch sizes, learning rates, and maximum prompt tokens.

All parameters are meticulously tuned to optimize performance.

Evan: And how has R-Quest performed compared to other models?

Ashley: R-Quest has consistently achieved the highest domain-average scores across evaluated methods on 12 benchmarks spanning mathematical reasoning, code generation, and general reasoning.

It maintains stable performance gains over ten rounds of self-evolution.

Notably, it reaches its highest score in the final round, surpassing R-Zero by a significant margin and outperforming the base model consistently.

Evan: That's substantial.

Any specific comparisons with baseline methods?

Ashley: Indeed.

Compared to baseline models like R-Zero, OCNR, and R-Diverse, R-Quest delivers significant improvements.

For mathematical benchmarks, it outperformed R-Zero by 2.54 points on average for Qwen3-4B-Base, and achieved a higher average performance on all metrics, including general-domain reasoning and code generation benchmarks.

These results validate the effectiveness of incorporating validity and novelty feedback in achieving sustained self-improvement.

Evan: That's the end of the Method section.

We're seeing a strong method approach here with clear improvements across varied benchmarks.

Evan: Now let's delve into the experimental setup and results that demonstrate the effectiveness of R-Quest.

Ashley: To evaluate the performance of R-Quest, the authors conducted extensive experiments using two model families: Qwen and Llama.

Evan: Which models did they specifically evaluate?

Ashley: They evaluated Qwen3-4B-Base and OctoThinker-3B-Hybrid-Base.

These models encompass different model families and scales, allowing a broad assessment of R-Quest's capabilities.

Evan: And what were the baseline methods they compared against?

Ashley: The comparison included the base models, R-Zero, OCNR, and R-Diverse.

R-Zero and OCNR are methods focused on self-evolution, while R-Diverse aims to mitigate diversity illusion in training.

Evan: Alright.

What benchmarks did they use for the evaluation?

Ashley: The benchmarks spanned three main areas: mathematical reasoning, general-domain reasoning, and code generation.

Specific benchmarks included AMC, Minerva, MATH, GSM8K, OlympiadBench, AIME24, and AIME25 for mathematical reasoning; MMLU-Pro, SuperGPQA, and BBEH for general-domain reasoning; and HumanEval+ and MBPP+ for code generation.

Evan: What metrics did they report for these benchmarks?

Ashley: For mathematical benchmarks, they reported mean accuracy (mean at 32) and strict accuracy under greedy decoding.

For general-domain reasoning, they followed the same protocol, reporting exact-match accuracy under greedy decoding.

For code generation, the metric was pass@1, evaluating the correctness of generated code on the test suites.

Evan: How did R-Quest perform across these benchmarks?

Ashley: R-Quest consistently outperformed existing methods.

It achieved the highest domain-average scores on all 12 benchmarks evaluated.

For example, on Qwen3-4B-Base, R-Quest surpassed R-Zero by 17.32 points and maintained stable performance gains over ten rounds of self-evolution.

Evan: That's remarkable.

Were there any specific highlights in their results?

Ashley: Certainly.

On mathematical reasoning benchmarks, R-Quest achieved an average score of 51.64%, which was notably higher than R-Zero’s score of 49.10%.

On code generation benchmarks, it improved the average pass@1 score to 62.69%, up from the base model's 58.79%.

Similarly, for general-domain reasoning, it achieved an average score of 32.75% compared to R-Zero's 31.21%.

Evan: Very impressive gains across the board.

What about the component ablations they tested?

Ashley: The authors also conducted component ablation tests to understand the contributions of validity and novelty feedback separately.

Removing either feedback lowered performance across all benchmark domains.

Validity feedback alone was shown to be more critical for sustained performance, indicating its key role in avoiding the collapse seen in R-Zero.

Evan: What did they find regarding validity assessment?

Ashley: R-Quest's validity assessment improved significantly over the ten rounds.

The model's balanced accuracy for detecting invalid questions increased from 64.80% to 72.93%, while invalid recall reached 86.36% by round ten compared to R-Zero’s range of 10.23% to 26.14%.

Evan: How did this validity improvement impact the quality of generated questions?

Ashley: R-Quest generated more valid questions as it progressed.

The validity rate of generated questions increased from 88.0% in round one to above 94.5% in later rounds.

This contrasts sharply with R-Zero, whose validity rate dropped to 49.0% by the tenth round.

Evan: And what about the solver’s ability to derive accurate answers from these questions?

Ashley: R-Quest maintained higher majority-answer accuracy on valid questions, ending at 54.69% in round ten, while R-Zero fell to just 2.06%.

This demonstrated R-Quest's ability to preserve the quality of self-generated supervision and support sustained self-evolution.

Evan: What did they find regarding question diversity during self-evolution?

Ashley: R-Quest limited the concentration of recurring exercise patterns effectively.

At the fifth round, the top-five share for R-Zero rose to 69.0%, whereas R-Quest remained much less concentrated at 24.0%.

However, the authors note that overly strict diversity control can negatively impact performance by suppressing challenging variants within recurring patterns.

Evan: A thoughtful balance is key.

Any insights from the analysis component?

Ashley: Yes, the analysis suggests that both validity and novelty feedback are crucial for achieving higher performance.

Continuous training through self-evolution strengthens the model's validity assessment while limiting question repetition effectively.

This highlights the essential nature of the proposed mechanism in R-Quest for sustained self-improvement.

Evan: It's clear that R-Quest presents a robust solution to the identified issues in self-evolving reasoning models.

That wraps up the detailed discussion of the Experiment and Results section.

Evan: Now, let's explore the Related Work section.

Ashley: In the context of self-evolving reasoning models, there's been a substantial amount of research focused on improving model performance through reinforcement learning and self-play.

Evan: Can you give some examples of these approaches?

Ashley: Certainly.

Label-free reinforcement learning is one such approach.

It relies on model confidence, answer agreement, and entropy as training signals.

Researchers like Zhao et al.

in 2026 and Zuo et al.

in 2026 have made significant contributions in this area.

Evan: What about self-play as an approach?

Ashley: Self-play is another key strategy where models learn by generating their own training tasks.

For example, Absolute Zero, introduced by Zhao et al.

in 2026, validates tasks and solutions through code execution.

Similarly, R-Zero and Self-Questioning Language Models, developed by Huang et al.

in 2025 and Chen et al.

in 2026 respectively, train question generators and solvers using model-derived feedback.

Evan: It sounds like there's been quite a bit of work in this area.

How does R-Quest's approach differ?

Ashley: R-Quest distinguishes itself by addressing the two main bottlenecks identified in previous self-evolution methods: the prevalence of invalid questions and the lack of diversity due to repeated question patterns.

Evan: Could you elaborate on how R-Quest tackles these issues differently?

Ashley: For validity, prior work includes studies on question validity in synthetic mathematical data.

For instance, Liang et al.'s work in 2026 and Shen et al.'s research in 2026 delve into this subject.

Typically, some methods rely on stronger external models to verify problem validity during self-play, like VHG from Lai et al.

in 2026, which uses GPT-5.4.

R-Quest, however, integrates validity judgment directly into the solver's training loop, avoiding the need for stronger external models during self-evolution.

Evan: And what about diversity in questions?

Ashley: Regarding diversity, recent methods like R-Diverse, proposed by Li et al.

in 2026, assess novelty using distances between solver representations or code-embedding similarity.

R-Diverse focuses on underlying reasoning procedures but depends on additional models like Qwen2.5-Coder-7B and Jina-Code-Embeddings-1.5B during self-evolution.

R-Quest simplifies this by using a frozen copy of the same base model to provide novelty feedback, determining if two questions are variations of the same task through random within-batch comparisons.

Evan: Seems like R-Quest has streamlined the process significantly.

Ashley: Exactly.

This approach not only reduces complexity but also enhances the efficiency of identifying valid and novel questions for the solver, improving sustained performance in self-evolution.

Evan: Are there other areas beyond self-evolution that are relevant to this research?

Ashley: Certainly.

Learning grounded in games, document corpora, vision tasks, and tool environments has also been explored in recent research.

Liu et al.

in 2026 examined model learning in document corpora, while Huang et al.

in 2026 and Li et al.

in 2026 focused on vision tasks and tool environments, respectively.

Evan: So, R-Quest builds on a broad foundation of existing work but specifically targets the issues of validity and task novelty to enhance self-evolution.

Ashley: Precisely.

By focusing on addressing the identified bottlenecks and leveraging the strengths of prior methods, R-Quest offers a robust solution for sustaining self-improvement in reasoning models.

Evan: And that concludes our discussion of the Related Work section.

Evan: Let's wrap up this episode with a summary of the key contributions and takeaways from the paper.

Ashley: Sure.

The paper ‘Questioning the Questions: Sustaining Self-Evolution in Reasoning Models’ primarily tackles two major issues faced by self-evolving reasoning models: the prevalence of invalid questions and the reduction in diversity due to repeated question patterns.

Evan: To address these, the authors introduced R-Quest, a method that combines question validity and task novelty feedback to guide the self-evolution of reasoning models.

Ashley: Exactly.

R-Quest starts by training a solver to recognize and reject invalid questions, ensuring that only well-formed questions are used during training.

This mechanism effectively reduces the proportion of invalid questions in the training data, enabling more reliable self-evolution.

Evan: For task novelty, R-Quest employs a frozen base model to compare question pairs within batches, identifying and minimizing repeated tasks.

This approach maintains a high diversity of training tasks, which is crucial for continuous model improvement.

Ashley: In their experiments, R-Quest demonstrated impressive performance gains across multiple benchmarks, including mathematical reasoning, general-domain reasoning, and code generation.

It outperformed existing methods like R-Zero, OCNR, and R-Diverse by a significant margin, showing sustained performance improvements over ten rounds of self-evolution.

Evan: Furthermore, the validity and novelty feedback mechanisms were shown to be critical, with continuous validation leading to stronger performance and reduced model collapse.

Ashley: Overall, this paper presents a robust framework for enhancing the robustness and effectiveness of self-evolving reasoning models, highlighting the importance of validity and diversity in achieving sustained self-improvement.

Evan: And that brings us to the end of today's episode.

We hope you found this discussion insightful.

Ashley: Be sure to join us next time as we continue to bring you the latest research from the world of AI, NLP, and computer vision.

Evan: Thanks for listening to Daily Paper Cast.

Until next time!

Ashley: Goodbye!

Have a great day ahead!