It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learning

🤗 Upvotes: 60 | cs.IR, cs.CL

Authors:
Runpeng Dai, Kaili Huang, Changsung Kang, Ciya Liao

Title:
It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learning

Arxiv:
http://arxiv.org/abs/2609.00638v1

Abstract:
Retrieval is the first stage of modern search and advertising systems, selecting a candidate set from a large item universe for downstream ranking and auction. Recent work increasingly leverages LLMs to improve retrieval through query expansion, data synthesis, and retrieval-feedback training. However, the generative component is typically used for query-side augmentation, while final matching is still delegated to a downstream retriever. We introduce CoGR, a retrieval framework that instead trains LLMs to directly construct retrieval representations on both query and item sides. Each generator produces a compact set of keywords, which are matched directly through an inverted index, preserving compatibility with existing keyword-based retrieval infrastructure. CoGR uses a two-stage training pipeline. Supervised fine-tuning first establishes an aligned keyword space, after which co-evolving reinforcement learning alternately optimizes the query- and item-side generators with GRPO against the opposite side's frozen index. Both sides optimize the same query-to-item retrieval $F_1$ objective: the query side receives retrieval $F_1$ directly, while the item side receives a counterfactual marginal reward measuring the change in query-side $F_1$ caused by its generated keywords. Across 10 representative sparse, dense, and generative baselines, CoGR achieves the best performance on both an internal APP Marketplace dataset and the public WANDS benchmark, improving $F_1$ over the strongest baseline by $10.9\%$ and $36.1\%$, respectively. Further analysis shows stable co-evolution and increasingly aligned query--item keyword spaces over training.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast.

Ashley: Today we’ll be discussing a paper from Hugging Face’s daily paper list dated September 3, 2026, which has received 60 upvotes.

Evan: The paper is titled 'It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learning'.

Ashley: This work was authored by Runpeng Dai from the University of North Carolina at Chapel Hill and Kaili Huang from Apple, with Runpeng Dai serving as the corresponding author.

Evan: Alright, let’s dive into what this paper's about.

Retrieval plays a pivotal role in modern search and recommendation systems.

At its core, retrieval selects a subset of relevant items or documents from a vast pool, which are then ranked or auctioned downstream.

Ashley: Exactly.

The retrieval stage is critical because its decisions are largely irreversible—if relevant items are missed here, they can't be recovered later.

On the flip side, irrelevant items making it through can bog down the ranking and auction processes.

Evan: Which, as you can imagine, makes finding a balance between precision and recall essential.

Traditional lexical retrieval methods, like BM25, rely on explicit term matches and inverted indexes.

But these methods often fall short in capturing deeper semantic relationships between queries and items.

Ashley: Exactly, Evan.

Thus, the community has explored various methods to transcend exact lexical matches.

Dense retrieval, for instance, maps queries and items into a shared continuous representation space, using vector similarity to find matches.

Generative retrieval, on the other hand, is a more recent approach where semantic identifiers are generated directly, replacing similarity-based matching with autoregressive forecasting.

Evan: That brings us to this paper's novel contribution.

The authors present a framework called Co-Evolving Generative Retrieval, shortened to CoGR.

Unlike typical generative methods which predominantly focus on query-side augmentation, CoGR aims to train large language models to generate keywords for both queries and items.

Ashley: Correct.

The beauty of CoGR lies in its two-stage training process.

The first stage, known as supervised fine-tuning, aligns the keyword space between query and item generators.

This stage ensures that generated keywords are semantically related, laying a solid foundation for meaningful retrieval.

Evan: And what happens in the second stage?

Ashley: The second stage involves co-evolving reinforcement learning.

Here, the query-side and item-side generators are alternately optimized using GRPO—Generalized Policy Optimization.

Each side is trained against a fixed reference index from the other side to iteratively refine the keyword generation process.

Evan: Interesting approach.

By alternately optimizing the generators, they aim to create an adaptive system where both keyword spaces can evolve and align more accurately over time.

Ashley: Yeah, and empirical results show this process works.

CoGR was tested against ten baselines across three categories: sparse, dense, and generative retrieval.

On both an internal APP Marketplace dataset and the public WANDS product-search benchmark, CoGR significantly outperformed the strongest baselines, improving the F1 score by 10.9% and 36.1%, respectively.

Evan: That's a notable improvement.

This paper definitely provides a promising new direction for retrieval in search and recommendation systems, balancing the trade-offs between recall and precision quite well.

Ashley: And with its innovative co-evolving training pipeline, CoGR demonstrates stable progression with increasingly aligned keyword spaces throughout training.

Evan: That wraps up the Introduction section of the paper.

Next, we'll delve into the specific methods used in CoGR's training pipeline.

Evan: Welcome back.

Let's dive into the methods behind CoGR, the co-evolving generative retrieval framework.

Ashley: Sure thing.

CoGR's method contains two primary phases: supervised fine-tuning, and co-evolving reinforcement learning.

These phases work together to align query and item keyword spaces for effective retrieval.

Evan: Let's break this down step by step.

What do we need to know about Phase 1, supervised fine-tuning?

Ashley: Phase 1, supervised fine-tuning, primarily serves to establish an aligned keyword space.

Initially, the items are each assigned a set of keywords using a base language model.

These keywords are sampled to create an initial keyword set for each item.

Evan: Got it.

What happens next with the queries?

Ashley: For queries, we pool keywords from all the relevant items and select the top keywords based on their frequency to create the query-side target keyword set.

By creating this overlap in keywords between queries and their relevant items, the system ensures that the query and item keyword spaces are aligned from the start.

Evan: And how is this used to fine-tune the models?

Ashley: These paired (query, keywords) and (item, keywords) datasets are then used to fine-tune the query-side and item-side keyword generators.

This yields initial policies for both sides, denoted as G_q_SFT for queries and G_i_SFT for items.

Evan: Alright, so that sets the stage for Phase 2: co-evolving reinforcement learning.

What does that involve?

Ashley: Phase 2 begins where Phase 1 leaves off, taking the pre-trained models and further optimizing them using reinforcement learning.

This phase alternates between optimizing the query-side generator and the item-side generator.

Let's start with the query-side reinforcement learning.

Evan: How does query-side reinforcement learning work?

Ashley: Given an item index generated by the frozen item-side LLM, the query-side generator produces a keyword set for each query.

These keywords are matched against the item index, and the retrieved items are evaluated for precision and recall.

The retrieval F1 score—balancing precision and recall—serves as the reward for the query-side generator.

Evan: And they keep track of the generated keywords to make sure they are effective?

Ashley: Yes, exactly.

There’s a maximum keyword budget, and any keyword set exceeding this budget gets no reward.

The GRPO training principle is followed here, where multiple keyword sets are sampled for each query, and the rewards are normalized to optimize the query generator.

Evan: What about the item-side reinforcement learning?

Ashley: Item-side reinforcement learning aims to refine the keywords for items.

A counterfactual index is constructed by replacing just one item's keywords while keeping all others unchanged.

The query-side index is frozen, and the item-side generator's keyword set is applied to see how much it affects overall retrieval quality.

The marginal reward is calculated based on the change in F1 score caused by the new keyword set.

Evan: It seems like there’s a lot of alternation going on.

Ashley: Indeed.

This alternation means each side gets optimized against a stable environment, avoiding a dynamic target that could complicate learning.

Through alternating reinforcement learning, the system progressively adapts the keyword spaces for queries and items.

Evan: What do you mean by optimizing against a stable environment exactly?

Ashley: Well, during each update round, the opposite-side index remains fixed.

This stability ensures that the generator isn’t chasing a moving target but rather refining its outputs with a consistent reference point.

Evan: That makes sense.

Now, what about efficiency?

How do they efficiently compute these changes?

Ashley: Naïvely computing item-side rewards would be computationally expensive because it would involve rebuilding the index and running retrieval for all queries each time.

Instead, they cache the retrieval states and only recompute for the queries affected by the keyword changes.

This significantly reduces the computational burden.

Evan: Now, based on what you’ve explained, this seems like a very flexible framework.

Can this method be adapted or extended in any way?

Ashley: The reward system in CoGR can be adapted to other RL algorithms like Proximal Policy Optimization (PPO).

And the system can also be adjusted to prioritize business-specific outcomes over pure retrieval metrics.

For instance, the reward signal could integrate business metrics like revenue gains.

Evan: And how about practical implementations?

I assume they tested this framework on real datasets?

Ashley: Yes, they evaluated CoGR with two industrial search datasets: an internal APP Marketplace dataset and the public WANDS benchmark.

Both datasets consist of de-identified user queries and provide categorical relevance annotations converted into binary relevant and irrelevant labels.

Evan: And from the results we mentioned earlier, it’s clear that CoGR outperformed the other methods, right?

Ashley: Correct.

Across these datasets, CoGR achieved better F1 scores compared to other sparse, dense, and generative retrieval baselines.

This confirms the effectiveness of their two-stage training and co-evolving approach.

Evan: That wraps up the Method section.

Next, we'll discuss the experimental results and insights from this approach.

Evan: Alright, let's move on to the experiments and results.

How did CoGR perform in its evaluations?

Ashley: The authors evaluated CoGR on two main datasets: the internal APP Marketplace search dataset and the public WANDS product-search benchmark.

Both datasets consist of de-identified user queries with categorical relevance annotations which are converted into binary relevant and irrelevant labels.

Evan: So, what kind of retrieval baselines did they compare CoGR against?

Ashley: They compared CoGR against three representative families of retrieval methods: sparse retrieval, dense retrieval, and generative retrieval.

These included traditional methods like BM25 for sparse retrieval; DPR, ANCE, and Qwen3-Embedding-4B for dense retrieval; and DSI, DSI-QG, RIPOR, and DeepRetrieval for generative retrieval.

Evan: That's a wide range of baselines.

How did CoGR perform compared to these methods?

Ashley: CoGR significantly outperformed all baselines across both datasets.

Specifically, it achieved the highest F1 scores consistently, demonstrating remarkable improvements.

On the internal APP Marketplace dataset, CoGR improved the F1 score over the strongest baseline by 10.9%.

On the WANDS dataset, it showed an even more impressive improvement of 36.1% over the strongest baseline.

Evan: Wow, those are substantial improvements.

Do they break down the performance details further?

Ashley: Yes, they report detailed metrics including precision, recall, mean reciprocal rank (MRR), normalized discounted cumulative gain (NDCG), and F1 scores at different retrieval cutoffs.

For instance, at the top-100 retrieval cutoff, CoGR consistently achieved the highest precision, recall, and F1 scores across both datasets.

Evan: What about the dynamics of the co-evolving training process?

Do they provide insights into how the training progressed?

Ashley: They do.

Figure 3 in the paper illustrates the evaluation F1 scores over cumulative training steps.

Starting from the SFT-initialized generators, the F1 scores showed a steady increase as query-side and item-side generators alternated in learning.

The scores notably increased from approximately 0.16 before co-evolving to around 0.40 after five rounds of alternation.

Evan: That's impressive to see such a stable improvement.

Did the paper discuss the significance of co-evolving both sides of the retrieval process?

Ashley: The experiments indicated that co-evolving both query and item generators was crucial for strong performance.

CoGR substantially outperformed versions where only one side was trained while the other remained static.

This dynamic co-evolving process was essential for learning a well-aligned retrieval system.

Evan: That makes sense.

So, it's clear that the iterative optimization of both generators played a crucial role.

Any other notable findings from their analysis?

Ashley: Yes, the paper also includes an ablation study examining the robustness and relevance of several design choices.

For example, they tested the effect of using different reward functions and whether or not to include the supervised fine-tuning stage.

Results showed that each design component contributed to the model's overall effectiveness and stability.

Evan: Got it.

Lastly, were there any qualitative insights or specific examples of how CoGR improved retrieval quality?

Ashley: Indeed.

The authors provided examples of generated keywords demonstrating how they evolve during training.

Initially, keywords were broad and generic, like 'mobile' or 'fun'.

Over time, they became more specific and relevant, such as 'investment tracking' or 'survival horror'.

This showed that the co-evolving process refined the specificity and alignment of keywords, improving overall retrieval quality.

Evan: That's fascinating.

It really shows how adaptive and refined the framework can become with proper training.

Ashley: Exactly.

That’s the end of the Experiment section.

Next, we’ll dive into the discussions and conclusions drawn from these results.

Evan: Now, let's delve into the related works that provide context for CoGR.

Ashley, take us through the landscape of retrieval methods explored in this paper.

Ashley: Sure thing, Evan.

The related works section in this paper spans across three major categories: sparse retrieval, dense retrieval, and generative retrieval, each contributing to the foundation upon which CoGR is built.

Evan: Let's start with sparse retrieval.

What's the traditional approach here?

Ashley: Classical sparse retrieval methods like TF-IDF and BM25, focus on lexical matching.

These methods rely on explicit term matches using inverted indexes to rank documents.

The paper references works like docTTTTTquery and SPLADE, which retain this inverted-index interface but improve it with learned term weights and lexical expansions.

Evan: And how do dense retrieval methods differ?

Ashley: Dense retrieval maps queries and items into a shared continuous representation space.

Matching is then done through vector similarity.

Representative methods in this category include DPR, ANCE, RocketQA, and GTR.

These methods involve dual encoders and often use advanced negative sampling techniques to improve retrieval accuracy.

Evan: Interesting.

How about generative retrieval, which seems relatively new?

Ashley: Generative retrieval is indeed a newer paradigm.

It replaces nearest-neighbor search with autoregressive generation of document identifiers.

Systems like DSI, GENRET, and ASI explore different approaches to document tokenization and identifier optimization.

These methods aim to generate unique identifiers for documents directly from queries, streamlining the retrieval process.

Evan: How does this generative approach work in practice?

Ashley: For example, DSI assigns semantic IDs to documents and trains a sequence-to-sequence model to generate these IDs from queries.

NCI improves this framework by adding query augmentation and prefix-aware decoding.

More recent methods like GLEN and NOVO learn lexical identifiers from retrieval supervision, which can be more flexible and interpretable.

Evan: It seems each category has its strengths.

How does CoGR build on these methods?

Ashley: CoGR unites elements from all these approaches.

It trains large language models to generate keywords for both queries and items, functioning within a keyword-based retrieval framework.

This dual-generation process is optimized via a two-stage method involving supervised fine-tuning and co-evolving reinforcement learning, combining the precision of sparse methods with the semantic depth of dense and generative retrieval.

Evan: And how does reinforcement learning fit into all this?

Ashley: Great question.

Recent works have incorporated reinforcement learning to enhance retrieval by optimizing LLMs with verifiable rewards.

For instance, DeepRetrieval trains an LLM query generator directly from retrieval-metric rewards.

CoGR adapts this approach by implementing a co-evolving reinforcement learning framework that optimizes both query-side and item-side generators.

Evan: Does the paper cite any specific works on this?

Ashley: Yes, it does.

Works like those by Jiang et al.

on DeepRetrieval, and others on RLVR—Reinforcement Learning with Verifiable Rewards—are highlighted.

These methods use LLMs to generate pseudo-documents or rewritten queries that improve retrieval performance when integrated with traditional sparse or dense retrievers.

Evan: Fascinating.

Are there any notable improvements the paper acknowledges from these previous works?

Ashley: Certainly.

One notable improvement is the integration of business metrics into the reward signal, going beyond pure retrieval metrics.

This aligns the optimization process more closely with practical business outcomes, which isn't commonly addressed in prior research.

Evan: That definitely sounds like a practical advancement.

Did the authors identify any trends or future directions in their related works section?

Ashley: They did.

The paper notes trends like the increasing flexibility and scaling of LLMs in retrieval tasks and the potential for self-evolving systems.

Future directions might involve expanding beyond retrieval metrics to include direct business metrics like revenue or ads relevance.

The related work emphasizes the ongoing evolution of hybrid LLM systems that can integrate generation and retrieval in more sophisticated ways.

Evan: A lot to consider for future research.

That concludes the Related Work section.

Next, we’ll move on to the concluding discussions and broader impacts.

Evan: To wrap up today's episode, let's summarize the key contributions and takeaways from this paper.

Ashley: Sure.

The paper introduces CoGR, a novel framework for co-evolving generative retrieval using reinforcement learning.

CoGR stands out because it generates keywords for both queries and items, allowing for direct matching using keyword-based infrastructure.

Evan: Exactly.

The two-stage training pipeline—starting with supervised fine-tuning to establish an aligned keyword space, followed by co-evolving reinforcement learning—ensures that the system progressively improves and aligns query and item representations.

Ashley: CoGR showed significant improvements in retrieval performance.

When tested on the internal APP Marketplace dataset and the WANDS benchmark, it outperformed strong baselines, delivering a 10.9% and 36.1% improvement in F1 scores, respectively.

Evan: The paper also highlights the importance of co-evolving both query and item generators, demonstrating that this dynamic process leads to better-aligned keyword spaces and overall retrieval performance.

Ashley: Additionally, by alternating the optimization process between query- and item-side generators, CoGR ensures that each iteration refines the keyword sets effectively, adapting them to serve retrieval tasks better.

Evan: Beyond just retrieval metrics, CoGR's framework can be adapted to include direct business outcomes, making it highly relevant for industrial applications where practical impact matters.

Ashley: This approach opens up new possibilities for creating agile and robust retrieval systems.

It points to future research directions, like incorporating business metrics into the reward functions and further refining the co-evolution process.

Evan: That concludes our discussion on CoGR.

Thanks for joining us on Daily Paper Cast.

Ashley: We hope you found today's insights valuable.

Be sure to tune in tomorrow for another deep dive into cutting-edge research from the world of AI and machine learning.

Evan: Until next time, keep exploring, and stay curious!