Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?

🤗 Upvotes: 95 | cs.AI

Authors:
Yibo Li, Jinhang Qiu, Zhi Zheng, Qianyun Guo, Jiaying Wu, Shuo Ji, Bryan Hooi

Title:
Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?

Arxiv:
http://arxiv.org/abs/2610.08215v2

Abstract:
Learning from experience is essential for LLM agents to adapt to unfamiliar and dynmaic environments. Evaluating this ability is therefore important for understanding how effectively agents acquire and use new knowledge. Existing benchmarks have sought to evaluate this ability, but they primarily evaluate tasks whose rules are provided in the instructions or already familiar to pretrained models, making it difficult to distinguish learning from interactions from reasoning with existing knowledge. To address this, we introduce Learn2Play Bench, a benchmark of newly designed text-based games, whose rules are novel or counterintuitive, requiring agents to acquire knowledge through interaction rather than rely solely on pretrained knowledge. These games provide reproducible feedback and automatic scoring, enabling controlled evaluation of learning across repeated attempts. We also vary game instances to test whether agents can apply what they have learned to new situations. Therefore, we evaluate how backbone models, self-evolving methods, and agent harnesses affect agents' learning ability, revealing three findings: (1) Experience retention: Retaining complete records of actions and feedback can support more effective learning than summarizing these experiences into rules or strategies. (2) Human agent gap: Top-performing human players achieve higher peak scores than the evaluated agents. Human explore more varied strategies, and repeat actions less. (3) Harness matters: With the backbone fixed, changing the harness can improve performance while reducing estimated inference cost. Together, these findings provide insights into how LLM agents learn from experience and suggest directions for future work to improve their learning ability. Project website: https://liushiliushi.github.io/learn2play-bench-website/

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast.

Ashley: Today's paper is from the Hugging Face daily paper list of October 9, 2026, and has received 95 upvotes.

Evan: The paper is titled 'Learn2Play Bench: How Well Do LLM Agents Learn From Experience in Unfamiliar Environments?'

Ashley: Authored by Yibo Li and Jinhang Qiu, with corresponding authors Yibo Li and Bryan Hooi from the National University of Singapore.

Evan: Alright Ashley, let's dive into the introduction of this paper.

Can you start by explaining the background and objectives of this research?

Ashley: Evan.

Large Language Models, or LLMs, have demonstrated impressive capabilities in performing complex tasks by leveraging extensive pretrained knowledge.

However, their ability to adapt to unfamiliar and dynamic environments by learning from interactions and experiences remains a crucial challenge in achieving artificial general intelligence, or AGI.

Evan: That's a significant challenge, indeed.

So, what exactly was the problem the authors were addressing in this paper?

Ashley: Existing benchmarks often fail to discern whether LLM agents truly learn from new experiences or merely rely on their pretrained knowledge.

They evaluate tasks where rules are either provided in the instructions or are familiar to pretrained models, which blurs the distinction between learning through interaction and reasoning with pre-existing knowledge.

Evan: Interesting.

How did the authors aim to tackle this problem?

Ashley: To address this issue, the authors introduced Learn2Play Bench, a benchmark comprised of newly designed text-based games.

These games feature novel or even counterintuitive rules, compelling agents to acquire knowledge through interaction rather than relying solely on their pretrained knowledge.

Evan: Novel and counterintuitive rules sound like an effective way to test learning through interaction.

What makes this benchmark particularly useful for evaluating LLM agents?

Ashley: Learn2Play Bench provides reproducible feedback and automatic scoring, facilitating controlled evaluations of learning across repeated attempts.

The authors also varied game instances to test whether agents could apply learnt knowledge to new situations.

Essentially, they evaluated how backbone models, self-evolving methods, and agent harnesses impact the agents' learning ability.

Evan: Let's break down those three aspects: backbone models, self-evolving methods, and agent harnesses.

What were the main findings regarding the experience retention of these agents?

Ashley: The authors discovered that retaining complete records of actions and feedback supports more effective learning than summarizing these experiences into rules or strategies.

This is because the original evidence remains available when earlier conclusions need revision.

Evan: That makes sense.

Complete records can provide a fuller picture for making informed decisions.

Now, what did the paper reveal about the performance gap between human players and LLM agents?

Ashley: Yes, the human–agent gap was quite significant.

Top-performing human players achieved higher peak scores than the evaluated agents.

Humans explored more varied strategies and repeated actions far less frequently than the agents, which enhanced their performance.

Evan: Fascinating.

And lastly, regarding the harnesses, what impact did they observe on agent performance and inference costs?

Ashley: When it comes to harnesses, the authors found that changing the harness can improve performance while reducing estimated inference costs, even with the backbone model fixed.

This suggests that the ways agents manage and use their context can significantly influence their learning effectiveness.

Evan: It’s evident how these insights could help in advancing LLM agents' adaptive capabilities in unfamiliar environments.

And I believe that's the wrap for the Introduction section of this paper.

Evan: Alright Ashley, now let’s dive into the methods used in the Learn2Play Bench paper.

Can you walk us through the core design of this benchmarking framework?

Ashley: Evan.

The Learn2Play Bench is designed around newly created text-based games that agents must explore to discover hidden rules through interaction.

These games are crafted to be unfamiliar and often counterintuitive, making it challenging for agents to rely on pretrained knowledge alone to succeed.

Evan: What specific principles guided the design of these games?

Ashley: There were four primary principles.

First, rule novelty.

The games include hidden rules that do not align with common pretraining knowledge.

Second, automatic scoring, which allows for large-scale automated evaluations without human intervention.

Third, reproducibility; every game produces deterministic outcomes that help ensure consistent and comparable evaluations.

Lastly, controlled variation across episodes.

This means that game instances can change in appearance and context while keeping the underlying rules fixed.

Evan: So, these games are not just static but also include varying elements to make the agents truly learn from their experiences.

Sounds interesting.

Could you give us examples of the types of games included in this benchmark?

Ashley: Sure, Evan.

Learn2Play Bench includes a suite of 20 game templates, each with numerous instances generated by varying layouts, characters, or parameters.

For instance, in the game 'POISONER,' agents serve poisoned dishes to a king with hidden food preferences.

They must determine the rules such as which dishes are likely to be accepted or rejected and aim to serve three successful courses to achieve high scores.

Evan: Interesting!

And how are these games evaluated?

What was the experimental protocol?

Ashley: The agents were evaluated through a black-box command interface, where they issued textual commands and received feedback.

Each agent was tested across 20 game instances, using one fixed seed per game template to maintain consistency.

Evan: And what does a typical evaluation cycle look like for these agents?

Ashley: Good question.

Each game evaluation involved multiple episodes—five for normal games and ten for more challenging games, which provide a broader observation window for learning.

The evaluations were repeated over three independent trials for each agent, and each trial started with no memory of previous attempts to ensure that within-trial learning was accurately measured.

Evan: Great, now what kind of metrics were used to assess the performance and learning abilities of the agents?

Ashley: Four primary metrics were used: Max, Mean, Learning Gain (LG), and Learning Slope (LS).

Evan: Can you explain each of these metrics in more detail?

Ashley: Of course. 'Max' measures the peak performance by averaging the highest scores of each trial within the evaluation window. 'Mean' averages scores across all evaluated episodes, indicating consistency. 'Learning Gain' compares average performances of early episodes with the final ones, measuring how much performance improved from start to finish.

Lastly, 'Learning Slope' fits a linear trend to measure the overall rate and direction of performance changes across episodes.

Evan: These metrics certainly give a comprehensive view of both peak performance and consistency.

Anything specific about how the learning curves were analyzed?

Ashley: Yes, aggregate learning curves were plotted for each game, mapping episode progress on a scale from 0 to 100%, with normalization for easier comparison across different games with varying episode lengths.

Evan: How did the authors ensure a fair comparison across different games and agents?

Ashley: They assigned weight to the games—three for more challenging games and one for normal games—to balance the impact on the aggregate metrics.

Also, the deterministic nature of game outcomes and the fixed seeds helped ensure that each agent faced identical scenarios for fair comparisons.

Evan: Got it.

With all these rigorous methods in place, how did the authors handle the different methods of retaining and reusing experiences?

Ashley: Agents were subjected to various self-evolving methods to see how different strategies for retaining experiences affected learning.

These methods included retaining unmodified interaction histories ('Memory'), reflecting on completed episodes to summarize rules and strategies ('Reflexion'), evolving action-guiding prompts and state-extraction programs ('EvoTest'), and others involving reusable workflows and reasoning patterns.

Evan: These sound quite comprehensive.

How did these methods compare in their effectiveness?

Ashley: Interestingly, the findings revealed that raw interaction histories generally supported stronger learning than those methods that summarized or processed experiences into rules or strategies.

This was primarily because raw histories kept the original evidence accessible for revising interpretations when needed.

Evan: That makes sense; sometimes, retaining the full context can be incredibly beneficial.

It sounds like they left no stone unturned in testing these methods.

Ashley: Indeed, by systematically comparing different models, methods, and game variations, the authors provided a deep insight into how LLM agents learn from experience.

This kind of detailed evaluation is essential for progressing towards more adaptive and intelligent AI systems.

Evan: And that’s the wrap-up for the Method section.

It’s fascinating to see so much thought going into making sure these agents are truly learning from scratch.

Next, we’ll dive into the experiments and results to see how these methods performed in practice.

Evan: Alright, Ashley.

Let’s delve into the Experiments and Results section of the Learn2Play Bench paper.

How did the authors structure their experimental evaluations?

Ashley: The authors conducted a comprehensive range of experiments to evaluate how well agents learn from experience.

They tested eleven different backbone models and several self-evolving methods, comparing their performance using the metrics we discussed earlier: Max, Mean, Learning Gain (LG), and Learning Slope (LS).

Evan: That sounds extensive.

Can you provide more details about the backbone models they evaluated?

Ashley: Sure.

The models evaluated were GPT-6 Astra, Claude Opus versions 5.5, 5, and Sonnet 5, Gemini 3.5 Flash and 3.7 Flash, GPT-5.6-SOL, GPT-5 Mini, DeepSeek V4 Pro, Kimi K3, and Qwen 3.8 Max.

These models were tested using a common harness to ensure a fair comparison.

Evan: How did these backbone models perform in terms of the key metrics?

Ashley: Performance varied significantly among the models.

For instance, Claude Opus 5.5 achieved the highest Max and Mean scores, indicating strong peak performance and consistency.

However, Qwen 3.8 Max showed the greatest Learning Gain and Learning Slope, suggesting it improved the most through interaction.

Evan: Were there any particular observations about how the models used feedback to learn?

Ashley: Yes, stronger models like Opus 5.5 and Qwen 3.8 Max adjusted their decisions based on feedback, testing their understanding of game rules and revising it as needed.

In weaker models, such as GPT-5 Mini, there were instances where the agent repeated actions that previously worked, showing less adaptability.

Evan: Interesting.

Now, what about the self-evolving methods?

How did they compare in effectiveness?

Ashley: The paper compared various methods like Memory, Reflexion, EvoTest, and ReasoningBank.

It turned out that the Memory method, which retains complete interaction histories, generally supported stronger learning across models like Kimi K3.

This is because keeping raw interaction histories allowed agents to revisit and revise interpretations effectively.

Evan: Were there any standout comparisons among these self-evolving methods?

Ashley: When comparing methods using Claude Opus 5, EvoTest showed high initial performance but didn’t sustain in later episodes.

On the other hand, Reflexion and Memory methods maintained better consistency and learning progress.

This indicates that how experiences are retained and reused can significantly impact ongoing learning.

Evan: And how did the learning abilities of agents compare to human players?

Ashley: Human players, particularly the top performers, achieved higher peak scores and exhibited more varied strategies.

They were better at recovering from setbacks and adapting strategies.

In contrast, agents often repeated actions and showed less exploratory behavior.

Evan: Did the study identify any specific areas where agents lagged behind humans in terms of learning?

Ashley: Yes, the study found that agents often struggled with forward planning and adaptability.

For example, in the game HAUNTED INN, an agent might know the taboos but fail to plan room assignments effectively, often leading to trapped scenarios for later guests.

Evan: What did the research reveal about the impact of harnesses on agent performance?

Ashley: Harnesses turned out to be quite influential.

When keeping the backbone fixed, changing the harness could lead to notable performance improvements and reduced inference costs.

For instance, transition from OpenCode to Claude Code significantly improved metrics in the same backbones.

This shows the importance of harness configurations in optimizing LLM agent performance.

Evan: All in all, these experiments provide deep insights into how agents learn from interactions in unfamiliar environments.

This extensive evaluation helps uncover key areas for improvement.

And that wraps up the Experiment section.

Evan: Now that we've looked at the experiments and their outcomes, Ashley, can you take us through the Related Work section of the Learn2Play Bench paper?

Ashley: Of course, Evan.

The Related Work section provides a comprehensive overview of existing research and benchmarks in the realm of self-evolving agents, memory mechanisms, and continual learning for LLM systems.

Evan: What significant pieces of research do the authors reference?

Ashley: The authors highlight several key advancements in self-evolving agents that reuse interaction feedback through memories, extracted rules, reasoning patterns, or evolving agent configurations.

For example, early methods such as Reflexion focus on verbal reflections, helping agents derive concise lessons about rules and strategies from past interactions.

Evan: What are some other methods mentioned, and how do they contribute to the development of these agents?

Ashley: Other significant methods include ExpeL, which utilizes extracted insights and retrieved trajectories, and Voyager which employs reusable executable skills.

Dynamic Cheatsheet curates compact strategies and code snippets, while Agent Workflow Memory induces action workflows from successful interaction histories.

These methods aim to organize experience into more usable formats that can guide future decision-making.

Evan: It sounds like there's a lot of emphasis on reusing experience to improve agent performance.

How does this compare with the approaches used in Learn2Play Bench?

Ashley: Indeed, Evan.

While these methods reuse experience without updating model weights through various structured formats, Learn2Play Bench focuses on keeping the raw interaction histories unmodified.

This approach ensures that the original evidence remains available for revising any premature conclusions, potentially supporting more effective learning compared to summary-based methods.

Evan: Did the authors mention any benchmarks for evaluating these adaptation mechanisms?

Ashley: Benchmarks like StreamBench, MemoryBench, and Evo-Memory evaluate learning improvement over repeated interactions and accumulated feedback.

StreamBench uses input–feedback streams, whereas MemoryBench studies how agents learn from accumulated user feedback.

Evo-Memory goes a step further by examining how agents update their memory across task streams.

Evan: How do these benchmarks differ from Learn2Play Bench in terms of testing environments?

Ashley: Good question.

One of the main differences is the nature of tasks used for evaluation.

Benchmarks like LifelongAgentBench and SWE-Bench-CL test skill accumulation across related tasks in familiar real-world settings.

On the contrary, Learn2Play Bench introduces entirely new text-based games with unfamiliar and counterintuitive rules, challenging agents to discover these rules through interactions.

Evan: This certainly seems to offer a different angle of evaluation.

Are there benchmarks that focus specifically on direct interaction learning with new environments as Learn2Play Bench does?

Ashley: Yes, there are some benchmarks that center around learning through direct interactions.

AutoEnv, for instance, creates new environments to evaluate agent learning across diverse settings.

J-TTL measures repeated improvement in Jericho games, although the public content in Jericho might already be part of model pretraining.

ARC-AGI-3 tests interaction with unfamiliar visual environments, but these evaluations also depend significantly on visual capabilities.

Evan: It seems like the benchmarks span different domains and approaches to learning.

How does this paper position itself among these existing works?

Ashley: The Learn2Play Bench paper positions itself uniquely by focusing on text-based games with hidden rules.

It aims to test the ability of LLM agents to discover new knowledge through actions and feedback in purely text-based unfamiliar environments, providing a controlled setting to measure effective learning without updating model weights.

Evan: Were there any specific comparisons made between the proposed benchmark and others like J-TTL or AutoEnv?

Ashley: Yes, the paper draws comparisons by emphasizing the novelty of hidden rules and the deterministic feedback provided by Learn2Play Bench, which allows for reproducible evaluation.

While AutoEnv and J-TTL also create new environments for evaluation, the text-based nature and hidden rule discovery in Learn2Play Bench add a distinctive layer of complexity and challenge.

Evan: Right, focusing on text-based interactions certainly provides an interesting angle.

And that's the wrap for the Related Work section.

Evan: We’ve covered the introduction, methodology, experiments, and related work.

Ashley, can you summarize the key contributions and takeaways from the 'Learn2Play Bench' paper?

Ashley: Certainly, Evan.

The paper makes several significant contributions to the evaluation of LLM agents.

Firstly, it introduces Learn2Play Bench, a controlled benchmark of newly designed text-based games with hidden rules and automatic scoring.

This novel setup tests agents' ability to acquire new knowledge through interaction rather than relying solely on pretrained models.

Evan: That’s a solid foundation for evaluating learning through experience.

What else stood out in their findings?

Ashley: The authors systematically compared different backbone models, self-evolving methods, and agent harnesses to reveal factors affecting learning from experience.

They found that retaining complete raw interaction histories led to more effective learning than compressing experiences into rules or strategies.

Additionally, the human–agent gap was notable, with human players achieving higher peak scores and employing more varied strategies than LLM agents.

Evan: The comparison with human players is particularly telling.

What about the impact of different harnesses on performance and cost?

Ashley: Harnesses had a significant impact as well.

Changing the harness improved performance and reduced inference costs, even when the backbone model remained the same.

This demonstrated the crucial role of harness configurations in optimizing agent performance and efficiency.

Evan: It sounds like a comprehensive study that offers valuable insights into improving LLM agents.

Any final thoughts on its implications?

Ashley: Evan.

The findings from this research pave the way for advancing LLM agents' adaptive capabilities in unfamiliar environments.

By focusing on text-based games with hidden rules, the paper offers a robust framework for evaluating how well agents truly learn from new experiences.

Evan: And that brings us to the end of today's episode.

We hope you found it insightful.

Thank you for joining us on Daily Paper Cast.

Ashley: We’ll be back with more deep dives into the latest AI research.

Don’t forget to tune in to our next episode for more exciting discussions.

Evan: Until then, keep exploring and learning!

Goodbye!