Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence

🤗 Upvotes: 28 | cs.AI

Authors:
Brian Wang, Bin Feng, Xiaoman Pan, Chenyang An, Felix Liu, Tangqi Fang, Gongbo Sun, Lingfeng Shen, Ning Wang, Handuo Zhang, Feng Chen, Fuchao Yang, Xiang Wang, Jiacheng Lin, Siting Li, Zixuan Liu, Chi Han, Zhenhailong Wang, Kunlun Zhu, Lawrence Zhao, Yueqi Guo, Kailong Wen, Feng Xing, Yiling Guo, Lidong Bing, David Tan, Bo An, Heng Ji, Sheng Wang

Title:
Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence

Arxiv:
http://arxiv.org/abs/2608.11341v1

Abstract:
Apollo did not reach the Moon merely because its engineers could solve difficult equations. It succeeded by turning a distant ambition into a mission architecture of explicit objectives, simulation, verification, and repeated correction. AI now faces a similar transition: frontier models can solve difficult tasks once the problem, tools, and success criteria are specified, yet consequential real-world challenges rarely arrive in an executable or verifiable form. We introduce Apodex Discovery, a framework for building and evaluating discoverative AI through the heavy-duty solver, a system comprising a foundation model, harness, tools, and control policies that pursues extended, stateful, verifiable investigations. It has three core components. First, a problem-scouting process surveyed 561 industries across 16 sectors, assembled 423 high-value real-world problems, and selected 20 for the initial release. Second, a common environment-task-episode abstraction provides data, tools, constraints, feedback, trajectory recording, and verification of intermediate artifacts and final submissions. Third, HDS6 evaluates Tools, Repair, Alternatives, Coherence, Evidence, and Scope independently of final-task success. In AAV capsid design, Apodex surpassed the published state of the art by 7% across viability, tropism, structure prediction, and generative design. In drug repurposing and reformulation, a task-specific biomedical environment improved the mean normalized prediction score of GPT-5.5 and GPT-5.6-sol by 2.5 and 7.6 points over the same closed-book backbone. Controlled ablations show that the fixed TRACES episode interface enables attribution of performance differences to specific solver components. Apodex Discovery moves AI evaluation beyond predefined benchmarks toward verifiable investigations aimed at genuine discovery.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Hello and welcome to Daily Paper Cast.

Ashley: Today’s paper comes from the Hugging Face daily paper list of August 18, 2026, and has received 28 upvotes.

Evan: The title of this paper is 'Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence.'

Ashley: This work is authored by Brian Wang, Bin Feng, and corresponds to Sheng Wang from Apodex.

Evan: Let's dive right into the paper, starting with the introduction.

Ashley: The Apollo program didn't succeed merely because its engineers could solve difficult equations.

It succeeded because a distant ambition was turned into a mission architecture consisting of explicit objectives, coordinated subsystems, simulation environments, telemetry, verification, and repeated correction.

In a similar vein, artificial intelligence is now at a significant transition point.

Evan: Advanced AI models are increasingly capable of solving difficult tasks once the problem, tools, and success criteria are defined.

However, the most consequential real-world challenges don't arrive packaged in a format ready for execution and verification.

Ashley: The aim that motivates this work is discovery—enabling AI systems to reach conclusions that are not yet known, rather than just reproducing answers that already exist.

Evan: The paper introduces Apodex Discovery, a framework designed to build and evaluate discoverative AI.

Discoverative AI involves making genuine discoveries and comprises a fully articulated heavy-duty solver.

This solver uses the provided reality benchmark named TRACES.

Ashley: Heavy-duty solvers are complete systems integrating a foundation model, harness, tools, and control policies aimed at conducting extended, stateful, and verifiable investigations.

Apodex Discovery makes three principal contributions: transforming open-ended real-world problems into verifiable discovery tasks.

Evan: First, they developed a systematic problem-scouting process that surveyed 561 industries, covering 16 sectors.

They compiled 423 high-value real-world problems and selected 20 for the initial release.

These tasks are split between retrospective tasks with hidden outcomes and prospective discovery challenges evaluated as new evidence emerges.

Ashley: Second, the paper formalizes executable environments through a common environment-task-episode abstraction.

This abstraction provides solvers with the necessary data, tools, constraints, and feedback, enabling them to act while recording trajectories and verifying key intermediate artifacts and final submissions.

Evan: This effort yields TRACES, the reality benchmark released along with this paper.

TRACES consists of executable environments, episodes, and tasks with hidden verifiers.

Ashley: Third, they introduce HDS6, a process-verification framework assessing six dimensions: Tools, Repair, Alternatives, Coherence, Evidence, and Scope.

This framework assesses tools and repairs independently from final-task success, enabling meaningful evaluations even when the definitive ground truth is delayed, incomplete, or unavailable.

Evan: For a practical case, in adeno-associated virus-or AAV-capsid design, Apodex surpassed the task-level published state of the art by 7% across all four tasks: viability prediction, tropism prediction, structure prediction, and generative design.

Ashley: In another example, in drug repurposing and reformulation, adding a task-specific biomedical environment improved the mean normalized prediction scores of GPT-5.5 and GPT-5.6-sol by 2.5 and 7.6 points, respectively, over the same closed-book backbone.

Evan: Controlled ablations further demonstrate that the fixed TRACES episode interface allows attributing performance differences to specific components in the solver system.

This goes beyond merely measuring aggregate performance.

Ashley: By leveraging real-world problems, executable environments, and process evaluation, Apodex Discovery establishes a new paradigm for discoverative AI.

This paradigm shifts the focus from solving predefined benchmarks towards conducting prolonged investigations through which discoveries are actually made.

Evan: That wraps up the introduction section of the paper.

Evan: Ashley, how do they convert these open-ended real-world problems into executable tasks for AI systems?

Ashley: Great question, Evan.

The authors of this paper break it down into several components to systematically scout high-value real-world problems and transform them into verifiable tasks that heavy-duty AI solvers can tackle.

Evan: I see.

So, how did they identify suitable problems for AI evaluation?

Ashley: First, they conducted a two-month survey by a team of researchers, all holding Ph.D.

degrees in STEM disciplines.

They surveyed 561 industries across 16 sectors, such as healthcare, engineering, finance, and more.

This effort produced an initial collection of 423 high-value real-world problems.

Evan: Pretty extensive.

561 industries is a huge number.

How did they decide which problems to select from those 423?

Ashley: Exactly.

They assessed each problem on multiple dimensions: reasoning depth, technical feasibility, verifiability, data and tool availability, expected verification latency, and potential real-world impact.

The resulting benchmark problems weren't just brainstormed lightly; they came from a robust process of domain analysis and expert review.

Evan: Interesting.

Once they had these problems, how did they move forward with the executable environments?

Ashley: To formalize these problems into executable environments, they used a structured environment-task-episode abstraction.

This abstraction bundles everything the solver needs: data, tools, constraints, and feedback, while also verifying intermediate and final outputs.

Evan: Can you explain what an episode is in this context?

Ashley: An episode is an individual instance of a task within an environment.

It includes its own data and hidden ground truth while exposing necessary inputs, tools, and budgets to the solver.

Each episode records the complete trajectory that the solver produces.

This setup ensures that the entire episode is verifiable and reproducible.

Evan: That makes the setup pretty clear.

But how do they ensure that these solvers are fairly evaluated across such diverse problems?

Ashley: Each environment has its verification mechanisms.

Verification is done in two complementary forms: outcome verification and process verification.

Outcome verification asks what the solver produced, scoring the final submission against the success criteria and hidden ground truth.

This method primarily checks the deliverable against measurable proxies or ground truth.

Evan: I assume process verification looks at how the answer was produced?

Ashley: Precisely.

Process verification reads the recorded trajectory to evaluate how the solver produced its result.

This is where HDS6, or the Heavy-Duty Solver Six Capabilities, comes in.

It assesses tools, repair, alternatives, coherence, evidence, and scope to gauge if the investigation was conducted reliably.

Evan: Interesting.

You mentioned earlier that the TRACES benchmark was released with this paper.

Can you elaborate on what TRACES entails?

Ashley: Yes, TRACES, or the reality benchmark, includes the executable environments, episodes, and tasks, all equipped with hidden verifiers.

These verifiers ensure that the solvers’ performance is attributable to their components and strategies rather than any variations in the environment.

Evan: And how are the six capabilities—Tools, Repair, Alternatives, Coherence, Evidence, and Scope—evaluated through HDS6?

Ashley: Each dimension is scored through specific sub-rubrics, resulting in a comprehensive evaluation profile.

For instance, the Tools dimension looks at whether the solver correctly selects and interprets external tools.

Repairs evaluate how well a solver responds to verification feedback or observed failure by rectifying underlying errors.

Evan: Fascinating.

What about the Alternatives and Coherence dimensions?

Ashley: Alternatives evaluate if the solver considers competing hypotheses and updates them as evidence accumulates.

Coherence checks if the solver maintains logical consistency over time, ensuring that intermediate steps do not silently contradict each other.

Evan: And Evidence and Scope?

Ashley: Evidence assesses whether every claim is grounded in observable data, tools, or references, distinguishing it from speculation.

Scope evaluates whether the solver correctly identifies the limits of its conclusions, recognizing the boundaries and conditions under which the results hold true.

Evan: Got it.

How do they integrate these verifications back into the solver?

Ashley: Both the outcome and process verifications feed back into the episode’s signals.

Outcome scores measure success but are sparse, often delayed.

Process scores provide more immediate evidence of whether the investigation was reliable.

This integration enables progressive solving by identifying strengths and weaknesses within each solver iteration.

Evan: It sounds like a pretty robust system.

Can you share a bit more about the adeno-associated virus capsid design and drug repurposing examples from the paper?

Ashley: Sure.

In adeno-associated virus capsid design, Apodex exceeded the published state of the art by 7% across all tasks.

For drug repurposing, adding a biomedical environment improved the normalized prediction scores of GPT-5.5 and GPT-5.6-sol significantly, by 2.5 and 7.6 points, respectively, over their closed-book baselines.

Evan: What role does the episode architecture play in these successes?

Ashley: The fixed episode architecture allows them to attribute any performance improvements or issues to specific solver components.

This detailed attribution helps isolate the impact of the foundation model, harness, initial skill guidance, and terminal verifier-guided repair.

Evan: That’s incredibly insightful, Ashley.

It looks like they’ve created a comprehensive framework to push the boundaries of what AI can achieve.

Ashley: That covers the Method section of the paper.

Evan: Alright, Ashley, I’m intrigued.

How did they test all these concepts in their experiments?

Ashley: They conducted several experiments targeting different high-value real-world problems.

Let's start with their work on Adeno-Associated Virus, or AAV, capsid design.

They structured this into four tasks: predicting viability, tropism, structure, and generative design.

Evan: What exactly were these tasks aiming to achieve?

Ashley: For viability, they predicted whether a capsid variant would remain functional.

Tropism involved predicting which tissues the capsid variant would target.

Structure prediction reconstructed the three-dimensional configuration of the capsid from its sequence.

Finally, generative design aimed to create novel sequences that are manufacturable and on target.

Evan: Got it.

So how effective was Apodex in these tasks?

Ashley: Quite effective.

For instance, in the viability task, Apodex surpassed the published state of the art by achieving an AUROC of 0.904 compared to the previous 0.878.

Similarly, in tropism prediction, the model scored 0.635, outperforming the existing Fit4Function method which scored 0.622.

Evan: And what about the structure and generative design tasks?

Ashley: For structure prediction, Apodex achieved a score of 0.649, outperforming AlphaFold 3, which scored 0.605.

In generative design, Apodex generated novel, viable, and on-target capsid sequences, surpassing specialist generative methods like AAVGen and AAVDiff, with a score of 0.180 compared to the previous best of 0.116.

Evan: That’s pretty impressive.

How did they perform in the drug repurposing and reformulation tasks?

Ashley: They demonstrated significant improvements here as well.

Adding a task-specific biomedical environment improved the mean normalized prediction score of GPT-5.5 and GPT-5.6-sol by 2.5 and 7.6 points over their closed-book backbones.

This environment allowed the solvers to integrate heterogeneous data sources and provide evidence-grounded predictions.

Evan: That’s fascinating!

Did they conduct ablation studies to understand which components contributed most to this performance?

Ashley: Indeed, they did.

Controlled ablations revealed the contribution of each solver component to the overall performance.

For example, by fixing the TRACES episode interface, they were able to attribute performance differences to specific elements such as the foundation model, harness, initial skill guidance, and verifier-guided repair.

Evan: So, what were the highlights from those ablations?

Ashley: One of the key findings was that domain-specific environments consistently enhanced model performance across various tasks.

For instance, in the AAV structure prediction task, the domain-specific environment significantly boosted the model’s performance compared to using a generic harness.

Evan: And how did HDS6 factor into their results?

Ashley: HDS6 provided deeper insights into the process quality behind the results.

For instance, models that performed better in the AAV structure prediction task also showcased higher process scores in terms of coherent hypothesis formation, effective data and tool use, and strong self-correction mechanisms.

Evan: It sounds like HDS6 allowed them to measure more than just the final outcome.

Ashley: Exactly.

It offered a finer-grained and immediately available evaluation of whether the investigation was conducted reliably, even when the definitive ground truth might be delayed or incomplete.

Evan: Were there any other notable results from their experiments?

Ashley: Yes, the framework also enabled significant performance attribution.

By leveraging controlled ablations and fixed episode interfaces, they could pinpoint specific components of the solver systems—highlighting whether the model, harness, agent loop, tool use, or feedback handling contributed most to task success.

Evan: It’s fascinating how detailed their analysis and insights are.

Anything else noteworthy from the results?

Ashley: Overall, their experimentation showcases that the Apodex Discovery framework is robust, enabling not just higher performance but more reliable and self-correcting AI models across diverse and high-value real-world tasks.

Evan: That’s a wrap for the Experiment section.

Evan: Ashley, how does Apodex Discovery build upon previous work in the field of AI and machine learning research?

Ashley: That’s a great question, Evan.

The paper dives into extensive related work, connecting Apodex Discovery to a wide array of prior research and methodologies in AI and machine learning.

Evan: Alright, let's break it down.

What’s the first category of related work they discuss?

Ashley: The first category they address is knowledge and reasoning benchmarks.

Traditional benchmarks such as MMLU, BIG-bench, AI2 Reasoning Challenge, and GPQA primarily assess a model’s ability to map a prompt to the correct answer.

These benchmarks focus on static question-answering capabilities without persistent environments or the opportunity for real-time interaction.

Evan: I see.

What about interactive and agentic benchmarks?

Ashley: Interactive benchmarks have been used for a while, particularly in reinforcement learning with platforms like OpenAI Gym.

More recently, we've seen benchmarks like WebShop, Mind2Web, and InterCode, which embed models in executable environments.

These benchmarks are moving towards more realistic problem-solving but often remain limited to narrowly specified tasks.

Evan: And how does Apodex Discovery differentiate itself from these existing benchmarks?

Ashley: Apodex Discovery differentiates itself by providing a fully-integrated system that includes reality-based environments, episodic tasks, and both process and outcome verification.

Unlike traditional benchmarks, it evaluates the full spectrum of discoverative AI capabilities, from problem formulation and tool use to hypothesis management and process integrity.

Evan: Alright.

How does the paper integrate agent scaffolds into this framework?

Ashley: They discuss various general prompting and control strategies like ReAct, Tree of Thoughts, Reflexion, and Self-Refine.

These strategies are crucial for turning a model into an agent capable of complex investigations.

Apodex Discovery evaluates the entire scaffold, including the foundation model and its harness, rather than just the model in isolation.

Evan: How about automated and LLM-based evaluation?

How does Apodex relate to that?

Ashley: Automated evaluation has become a staple in large language model assessment.

Methods like pairwise judging, arena-style judging, and reference-free scoring with tools like G-Eval are becoming standard.

Apodex leverages these approaches but extends them through HDS6 to assess how well an AI system maintains long-term process reliability and evidence-grounded investigation.

Evan: Interesting.

Any mention of process supervision and verifiable rewards?

Ashley: Yes, the paper takes inspiration from process supervision strategies that score the reasoning process rather than just the end result.

This includes work like Solving Math Word Problems with Process- and Outcome-Based Feedback and reinforcement learning models like DeepSeek-R1 that rely on verifiable rewards.

Apodex expands this by evaluating multi-step, tool-using trajectories through a six-dimensional process-verification metric.

Evan: So, they’re bringing together a lot of different threads from existing research.

Ashley: Exactly.

They also address specification gaming and contamination, which are long-standing concerns in AI.

Ensuring that benchmarks are free from shortcuts and data leaks is a critical part of making sure the models are genuinely solving the tasks as intended.

Evan: And what about the current landscape of autonomous scientific discovery?

How does Apodex fit in there?

Ashley: They cite several autonomous scientific discovery systems like Coscientist, ChemCrow, and the AI co-scientist framework.

These systems automate aspects of scientific research, from hypothesis generation to experimental execution.

Apodex Discovery builds on this by providing a framework where such autonomous systems can be rigorously evaluated across diverse scientific and engineering challenges.

Evan: It's impressive to see how Apodex Discovery isn't just a standalone innovation but a comprehensive integration of multiple advances in AI and machine learning.

Ashley: By combining these advancements into a single framework, Apodex Discovery sets a new standard for evaluating and building discoverative AI.

Evan: That wraps up the Related Work section.

Evan: Ashley, let's summarize the key contributions and takeaways from this fascinating paper.

Ashley: Sure, Evan.

The major contribution of the paper is the introduction of the Apodex Discovery framework.

This framework systematically transforms open-ended real-world problems into verifiable discovery tasks by scouting high-value problems across various industries.

Evan: And through this framework, they've turned these problems into executable environments using a structured environment-task-episode abstraction.

Ashley: Right.

Each environment provides all the necessary data, tools, constraints, and feedback required for the solver, recording the complete trajectory for thorough process and outcome verification.

Evan: Their verification methods, including HDS6, provide a comprehensive evaluation of how well the AI systems conduct reliable and self-correcting investigations.

Ashley: Exactly.

Their experiments demonstrated significant advances in tasks like AAV capsid design and drug repurposing, showing that the Apodex Discovery framework helps achieve higher performance levels while also allowing detailed performance attribution to specific components of the solvers.

Evan: By leveraging real-world problems, executable environments, and robust process evaluation, Apodex Discovery essentially shifts the paradigm from solving predefined benchmarks to conducting meaningful, consequential investigations.

Ashley: Indeed, it's an exciting step forward for the field of AI research.

That wraps up our focus on this paper.

Evan: Thank you all for listening to today’s episode of Daily Paper Cast.

We hope you found the discussion insightful.

Ashley: Make sure to tune in tomorrow for more cutting-edge research.

Have a great day!

Evan: Until next time, this is Evan and Ashley signing off.