WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data

🤗 Upvotes: 24 | cs.CL

Authors:
Ji Soo Lee, Xilun Chen, Pierce Chuang, Ashish Shenoy, Jason Wei, Dohwan Ko, Hyunwoo J. Kim, Benoit Corda

Title:
WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data

Arxiv:
http://arxiv.org/abs/2609.05405v1

Abstract:
Recent advances in wearable sensing enable continuous monitoring of physiological and behavioral signals, yet existing benchmarks rarely evaluate whether AI systems can reason over a real user's longitudinal wearable record. We introduce WearableQA, a benchmark comprising 4,084 10-option multiple-choice questions constructed from the wearable time series, blood biomarkers, and demographics of 200 real users, each with up to 500 days of daily measurements. WearableQA preserves authentic wearable distributions that include device noise and inter-individual variability. To evaluate distinct reasoning capabilities, we introduce 16 question types organized along two complementary axes: data versus health reasoning, which distinguishes computation over longitudinal measurements from physiological interpretation; and single- versus cross-signal reasoning, which separates reasoning about individual signals from the integration of multiple signals. To construct reliable questions at scale, we adopt a dual-grounding framework that combines literature-grounded physiological findings with statistically validated population-grounded physiological patterns. This enables the capture of meaningful relationships observed in real-world wearable data. Evaluation of 14 proprietary and open-source LLMs demonstrates that WearableQA effectively differentiates model capabilities, with performance ranging from 19.6% to 72.9% against a 10% chance baseline. Moreover, WearableQA remains far from solved: most models achieve accuracies below 60%. Overall, WearableQA provides a realistic and diagnostic benchmark for evaluating LLM reasoning over real-world wearable data.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast, your source for cutting-edge research papers in AI, NLP, CV, and related fields.

Ashley: Today's paper comes from the Hugging Face daily paper list of September 10, 2026, and has received 24 upvotes.

Evan: The paper we’re diving into is titled, 'WearableQA: A Benchmark for Health Reasoning Over Real-World Wearable Data'.

It’s authored by Ji Soo Lee from KAIST, Xilun Chen from Meta, and the corresponding author is Ji Soo Lee from KAIST.

Ashley: Let’s get into the Introduction section.

Recent advances in wearable sensing technologies have made it possible to continuously monitor physiological and behavioral signals.

These signals include heart rate, sleep patterns, physical activity, and heart rate variability.

Evan: These developments, combined with the rapid progress in large language models or LLMs, are now being used to support personalized health understanding, lifestyle assessments, and health monitoring.

Ashley: Exactly.

However, reasoning with wearable data requires more than just recalling general health knowledge.

It involves identifying and computing information from often noisy longitudinal physiological measurements and then interpreting it in a health context that can be user-specific.

Evan: But how well do current LLMs perform in this regard?

Can they effectively reason over a user's longitudinal wearable data?

Ashley: Good question.

The paper highlights that this area is still underexplored.

While there have been some recent studies, systematic benchmarks to evaluate this capability are scarce.

Existing benchmarks often rely on synthetic or simulated signals instead of real-world wearable measurements.

Evan: So, it seems there's a gap in assessing how well these models can reason about real-world wearable data over time.

Ashley: Precisely.

To address this gap, the authors introduced WearableQA.

It is a benchmark specifically designed for data and health reasoning on real-user longitudinal wearable records.

Evan: Interesting.

What does WearableQA include?

Ashley: WearableQA comprises 4,084 multiple-choice questions constructed from 200 real users' wearable measurements taken over hundreds of days per individual.

These measurements are authentic and include complementary blood biomarkers and demographic information.

Evan: And what does this benchmark aim to evaluate?

Ashley: WearableQA is structured to evaluate model capabilities along two main axes: reasoning type and signal complexity.

Reasoning types include data reasoning, which involves computing trends, anomalies, and relationships from raw wearable measurements, and health reasoning, which requires clinical and physiological interpretation.

Evan: And signal complexity?

Ashley: Signal complexity distinguishes between questions about a single signal and those requiring the integration of multiple signals.

For example, combining physical activity data with heart rate readings to form a comprehensive understanding.

Evan: It sounds like this benchmark can give a detailed assessment of where models struggle.

Ashley: Exactly.

The paper notes that constructing reliable questions from longitudinal wearable data at scale is challenging.

The questions need to reflect genuine physiological patterns and be annotated deterministically according to those patterns.

Evan: How did the authors approach this challenge?

Ashley: They adopted a dual-grounding framework, combining peer-reviewed literature-grounded physiological findings with statistically validated population-grounded patterns.

This ensures that the questions capture meaningful relationships observed in real-world wearable data.

Evan: And how effective is WearableQA in differentiating model capabilities?

Ashley: The paper reports the evaluation of 14 proprietary and open-source language models using WearableQA.

The models' performance ranged from 19.6% to 72.9% accuracy against a baseline accuracy of 10%.

Interestingly, even the best models achieved accuracies below 60% on most questions, indicating that WearableQA is a rigorous and challenging benchmark.

Evan: That sets a high bar for future research.

Clearly, there's a lot more to be done in improving the reasoning capabilities of LLMs over real-world wearable data.

Ashley: WearableQA provides a realistic and diagnostic benchmark for evaluating these capabilities, and it's clear there's room for improvement.

Evan: That wraps up the Introduction section of the paper.

Moving forward, we'll delve into the methods and experimental setup used in this study.

Evan: Alright, Ashley, let's dive into the methods used in this study.

How did the authors construct the WearableQA benchmark?

Ashley: Great question, Evan.

The authors began by compiling longitudinal wearable and blood-panel records from 200 real users.

These users were sampled from a larger cohort of in-the-wild wearable time-series data ensuring high coverage across different demographic characteristics.

Evan: What kinds of signals and metrics were these users contributing?

Ashley: Each user contributed a multivariate time series comprising 16 daily wearable metrics.

These metrics spanned four main physiological domains: cardio-fitness, activity and energy, sleep, and stress levels.

Specific metrics included resting heart rate, heart rate variability, VO2 max, steps, active calorie burn, basal metabolic rate calories, exercise count and duration, as well as sleep duration, efficiency, and stages like deep and REM sleep.

Evan: And this data collection took place over quite a period, right?

Ashley: Yes, Evan.

Each instance in WearableQA is grounded in real-user measurements taken over hundreds of days.

Specifically, each user provided up to 500 days of daily aggregated measurements.

Additionally, each user's profile included a 17-biomarker blood panel and demographic attributes such as age, sex, BMI, and ethnicity.

Evan: That’s a comprehensive dataset.

So, how did they structure this benchmark to evaluate model capabilities?

Ashley: WearableQA is structured along two primary axes for evaluating model capabilities: reasoning type and signal complexity.

The reasoning type axis includes data reasoning, which involves computing trends, anomalies, and relationships from the raw wearable measurements, and health reasoning, which requires clinical and physiological interpretation.

Evan: And what about the signal complexity axis?

Ashley: The signal complexity axis distinguishes between single-signal and cross-signal questions.

Single-signal questions focus on one signal in isolation, while cross-signal questions require integrating two or more signals, such as correlating physical activity with resting heart rate.

Evan: Okay, so they’ve set up a thorough framework for evaluation.

How did they ensure that the questions were reliable and meaningful?

Ashley: To construct reliable questions, the authors adopted a dual-grounding framework.

This approach combines literature-grounded physiological findings with statistically validated population-grounded patterns.

Peer-reviewed studies provided clinically validated relationships among wearable signals, physiological biomarkers, and health outcomes.

Evan: So, they ensured the questions were anchored in verified scientific evidence.

What about the population-grounded patterns?

Ashley: For population-grounded patterns, they mined large-scale wearable data directly from the cohort.

This included identifying temporal trends, anomalous episodes, recovery dynamics, and multi-signal interactions over a 28-day window.

Importantly, these patterns were rigorously validated for statistical consistency and robustness across the cohort.

Evan: And once they identified these patterns, how did they incorporate them into the questions?

Ashley: Each reasoning objective, whether derived from literature or population patterns, was instantiated on individual user trajectories through deterministic computations.

This means that the model had to literally compute the answers from the users’ data, rather than relying on pre-defined answers.

Evan: That sounds quite robust.

What about the question types in WearableQA?

Ashley: WearableQA comprises 16 question types, evenly divided between data reasoning and health reasoning.

For example, data-reasoning questions include identifying trends, anomalies, signal summaries, and computing recovery times.

Health-reasoning questions range from prognostic predictions to differential diagnoses and health recommendations.

Evan: How does one ensure distractors or wrong options were appropriately challenging for these models?

Ashley: Good point.

For data reasoning, distractors often represented incorrect trend directions, adjacent time windows, or cohort-based incorrect magnitudes.

For health reasoning, distractors were clinically plausible but inconsistent with the individual’s measurements.

This systematic approach ensured that the wrong options were non-trivial.

Evan: Every question sounds meticulously constructed.

What about ensuring quality and avoiding shortcuts?

Ashley: To ensure computational validity, the authors designed the benchmark to allow every answer to be reproducible from the underlying wearable measurements.

They applied rigorous quality control and shortcut auditing loops.

For instance, instances of predictable metric pairs were identified and handled to prevent models from answering based on prior knowledge alone without examining the data.

Evan: Certainly sounds thorough.

Did they also involve human verification?

Ashley: The questions were reviewed by three different models and subsequently validated by human reviewers.

This multi-step process ensured that the questions were both diverse and free from unintended shortcuts.

Evan: And finally, let’s touch on the models they evaluated using WearableQA.

How broad was their evaluation?

Ashley: They evaluated a total of 14 models, both proprietary and open-source.

Proprietary models included several versions from OpenAI, Google, and Anthropic.

Open-source models came from diverse sources including Meta and Google.

They ensured each model was evaluated under the same protocol to maintain consistent comparisons.

Evan: Very comprehensive evaluation.

Well, that’s the end of the Method section.

Next, we’ll explore the experiments and the results they obtained with these models on WearableQA.

Evan: Ashley, now that we’ve covered the methodology, let’s dive into the experiments and results.

How did the authors set up their experiments?

Ashley: Sure, Evan.

The authors evaluated a diverse set of 14 instruction-tuned large language models, both proprietary and open-source, to analyze their reasoning capabilities over the WearableQA benchmark.

This included models like OpenAI's GPT-5.4, Google’s Gemini series, and Anthropic’s Claude models, among others.

Evan: How did they ensure consistent comparison across these models?

Ashley: Each model was evaluated under the same protocol.

The models were presented with questions along with associated context, which included wearable measurements from the past 500 days, demographic information, blood biomarkers when applicable, and cohort reference statistics.

The accuracy of each model was determined by comparing their responses to a deterministic ground truth.

Evan: And how did the models perform?

Ashley: The performance varied widely, with accuracy ranging from 19.6% to 72.9%.

Notably, no model consistently achieved over 60% accuracy on most questions, underlining the challenge presented by WearableQA.

Evan: Which model performed the best overall?

Ashley: The best overall performer was Google’s Gemini-3.1 Pro, with an accuracy of 72.9%.

This was followed by Claude Opus 4.6, which achieved an accuracy of 60.2%.

Evan: What about the open-source models?

Ashley: Among open-source models, the top performer was Gemma-4-26B-A4B, which achieved an accuracy of 42.5%, outperforming some of the proprietary models like GPT-4o.

Evan: Interesting.

Let’s break down the performance a bit.

How did the models fare in data reasoning versus health reasoning tasks?

Ashley: The models generally found data reasoning more challenging than health reasoning.

For example, GPT-4o achieved 25.3% accuracy in data reasoning compared to 53.5% in health reasoning.

Similarly, the gap was noticeable for other models as well, indicating that deriving answers directly from raw measurements is particularly tough.

Evan: What about single-signal versus cross-signal questions?

Ashley: Cross-signal reasoning posed a significant challenge across all models.

Proprietary models like GPT-5.4 saw a performance drop from 59.1% on single-signal questions to 45.7% on cross-signal questions.

Similar trends were observed across other models, highlighting the complexity of integrating multiple signals.

Evan: Were there differences in how well the models handled different types of questions?

Ashley: Yes, indeed.

For data reasoning questions, tasks like identifying the strongest pair relationships and recovery times were particularly challenging.

On the other hand, signal summaries and excursion counts were relatively easier.

In health reasoning, while most tasks were found to be easier, cross-signal prediction and some prognostic tasks still posed difficulties.

Evan: How did the models perform with chain-of-thought versus direct-answer prompting?

Ashley: Chain-of-thought prompting generally improved accuracy.

For instance, Claude Opus 4.6 saw a jump from 40.6% in the direct answer mode to 60.2% with chain-of-thought prompting.

This indicates that explicit reasoning improves performance, though the gains were more significant in some models compared to others.

Evan: Were there any notable biases or tendencies observed in the models' responses?

Ashley: Yes, positional bias was noted, especially in smaller models.

For example, Llama-3.2-3B under direct-answer mode had a strong bias towards choosing a specific option.

This bias was mitigated when chain-of-thought prompting was used.

Evan: It sounds like there’s a lot of room for improving how these models process and reason over wearable data.

Ashley: Yes, and the authors noted that direct computation from longitudinal wearable data remains a challenging task.

They also observed that providing the models with more explicit summaries or visual cues did not significantly improve their performance.

Evan: It seems like WearableQA really pushes the boundaries in evaluating these models.

That wraps up our discussion on the Experiment and Results section.

Evan: Ashley, let's move on to the Related Work section.

How does WearableQA build on or differentiate from existing benchmarks?

Ashley: Great question, Evan.

The authors highlight that there has been a growing body of work evaluating language models on medical and health reasoning tasks.

Benchmarks like MedQA, MedMCQA, and PubMedQA primarily assess medical knowledge through examination-style questions derived from biomedical text.

Evan: But how do these benchmarks differ from WearableQA?

Ashley: The key difference is that these benchmarks are largely text-based and focus on static questions or cases.

They provide limited insight into whether models can reason over an individual’s longitudinal physiological history, which is essential for personalized health assessments.

Evan: Were there any benchmarks that incorporated physiological measurements?

Ashley: Yes, one such benchmark is ECG-QA.

It moves beyond purely textual inputs by incorporating physiological measurements through ECG question answering.

However, reasoning over an ECG is distinct from reasoning over longitudinal wearable measurements, where models must aggregate information across time and integrate various behavioral and physiological signals.

Evan: So, ECG-QA laid some groundwork, but it doesn't fully address the unique challenges presented by wearable data.

Ashley: Exactly.

More recently, several studies have started to evaluate LLM reasoning over wearable measurements.

For example, PHIA explores questions over wearable data, but its objective evaluation is based primarily on simulated user trajectories and focuses on retrieval and aggregation over the measurements.

Evan: And what about other existing wearable evaluations?

Do they jointly incorporate other health information like blood biomarkers?

Ashley: Most existing evaluations focus solely on wearable signals without integrating complementary health information like blood biomarkers.

WearableQA extends this by evaluating reasoning over real-user longitudinal wearable data combined with blood biomarkers and demographic information.

Evan: It seems like WearableQA aims to capture a more comprehensive picture of an individual's health context.

Ashley: Definitely.

Additionally, WearableQA addresses the complexity of real-world longitudinal wearable measurements, something that general time-series reasoning benchmarks often miss.

For instance, TimeSeriesExam evaluates general time-series understanding using synthetic signals for tasks like pattern recognition and anomaly detection but lacks the health-specific context.

Evan: So, TimeSeriesExam deals with sequential data but doesn't delve into the health interpretation, which is crucial for WearableQA.

Ashley: Exactly.

Unlike generic time-series benchmarks, WearableQA targets reasoning over actual longitudinal wearable measurements combined with health-related interpretation.

It systematically evaluates models across a diagnostic taxonomy of 16 question types with ground truth derived from peer-reviewed literature and statistically validated population patterns.

This makes WearableQA unique in the field.

Evan: How about the general trends in AI-driven health assistants?

How is WearableQA positioned within this context?

Ashley: AI-driven health assistants are increasingly emerging, leveraging large language models to provide personalized health insights.

However, current benchmarks don't fully capture the challenges these models face when interpreting continuous, real-time health data.

WearableQA fills this gap by offering a rigorous testbed to evaluate their performance in real-world settings.

Evan: So, in summary, WearableQA stands out by not only evaluating data reasoning but also providing a realistic framework for health reasoning, utilizing both wearable data and complementary health metrics.

Ashley: That's right.

WearableQA is designed to push the boundaries of what LLMs can achieve in the realm of personalized health monitoring and reasoning.

It bridges the gap by integrating real-world data, clinically validated patterns, and comprehensive reasoning tasks.

Evan: That wraps up the Related Work section.

In the next part, we'll delve into the paper's conclusions and future directions.

Evan: Ashley, let’s wrap up with a summary of the key contributions and takeaways from this paper.

What are the standout contributions?

Ashley: Sure, Evan.

The primary contribution of this paper is the introduction of WearableQA, a comprehensive benchmark specifically designed for evaluating LLMs’ data and health reasoning over real-world wearable data.

This is a step forward in assessing AI's ability to interpret longitudinal health data.

Evan: Right, and they constructed this benchmark with real measurements from 200 users, including wearable signals, blood biomarkers, and demographic data, which adds a layer of authenticity and complexity.

Ashley: Exactly.

Another key aspect is their dual-grounding framework.

It combines literature-grounded physiological findings with statistically validated population-grounded patterns, ensuring that the questions are both meaningful and grounded in real-world data.

Evan: And for the evaluation, they put 14 different LLMs through the benchmark, noting considerable variance in performance and highlighting the challenges in cross-signal reasoning and data interpretation.

Ashley: That's right.

One of the big takeaways is that while current LLMs show promise, especially for health reasoning tasks, there’s still a significant gap in their ability to process and reason over raw, noisy wearable data longitudinally.

Evan: In summary, WearableQA pushes the boundaries for AI in personalized health monitoring, setting a rigorous standard for future advancements.

And that brings us to the end of our discussion on this paper.

Thank you for tuning in to Daily Paper Cast.

Ashley: We hope you found today's episode insightful.

Join us in our next episode where we continue to explore the latest research papers in AI, NLP, CV, and more.

Evan: Don’t forget to subscribe and follow us for more daily insights.

See you next time!