Subscribe
Share
Share
Embed
🤗 Upvotes: 37 | cs.CL, cs.AI, cs.LG
Authors:
Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos
Title:
Language Models Can Control Their Own Attention
Arxiv:
http://arxiv.org/abs/2609.02737v1
Abstract:
Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? To this end, we introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes:
We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: dailypapercast.ai@gmail.com
Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com
Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236
Cover Image by Kawen Kuang https://kawen.art
Evan: Welcome to Daily Paper Cast.
Ashley: Today’s paper is from the Hugging Face daily paper list of September 3, 2026, and it has received 37 upvotes.
Evan: The paper we’re discussing today is titled: 'Language Models Can Control Their Own Attention.'
Ashley: The first two authors are Namgyu Ho and Huzama Ahmad, with Se-Young Yun as the corresponding author.
They are affiliated with KAIST AI.
Evan: Alright, let’s dive into the Introduction section of the paper.
Ashley: Transformers, the architecture behind many state-of-the-art language models, calculate attention over all preceding tokens at every decoding step.
This design, while powerful, becomes computationally expensive as the length of the context grows.
Evan: Exactly.
At the core of this complexity is the Key-Value, or KV, cache memory access latency, which significantly impacts the decoding time for long contexts.
Ashley: To give you a perspective, in a model like Qwen-3.5-397B-A17B, handling a conversation with a 1-million-token context could require loading roughly 15 gigabytes of KV cache per sequence at every decoding step.
This demand is comparable to loading the model’s 17 billion active parameters.
Evan: Wow, and that’s just for one sequence.
So, how do humans approach similar tasks?
Ashley: Humans typically don't re-read every prior word when processing long texts.
Instead, we focus on relevant sections.
This intuition aligns with empirical findings—attention scores usually concentrate on a small fraction of the context tokens.
Evan: So, if attention naturally focuses on smaller, critical parts of the context, why is it challenging for models to do the same?
Ashley: The core challenge lies in the fact that true attention scores are unknown until the entire attention matrix is computed, making it infeasible to identify important tokens dynamically.
Previous methods have attempted to predict and mask attention-heavy tokens based on static heuristics and lightweight scans, but these solutions still struggle with efficiency when scaling to long contexts.
Evan: That brings us to this paper’s approach.
What exactly do the authors propose?
Ashley: They introduce an innovative solution called Declarative Attention, or DA.
Instead of externally scoring the relevance of tokens, DA allows the model to explicitly declare where it needs to focus its attention during its chain of thought.
Essentially, they propose that the model should narrate its attention strategy as part of the text generation process.
Evan: Interesting.
So, how does Declarative Attention work?
Ashley: The DA protocol segments the generation process into three distinct attention modes: global, focus, and local.
Evan: Could you break down these modes for us?
Ashley: Certainly.
In global mode, the model surveys the entire context to identify relevant segments.
In focus mode, the model zeroes in on specific, pre-named contextual regions.
Finally, in local mode, it focuses solely on recent output, ignoring the broader context.
Evan: Got it.
So, by switching between these modes, the model avoids reading the entire KV cache at every step?
Ashley: Exactly.
This selective attention approach dynamically constructs the attention mask based on declarations present in the model’s output.
By parsing these declarations, the inference engine can skip reading most of the KV cache, leading to significant efficiency gains.
Evan: And how well does this approach work in practice?
Ashley: Under zero-shot evaluations, DA reduced the total attended tokens during decoding by 52% on the Gemma-4-31B model and 31% on the Qwen-3.6-27B model.
The accuracy drop was modest, with only a 1.27 percentage points reduction in accuracy for Gemma-4-31B and 2.75 percentage points for Qwen-3.6-27B, and the gap narrows as the models scale.
Evan: That's quite impressive, reducing the attended tokens significantly while maintaining accuracy.
Ashley: It is.
Their results show a promising new axis of sparse attention that doesn’t require auxiliary scorers or model retraining, making it applicable to off-the-shelf models.
Evan: That wraps up our coverage of the Introduction section of this paper.
Ashley: Let's now move on to the Method section detailing Declarative Attention, or DA.
Evan: Ashley, how does DA fundamentally restructure the model’s attention mechanism?
Ashley: The DA protocol fundamentally changes the way a language model handles its attention spans during the text generation process by making the attention plan legible.
Essentially, it has the model declare its attention mode at each step.
Evan: Right, and you mentioned three attention modes earlier: global, focus, and local.
How does the model decide which mode to use?
Ashley: Each mode is chosen based on the model's current reasoning needs.
The model uses special tags to signal these modes as it generates text.
Let's break down how these modes are structured and how they guide the attention process.
Evan: Let's dive into each of these modes.
Ashley: Sure.
We'll start with global mode.
In global mode, the model can access the entire context.
This mode is used for navigation - when the model needs to identify which part of the context is relevant to the current task.
Evan: So, essentially, global mode is for scanning the broader context to zero in on specific information?
Ashley: Exactly.
Once the model identifies a relevant section, it switches to focus mode.
Evan: And what happens in focus mode?
Ashley: In focus mode, the model restricts its attention to specific, named segments of the context.
These segments contain the pertinent information needed for the next steps of reasoning.
The model uses tags to mark these segments.
Evan: Got it.
So, in focus mode, the model hones in on a targeted piece of information, right?
Ashley: Exactly.
Once the necessary information has been extracted, the model can then use local mode.
Evan: And what’s the role of local mode?
Ashley: Local mode is used for self-contained reasoning based on the information accumulated so far.
Here, the model only refers to its own recent outputs, ignoring the broader context.
Evan: So, it’s like the model is summarizing or synthesizing the gathered information without looking back into the larger context?
Ashley: That’s correct.
This mode helps to reduce computational overhead by avoiding unnecessary context reads.
Evan: Interesting.
How is the context segmented to make this process efficient?
Ashley: The context is divided into addressable segments called magic chunks.
Each segment is roughly 2048 tokens long, ensuring each chunk is a manageable size and aligns with natural language boundaries like sentences or paragraphs.
Evan: Why use the term 'magic chunks'?
Ashley: The name was likely chosen to emphasize that these segments are arbitrary retrieval units and don’t necessarily align with the document's original sections or chapters.
Evan: And how does the model interact with these magic chunks during the different modes?
Ashley: Great question.
During focus mode, the model references specific magic chunks by marking them with tags.
This signals the inference engine to only consider those chunks during the attention process, ignoring everything else.
Evan: Efficient.
And the inference engine dynamically constructs the attention mask based on these tags?
Ashley: Exactly.
The inference engine parses the tags emitted by the model and updates the attention mask in real-time.
This minimizes the number of tokens read from the KV cache during each decoding step.
Evan: So this approach eliminates the need for external scoring mechanisms that were used in previous methods?
Ashley: Yes, the DA protocol doesn’t rely on auxiliary scorers or any extrinsic selection methods.
It leverages the model’s own reasoning capabilities to handle attention efficiently.
Evan: What about implementing this on existing models?
Is training required?
Ashley: No training is required.
DA works zero-shot on off-the-shelf models, meaning it can be applied directly without the need for task-specific fine-tuning.
Evan: That’s highly practical.
Are there any results on how effective this method is?
Ashley: Indeed.
The authors evaluated DA on a range of long-context tasks using models Gemma-4-31B and Qwen-3.6-27B.
The results demonstrated substantial reductions in the number of attended tokens.
Evan: Can you provide some specifics on the evaluations?
Ashley: Of course.
On average, DA reduced the attended tokens by 52% for the Gemma-4-31B model and 31.1% for the Qwen-3.6-27B model.
These savings came with only modest accuracy drops, showcasing the efficiency of DA.
Evan: That’s impressive.
How do these results vary with model scale?
Ashley: The accuracy drop tends to decrease as the model scales up.
For instance, the accuracy drop for smaller models was slightly higher compared to larger models.
This indicates that the DA protocol leverages the increased capability of larger models more effectively.
Evan: It definitely sounds like a promising direction.
Before we wrap up the Method section, how does this protocol integrate with existing inference systems?
Ashley: The authors integrated DA into vLLM to utilize a block-aligned, in-place KV cache masking that is compatible with FlashAttention.
This implementation projects that DA’s attention savings could reduce total decode wall-clock costs to 71% of the vanilla setup on the Gemma-4-31B model and to 77% on the Qwen-3.6-27B model.
Evan: So to sum up, DA provides a protocol for models to declare their attention focus in real-time, significantly cutting down the attention costs without compromising much on accuracy.
And it works out-of-the-box with no additional training required.
Ashley: Precisely.
That concludes our discussion on the Method section.
Evan: Let's continue by diving into the Experiment and Results section of the paper.
Ashley: The authors conducted a thorough evaluation of Declarative Attention, or DA, across various models and tasks to measure its efficacy.
Evan: Right.
They used six models from two different families: Gemma-4 and Qwen-3.5/6.
The main comparison focused on the largest models from these families, namely Gemma-4-31B and Qwen-3.6-27B, with a broader analysis covering all six models.
Ashley: Exactly.
The evaluation was conducted on 15 long-context sources divided into two task categories: single-span retrieval and reasoning, and multi-span reasoning.
These sources ranged from textbooks and novels to code repositories and conversation transcripts.
Evan: So, what kind of benchmarks did they use for these tasks?
Ashley: They drew from several long-context benchmarks including RULER, LongBench v1 and v2, LooGLE, and ZeroScrolls.
The contexts varied significantly in length, with the longest reaching up to 1 million tokens.
Evan: That’s a wide range.
What were the key metrics they focused on for evaluation?
Ashley: They used two main metrics: accuracy and attended tokens.
Accuracy measured how often the model's answers were correct, while attended tokens counted the total number of tokens the model attended to during decoding.
They also estimated real-world costs using roofline wall-time analysis.
Evan: Let’s get into the results then.
How did DA perform overall?
Ashley: DA produced impressive results.
On average, it reduced the attended-token count per response by 52% on Gemma-4-31B and 31.1% on Qwen-3.6-27B, with accuracy drops of only 1.27 and 2.75 percentage points, respectively.
Evan: That’s quite a reduction in computational load.
Did the performance vary significantly across different tasks?
Ashley: Yes.
The reduction in attended tokens was more consistent across single-span tasks compared to multi-span reasoning tasks.
For instance, in the single-span tasks, DA showed an average token reduction of 54.8% on Gemma and 29.7% on Qwen.
Evan: And in multi-span tasks?
Ashley: In multi-span tasks, the average token reduction was slightly lower—48.5% on Gemma and 32.6% on Qwen.
This is attributed to the complex nature of multi-span reasoning, which requires navigating through various segments of the context more frequently.
Evan: Interesting.
And how about the accuracy—were there any significant gains or losses depending on the task type?
Ashley: Accuracy varied by task as well.
DA consistently matched or even improved upon the vanilla setup in certain tasks, with clear single-task gains observed in areas like long-dependency QA and code repository QA.
Evan: What about the impact of model size on DA's performance?
Ashley: DA scaled positively with model size.
Larger models like Gemma-4-31B and Qwen-3.6-27B showed smaller accuracy drops, indicating that more capable models could leverage the DA protocol more effectively.
For instance, the accuracy gap between DA and vanilla narrowed as the model size increased from 4 billion to 31 billion parameters.
Evan: That’s encouraging.
Did the authors explore any specific integration details with existing systems?
Ashley: Yes, they integrated DA into the vLLM framework.
By adopting block-aligned, in-place KV cache masking, they projected significant reductions in decode wall-clock costs.
For instance, they estimated decode time could be reduced to about 71% of the vanilla configuration on Gemma-4-31B and to 77% on Qwen-3.6-27B.
Evan: That’s quite practical.
Any notable failure modes or areas where DA struggled?
Ashley: The authors noted some failure modes.
In cases where the context required global arithmetic or tallying over large spans, DA struggled due to the segmentation of information.
Additionally, tasks that inherently generated long output per segment also saw less dramatic token savings.
Evan: So, while there are areas for improvement, DA seems to offer a substantial leap in computational efficiency.
Ashley: The DA protocol opens up new avenues for more efficient, context-aware language model usage, with significant real-world implications for deployments in resource-limited settings.
Evan: That brings us to the end of the Experiment section.
Evan: Let's move on to the Related Work section.
Ashley: In the Related Work section, the authors discuss various lines of research that intersect with the Declarative Attention protocol.
Evan: Right, so what are the primary areas of focus in this section?
Ashley: The authors categorize prior work into five main areas: dynamic sparse attention at decode, KV-cache eviction, complementary KV-cache axes, model-controlled inference and self-generated control tokens, and self-declared and steered attention.
Evan: Great, let's start with dynamic sparse attention at decode.
What does this involve?
Ashley: Dynamic sparse attention aims to reduce the number of tokens attended to at each decoding step by identifying the most relevant tokens dynamically.
Early methods used fixed heuristics like recency and attention magnitude to predict these tokens, but more recent approaches use lightweight scans or learned sparse indexing to achieve the same goal.
Evan: So how does this compare to Declarative Attention?
Ashley: The key difference is that Declarative Attention eliminates the need for these external predictors by having the model declare its own attention plan.
Instead of dynamically approximating attention scores, the DA protocol directly reads the intended scope from the model’s own generated text, making it more efficient.
Evan: I see.
How about KV-cache eviction?
Ashley: KV-cache eviction strategies focus on permanently dropping certain tokens from the attention cache based on their calculated importance.
Techniques like Scissorhands, SnapKV, and FastGen are notable examples.
However, these strategies risk not being able to retrieve previously discarded tokens if needed later.
In contrast, DA keeps all tokens resident and uses masking to selectively attend to them, providing reversibility and preserving accuracy.
Evan: And what about complementary KV-cache axes?
Ashley: Complementary KV-cache axes focus on reducing the size and memory bandwidth required for KV-cache reads.
This includes methods like low-bit quantization and offloading KV-cache to host memory.
Although these techniques can be combined with DA, they differ in that DA specifically reduces the number of tokens read per step by masking, not by shrinking or relocating the cached entries.
Evan: Interesting.
Now let's discuss model-controlled inference and self-generated control tokens.
Ashley: Model-controlled inference involves the model generating tokens that control and steer its own computation.
Examples include ReAct, Self-RAG, and MemGPT, which use control tokens to trigger specific actions like retrieving information or allocating more compute resources.
Declarative Attention fits into this broader trend by having the model control its own attention process through parseable declarations embedded within its output.
Evan: Okay, and what constitutes self-declared and steered attention?
Ashley: Self-declared and steered attention involves methods where the model explicitly dictates its attention spans.
Self-Selected Attention Span, or SSAS, is a prime example where the model outputs the spans to focus on during training.
While similar in spirit to DA, SSAS is task-specific and requires fine-tuning.
DA, on the other hand, is elicited zero-shot and does not need specific training, making it more versatile.
Evan: It sounds like Declarative Attention borrows concepts from these areas but adds its unique twist for real-time, zero-shot application.
Ashley: Exactly.
It combines the interpretability of model-generated control with the efficiency of dynamic sparse attention without needing external scores or extensive training.
Evan: That wraps up our look into the Related Work section.
Evan: Let's wrap up today's discussion by summarizing the key contributions and takeaways from this paper.
Ashley: To recap, the paper introduces a novel protocol called Declarative Attention, or DA, that allows language models to manage their own attention spans by declaring where they need to focus during the text generation process.
Evan: The core innovation here is the use of three distinct attention modes: global, focus, and local.
These modes enable the model to efficiently navigate and process long contexts by dynamically adjusting the scope of attention.
Ashley: Through zero-shot evaluations, DA has shown impressive results, cutting down the number of attended tokens during decoding by 52% on the Gemma-4-31B model and 31.1% on the Qwen-3.6-27B model.
This significant reduction in computational load comes with only modest drops in accuracy.
Evan: Another key advantage of DA is that it requires no additional training and can be applied to off-the-shelf models directly.
This makes it highly practical for real-world applications, particularly in resource-constrained settings.
Ashley: The authors also integrated DA into existing inference systems like vLLM, demonstrating that it can reduce total decode wall-clock costs substantially.
This shows the real-world feasibility and efficiency of DA.
Evan: Overall, Declarative Attention opens up new possibilities for efficient long-context processing in language models, making it a noteworthy contribution to the field.
Ashley: That brings us to the end of our discussion for today.
Thanks for tuning in to Daily Paper Cast.
Evan: We hope you found this paper as fascinating as we did.
Be sure to join us next time as we continue to explore the latest research in AI and NLP.