STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

🤗 Upvotes: 84 | cs.CL, cs.AI, cs.LG

Authors:
Bingchen Yao, Haobo Xu, Haokun Lin, Yichen Wu, Ziyu Guo, Renrui Zhang, Zhichao Lu, Zhenan Sun, Ying Wei

Title:
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

Arxiv:
http://arxiv.org/abs/2609.38169v1

Abstract:
Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through successive state updates. We discover that the impact of these errors depends on two complementary dimensions: temporally, errors in long-lived memory can persist across many decoding steps; spatially, errors in different key rows affect model outputs differently, while state magnitudes vary substantially along both rows and columns. Motivated by these observations, we propose STEPQuant, a spatial-temporal post-training quantization framework for Delta-rule recurrent states. STEPQuant allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales based on state distributions and key-row impact on output error. Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across both long- and short-generation benchmarks show that STEPQuant closely matches FP32-state accuracy under a nominal 6-bit budget and outperforms uniform INT8 in its 4-bit configuration. Integrated into SGLang with optimized GPU kernels, 6-bit STEPQuant achieves over 5x recurrent-state compression and reduces total serving memory by up to 68.7%. Our code is available at https://github.com/Dreamer-Toby/STEPQuant.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to another episode of Daily Paper Cast.

Ashley: Today's paper is selected from the Hugging Face daily paper list of October 8, 2026, receiving 84 upvotes.

Evan: It's titled 'STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization.'

Ashley: The lead authors are Bingchen Yao from Zhejiang University and Haobo Xu from Tsinghua University.

The corresponding author is Ying Wei from Zhejiang University.

Evan: Let's dive into the introduction.

This paper addresses a significant challenge in linear attention mechanisms, which are increasingly popular in NLP models due to their fixed memory footprint, unlike the growing memory demands of traditional softmax attention.

Ashley: Linear attention mechanisms maintain a fixed-size recurrent state instead of a KV cache.

However, these recurrent states can still become a substantial memory bottleneck during concurrent serving, especially when targeting high concurrency scenarios.

Evan: Exactly, and to mitigate this, the authors explore the quantization of recurrent states to lower precision.

But simply applying uniform quantization results in severe accuracy degradation because the errors from quantization propagate through successive state updates.

Ashley: They discovered that these errors impact the model's performance across two key dimensions: temporally, where errors can persist across many decoding steps; and spatially, where errors in different key rows can affect the outputs differently, given the substantial variation in state magnitudes along both rows and columns.

Evan: Motivated by these findings, the authors proposed STEPQuant, a spatial-temporal post-training quantization framework designed specifically for Delta-rule recurrent states.

Ashley: STEPQuant dynamically allocates precision based on the magnitude of errors and the lifetime of memory.

It jointly fits scales for key rows and value columns, tailored to the state distributions and the impact of key rows on output error.

Evan: Their experiments with hybrid models like Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct show that STEPQuant closely matches the accuracy of FP32 states under a nominal 6-bit budget.

At a 4-bit configuration, it even outperforms uniform INT8 quantization.

Ashley: When integrated into SGLang with optimized GPU kernels, the 6-bit STEPQuant achieves over 5× recurrent-state compression and significantly reduces total serving memory by up to 68.7%.

Evan: In summary, the paper presents several key contributions.

First, it provides a thorough error analysis demonstrating how quantization errors propagate temporally and spatially in Delta-rule recurrent states.

Ashley: Second, it introduces the STEPQuant framework, which offers a novel combination of lifetime-aware bit allocation and key-row-aware dual-axis fitting.

This approach maintains high accuracy even under lower precision settings.

Evan: And third, the paper details the integration of STEPQuant into a serving system, showcasing a practical implementation that achieves significant memory compression and faster state updates.

Ashley: That wraps up the Introduction section of the paper.

Stay tuned as we unpack the methods and experiments in the following segments.

Evan: Let's dive into the methods proposed in this paper, Ashley.

Can you start by explaining the foundational concepts?

Ashley: Sure, Evan.

At the core, STEPQuant builds upon linear attention mechanisms, specifically the gated DeltaNet (GDN) and Kimi Delta Attention (KDA), which are both well-regarded for their efficiency compared to traditional softmax attention.

Evan: How do these methods differ from softmax attention?

Ashley: Unlike softmax attention, which maintains a KV cache that grows with sequence length, linear attention summarizes past tokens into a fixed-size recurrent state.

This fixed-size state matrix avoids the quadratic complexity typically seen in softmax attention.

Evan: And are these fixed states the same thing causing the memory bottlenecks when serving multiple requests concurrently?

Ashley: Exactly.

Each request requires a separate persistent state.

When serving many concurrent requests, the memory footprint increases dramatically, creating bottlenecks despite the fixed state size.

Evan: Got it.

So what's the proposed solution to these memory bottlenecks?

Ashley: The solution proposed is STEPQuant, which stands for Spatial-TEmPoral Quantization.

The method focuses on quantizing the recurrent states to reduce their memory footprint while preserving model accuracy.

It operates on two core dimensions: temporally and spatially.

Evan: Let's break down those dimensions.

What does STEPQuant do temporally?

Ashley: Temporally, STEPQuant hinges on the concept of Lifetime-aware Bit Allocation.

This method estimates how quantization errors accumulate over time and assigns precision according to error magnitude and memory lifetime.

Essentially, units with longer memory lifetimes and larger accumulated errors receive higher precision.

Evan: That sounds intriguing.

How is the error propagation handled?

Ashley: The paper provides a conditional error propagation model.

It shows that previously accumulated errors are attenuated by the retention gate and adjusted by the Delta update according to the key vectors.

This selective process helps in understanding how errors evolve with each update step and motivates the need for finer precision in longer-lived states.

Evan: That's clear enough.

Now, what about the spatial dimension?

Ashley: Spatially, STEPQuant incorporates Key-Row-Aware Dual-axis Fitting.

This technique addresses the differences in error impact across key rows and columns within the recurrent state matrix.

Since errors in different key rows affect the model output differently, STEPQuant assigns separate scaling factors for key rows and value columns based on their impact and magnitude.

Evan: So there's a dual-axis scaling strategy involved.

How does this contribute to error reduction?

Ashley: Yes, the dual-axis scaling helps in better accommodating the varying magnitudes and impact patterns within the state matrix.

For key rows, the scaling factor considers both the row's current magnitude and its influence on readout error.

For value columns, the scales minimize reconstruction error with greater emphasis on rows that contribute significantly to output error.

Evan: What role do the FP16 pivots play in STEPQuant?

Ashley: FP16 pivots are used to protect the highest-risk units.

These are units that even with mixed-precision allocation prove difficult to quantize effectively.

The protected units are kept at FP16 precision to prevent substantial accuracy loss during decoding.

Evan: Interesting.

How does the quantization process integrate into the serving system?

Ashley: STEPQuant's integration into the serving system, particularly SGLang, involves optimized GPU kernels that handle the packed representation of recurrent states.

During each decoding step, the packed state kernels reconstruct the state, apply the Delta update, calculate the current readout, and subsequently fit the updated scales.

This enables efficient state updates and reduced memory consumption.

Evan: What were the results of the experiments on the hybrid models, Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct?

Ashley: Experiments demonstrated that STEPQuant closely matches the FP32-state accuracy under a nominal 6-bit budget.

The method achieves significantly better performance than uniform INT8 quantization when operating under a 4-bit budget.

Evan: Did they report any notable improvements in efficiency?

Ashley: Integrating STEPQuant into SGLang with optimized GPU kernels delivers over 5× recurrent-state compression and up to 2.91× faster state updates.

Total memory consumption during serving was reduced by up to 68.7% for Qwen and 53.7% for Kimi, showcasing substantial efficiency gains.

Evan: That's impressive.

So, the combination of temporal and spatial quantization strategies really allows for a refined balance between accuracy and memory efficiency.

Ashley: Precisely.

By understanding where and when errors matter most, STEPQuant can allocate resources more effectively to maintain high model performance even under constrained precision budgets.

Evan: This brings us to the end of the methods section.

Next, we'll review the experimental setups and results that validate STEPQuant's effectiveness.

Evan: Alright, Ashley.

We've covered the methods behind STEPQuant.

Now, let's discuss the experiments that validate its effectiveness.

What did the authors set out to test?

Ashley: The experiments aimed to evaluate STEPQuant on several fronts: accuracy across long- and short-generation tasks, memory efficiency, and latency in a real-world deployment setting using SGLang.

Evan: That sounds comprehensive.

Which models and benchmarks were used in these experiments?

Ashley: They evaluated two hybrid models: Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct.

For benchmarks, they used a mix of seven long-generation reasoning tasks and six short-generation language understanding tasks.

Evan: Can you provide some details about these tasks?

Ashley: Of course.

The long-generation tasks included benchmarks like LiveCodeBench v6, EvalPlus, AIME 2026, MATH-500, HMMT February 2026, GPQA Diamond, and IFBench, which assess the model's reasoning capabilities over extended text sequences.

Evan: And what about the short-generation tasks?

Ashley: The short-generation tasks were MMLU, ARC-C, OpenBookQA, HellaSwag, WinoGrande, and LAMBADA.

These tasks are designed to measure language understanding and require the models to generate concise responses.

Evan: Alright, let's talk numbers.

How did STEPQuant perform in terms of accuracy?

Ashley: For long-generation tasks, STEPQuant@6bit closely matched the FP32-state baseline with an average accuracy of approximately 80.59% on Qwen and 61.47% on Kimi.

In contrast, uniform INT6 quantization resulted in significantly lower accuracies of 45.04% and 45.70%, respectively.

Evan: And how did it fare at the 4-bit configuration?

Ashley: At 4-bit, STEPQuant still outperformed uniform INT8 quantization.

It showed competitive performance, trailing the FP32 baseline by only 0.15 points on Qwen and 0.25 points on Kimi for short-generation benchmarks.

Evan: That's quite a difference.

How about memory efficiency?

Ashley: In terms of memory efficiency, STEPQuant@6bit achieved over 5× compression of recurrent states and reduced total serving memory by up to 68.7% for Qwen and 53.7% for Kimi.

Evan: Those are impressive reductions.

What about the impact on latency and serve times?

Ashley: The integration of optimized GPU kernels significantly sped up state updates.

STEPQuant@6bit reduced state update time by 65.6% for Qwen and 43.9% for Kimi, leading to faster overall serving times.

Evan: Did they break down these improvements at different batch sizes?

Ashley: Yes, they did.

For batch sizes ranging from 32 to 512, STEPQuant consistently improved decoding throughput.

For instance, at a batch size of 512, Qwen's throughput increased by 20.53%, and Kimi's by 11.80%.

Evan: It's clear that STEPQuant not only preserves accuracy but also enhances performance efficiency.

Did the authors perform any ablation studies to understand the contributions of individual components in STEPQuant?

Ashley: They did.

The ablation studies showed that both temporal and spatial components are crucial for STEPQuant's performance.

For example, on Qwen, the spatial fitting alone achieved 80.19% accuracy at 6-bit, but combined with temporal allocation, it reached 84.51%, almost matching the FP32 baseline.

Evan: That's insightful.

How does STEPQuant handle combinations with other quantization methods, like AWQ?

Ashley: Interestingly, they tested STEPQuant with 4-bit AWQ-quantized weights.

The results showed that STEPQuant maintained high accuracy, closely trailing the FP32 baselines by merely 0.05 points on Qwen and 0.33 points on Kimi, while also achieving significant memory savings.

Evan: So, combining STEPQuant with other quantization techniques seems to further optimize memory and efficiency.

Were there any notable observations related to the generation lengths across different tasks?

Ashley: Yes, STEPQuant maintained generation lengths similar to the FP32 states, whereas uniform quantization often resulted in excessively long outputs without corresponding accuracy benefits.

This was particularly evident in tasks like AIME and HMMT, where uniform 4-bit quantization generated near-limit outputs with almost zero accuracy.

Evan: In summary, STEPQuant not only preserves accuracy across different tasks but also offers significant memory compression and reduces latency.

Its sophisticated approach to recognizing when and where errors matter most allows it to deliver highly efficient performance.

Ashley: That ends the experiment section.

Next, we'll look into the related work and how STEPQuant compares with other approaches.

Evan: Ashley, we've discussed the experiments, but I'm curious about one thing.

How does STEPQuant stack up against existing methods in the field?

Was there a Related Work section in the paper that covered this?

Ashley: Yes, there was.

The Related Work section provides a thorough comparison of STEPQuant with other approaches in efficient linear attention mechanisms and post-training quantization methods.

Let's break it down.

Evan: Let's start with the attention mechanisms.

What other architectures are highlighted in this context?

Ashley: The paper begins by discussing linear attention transformers, which offer more efficient alternatives to the quadratic complexity of standard self-attention.

One notable architecture mentioned is RetNet, which introduces multiscale retention with parallel and recurrent formulations, improving memory management and computational efficiency.

Evan: I've heard of RetNet.

Are there any other attention models discussed?

Ashley: Yes.

There's the Gated Linear Attention (GLA) model, which uses data-dependent gates to control information retention.

Another example is the Selective State-Space Model (SSM), seen in architectures like Mamba and Mamba-2.

These models also employ recurrent inference mechanisms to optimize memory use and training algorithms.

Evan: It sounds like these models are focused on improving memory and computational efficiency.

How do they do that?

Ashley: Exactly.

For instance, Gated DeltaNet (GDN) and Kimi Delta Attention (KDA) use head-wise and channel-wise forgetting mechanisms alongside delta-rule updates.

This selective method helps manage memory more efficiently.

Evan: So that's how other linear attention architectures are designed to be efficient.

How does STEPQuant's approach to quantization compare to other post-training quantization methods?

Ashley: STEPQuant relates closely to post-training quantization (PTQ) strategies, which aim to reduce model memory and computational overhead by representing weights in lower-bit formats.

One widely recognized PTQ approach is GPTQ, which accurately quantizes weights after training, contributing to less memory-intensive inference.

Evan: Ah, GPTQ is familiar to me.

What about the quantization efforts specifically for linear attention or state-space models?

Ashley: There's a few worth mentioning.

Quamba and its successor Quamba2 focus on post-training quantization for models using selective state representations.

There's also MambaQuant, which targets both weights and activations for quantization, including the states.

Evan: How does STEPQuant distinguish itself from these methods?

Ashley: STEPQuant stands out by specifically addressing the temporal and spatial quantization needs of recurrent states in gated linear attention models.

Unlike general PTQ methods, it combines Lifetime-aware Bit Allocation and Key-Row-Aware Dual-axis Fitting to address how quantization errors accumulate and impact model performance over time and across different parts of the state matrix.

Evan: Has there been direct comparison with concurrent work?

Ashley: Yes, the paper compares STEPQuant's performance with DAMP, another recent method that focuses on decay-based persistence for selecting key channels to protect with FP16 precision while quantizing others to INT8.

STEPQuant shows competitive, if not better, performance by achieving near-FP32 accuracy at a lower effective bit precision.

Evan: It seems like STEPQuant's sophisticated approach to allocating precision based on both temporal and spatial dimensions really gives it an edge.

Ashley: Indeed, by recognizing when and where errors matter, STEPQuant effectively balances the trade-off between precision and memory utilization, setting it apart from other approaches that might only focus on one aspect of quantization.

Evan: And that concludes the Related Work section.

Stay with us as we move on to discuss the conclusions drawn from this study.

Evan: We've unpacked quite a bit of detail about STEPQuant.

Ashley, could you summarize the key contributions and takeaways from the paper?

Ashley: Certainly, Evan.

The paper makes several significant contributions that stand out.

First, it provides an in-depth error analysis, showing how quantization errors propagate through Delta-rule recurrent states both temporally and spatially.

This analysis is crucial for understanding the impact of quantization on model performance.

Evan: That's right.

And the second contribution?

Ashley: The second contribution is the introduction of the STEPQuant framework.

This novel approach combines Lifetime-aware Bit Allocation and Key-Row-Aware Dual-axis Fitting.

By addressing both how long errors persist and where they occur, STEPQuant ensures high accuracy even at lower precision settings.

Evan: And how does this integration into practical systems come into play?

Ashley: The third contribution is the practical implementation of STEPQuant into the SGLang serving system with optimized GPU kernels.

This implementation demonstrates significant efficiency gains, achieving over 5× recurrent-state compression and faster state updates, reducing total serving memory by up to 68.7%.

Evan: That's impressive.

So essentially, STEPQuant not only enhances model accuracy under constrained precision budgets but also provides substantial memory savings and performance improvements during inference.

Ashley: It's a sophisticated approach that meticulously allocates precision based on where and when errors matter most, making it a highly effective method for optimizing neural network performance and efficiency.

Evan: That wraps up our discussion on STEPQuant.

Thank you, Ashley, for breaking down the paper's insights so clearly.

Ashley: It was my pleasure, Evan.

And thank you to our listeners for joining us today.

Evan: We hope you enjoyed this episode of Daily Paper Cast.

Be sure to tune in for future episodes where we continue to explore and unpack interesting papers from the world of AI and machine learning.

Ashley: Until next time, stay curious and keep learning!

Evan: Have a great day, everyone.

Goodbye!