Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM

🤗 Upvotes: 51 | cs.AI

Authors:
Sergii Kozyrev, Davyd Maiboroda

Title:
Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM

Arxiv:
http://arxiv.org/abs/2609.04098v1

Abstract:
Hybrid LLMs pair softmax attention with linear-attention layers such as Gated DeltaNet (GDN), whose recurrent state summarizes the context in fixed size. Early community 4-bit quantizations of Qwen3.8-27B (48 GDN layers, 16 attention layers) left the GDN block in 8- or 16-bit precision -- especially its decay and write-strength gates -- on the intuition that errors in a recurrence accumulate over long contexts. We test that intuition by building Minima: NVFP4 W4A4 on all 496 linear layers, GDN included. Across perplexity at 4K/32K, MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, LiveCodeBench, and RULER retrieval to 64K, Minima matches BF16 within seed noise (5-task average -0.52) while being the smallest (17.5 GiB) and fastest-prefill (+14-19%) recipe we compare, and its 32K perplexity gap shrinks with position. A four-part mechanism study explains why: (i) NVFP4's 16-element block scaling localizes the residual stream's extreme outliers, equalizing activation error across layer roles; (ii) the supposedly fragile gate projections are the least sensitive -- softplus/exponential and sigmoid parameterizations compress ~11% GEMM error to ~2% output error; (iii) the delta-rule recurrence holds injected noise at a flat plateau over 32K tokens and forgets a state impulse within hundreds of steps, because each write overwrites the state along the current key direction; (iv) the per-token quantization cost washes out with context instead of compounding. We also repair a global-scale mismatch that arises when per-module-calibrated NVFP4 checkpoints are served by kernels that fuse those modules into one GEMM, and show calibrated FP8 KV-cache scales are performance-free. The result: a practical recipe -- quantize everything, ship KV scales -- and a mechanistic account of why the recurrent half of a hybrid LLM is the easy half to quantize. Checkpoint: https://huggingface.co/minima-ai/mnma_qwen3.8_27b_nvfp4

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast.

Evan: Today’s paper is from Hugging Face’s daily paper list of September 4, 2026, and it has received 51 upvotes.

Ashley: It's titled, 'Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM.'

Evan: The authors are Sergii Kozyrev and Davyd Maiboroda, with Sergii being the corresponding author.

Both are affiliated with MNMA AI.

Evan: Let’s dive into the Introduction section, shall we?

Ashley: Sure thing.

The paper explores the intersection of two major trends in large language models: the use of hybrid models that combine softmax attention with linear-attention layers and 4-bit quantization methods for weight and activation efficiency.

Evan: Interesting.

Could you give us a little more background on these trends?

Ashley: Hybrid language models, such as the Gated DeltaNet used in Qwen3.8-27B, leverage linear attention alongside traditional softmax attention.

This allows the model to maintain a fixed-size recurrent state that summarizes the entire context.

Evan: So the fixed-size recurrent state is a key part of the architecture.

What about the 4-bit quantization trend?

Ashley: 4-bit quantization, referred to in the paper as W4A4, is about using 4 bits for both weights and activations during inference to reduce the computational load and resource requirements.

NVFP4 is a specific format used in the paper, which pairs E2M1 4-bit values with an E4M3 scale per 16-element block.

Evan: And this is to make inference more efficient on current hardware, correct?

Ashley: Exactly.

The authors are addressing the challenge of quantizing the Gated DeltaNet (GDN) block of a large language model entirely to 4 bits, which hadn't been fully attempted before.

Evan: Got it.

So, the paper’s core objective is to show that the Gated DeltaNet can reliably operate under full 4-bit quantization.

What contributions do they make in this aspect?

Ashley: They make several key contributions.

First, they present Minima, the first true W4A4-GDN model where all 496 linear layers of Qwen3.8-27B are quantized to NVFP4 W4A4, including the gate projections.

They show it matches float-level performance within seed noise across six different accuracy suites.

Evan: Matching float-level performance with 4-bit quantization is impressive.

How do they demonstrate why it works?

Ashley: The authors conduct a detailed mechanism study to explain why GDN works with 4-bit quantization.

This includes activation statistics, sensitivity replays for projections, error propagation in the recurrence, and a positional decomposition of the perplexity gap.

They find that block scaling helps localize activation outliers, gate nonlinearities effectively squash errors, and the delta rule bounds and forgets state errors quickly.

Evan: What are some specific findings from their mechanism study?

Ashley: One of their findings is that the supposedly fragile gate projections are actually the least sensitive to quantization noise.

They show that errors from the softplus and sigmoid parameterizations of the gates’ projections result in minimal impact on overall output errors.

Furthermore, the recurrence mechanism actively overwrites state errors along the current key direction, making the GDN resilient to quantization noise.

Evan: Interesting.

They also mentioned something about a practical recipe for deployment.

Can you elaborate on that?

Ashley: Definitely.

In terms of practical deployment, the authors suggest a 'quantize everything' approach, emphasizing the inclusion of calibrated FP8 scales for the KV-cache.

This ensures better performance with minimal impact on long-context tasks.

Evan: Alright, that brings us to the end of the Introduction section.

Evan: Now let’s move on to the Method section, where the authors detail their approach to quantizing the Gated DeltaNet.

Ashley: The core aspect of their method is the implementation of NVFP4 W4A4 quantization across all the linear layers of the Qwen3.8-27B model.

Specifically, they include 48 GDN layers and 16 attention layers.

Evan: Right, so what exactly is NVFP4 W4A4, and how does it work in their approach?

Ashley: NVFP4 stands for a format that stores values in a combination of E2M1 and E4M3, supporting 4-bit values for weights and activations.

Essentially, E2M1 represents 4-bit values, and E4M3 provides a scale per 16-element block to maintain precision.

W4A4 denotes the quantization of both weights and activations to 4 bits.

Evan: Got it.

How does this play out in the context of recurrent state GDN specifically?

Ashley: Great question.

In the context of the Gated DeltaNet, this involves quantizing the recurrent state, which summarizes the context in a fixed size matrix.

This matrix gets updated recurrently as the model processes sequences of tokens.

Evan: Interesting.

What makes the GDN quantization particularly challenging?

Ashley: Quantizing GDN is challenging because of the belief that errors accumulate over long contexts, especially in the decay and write-strength gate projections.

Previous efforts safeguarded these projections by keeping them in higher precision.

Evan: But the authors' experiments have shown otherwise, right?

Ashley: Indeed.

They proposed to quantize everything, including these projections, and demonstrated that it doesn't negatively impact the model's performance.

They conducted thorough tests across multiple datasets to substantiate this.

Evan: Let's talk about those datasets and evaluation methods.

What were the tasks and metrics used?

Ashley: They evaluated the model on an array of benchmarks including perplexity at 4,000 and 32,000 tokens, MMLU-Pro, GSM8K, AIME’25, GPQA-Diamond, LiveCodeBench, and RULER retrieval up to 64K tokens.

Perplexity measures the model's ability to predict the next word in a sequence, and the other tasks assess various aspects like language understanding, problem-solving, and code generation.

Evan: That’s a a comprehensive evaluation.

What did they find in terms of performance?

Ashley: They reported that Minima, their fully quantized model, matches the performance of the BF16 model within seed noise across these benchmarks.

Specifically, they found the average difference to be about -0.52 on a five-task average, which is negligible.

Evan: They also mentioned efficiency improvements.

Can you elaborate on those?

Ashley: Efficiency was a significant finding.

The Minima model is 2.9 times smaller in VRAM compared to BF16 and up to 19% faster in prefill operations.

This is significant for practical deployment because it reduces the resource footprints and speeds up inference times.

Evan: Great, and they mentioned something about block scaling and activation statistics helping in localization of errors?

Ashley: Yes, that’s correct.

They found that by using block scaling to localize extreme outliers in the activation values, the resulting quantization errors were confined to within the 16-element blocks.

This significantly reduced the error propagation.

Evan: Can you explain how gate nonlinearities played a role?

Ashley: Sure.

They observed that the softplus and sigmoid function parameterizations of the gate projections compress quantization noise effectively.

An 11% error in gate projection GEMM was translated to only about a 2-3% output error, thanks to these nonlinearities.

Evan: What about the delta rule recurrence you mentioned earlier?

Ashley: The delta rule, which forms the basis for GDN updates, helps in managing quantization noise.

It does this by overwriting the state in the direction of the current key, effectively canceling out errors quickly as new tokens are processed.

Evan: This mechanism seems to eliminate error accumulation over long contexts, right?

Ashley: Precisely.

They demonstrated that the state error remains flat over 32,000 tokens, and longer contexts actually see the per-token impact diminishing, contradicting the initial community intuition.

Evan: Were there any other interesting findings during their method evaluation?

Ashley: One more key finding was regarding the consistency in calibration and serving stack modifications.

They identified and corrected a scale mismatch issue that arose when running per-module calibrated NVFP4 checkpoints on fused GEMM kernels.

This fix was necessary to correctly measure the performance of the quantized model.

Evan: That’s a vital point for anyone looking to replicate these results.

Anything else on the KV-cache performance?

Ashley: Yes, they also addressed FP8 KV-cache precision, finding that the quantization cost for weight and key-value cache compound to a more substantial penalty in Minima unless calibrated scales per attention layer were included.

With these scales, they recovered 83% of the performance hit.

Evan: That's the end of the Method section.

The team has certainly laid out a robust and detailed approach to 4-bit quantization for Gated DeltaNet.

Evan: Now let’s proceed to the Experiment and Results section, where the authors present their findings.

Ashley: The authors evaluated the Minima model on a variety of benchmarks to compare its performance against BF16 and other quantized versions of the Qwen3.8-27B model.

Evan: That's right.

They used six specific task suites and also measured perplexity at 4,000 and 32,000 tokens.

What did they discover?

Ashley: The primary discovery was that Minima matches the BF16 model's performance within seed noise across all tasks.

For instance, in the five-task average, the performance gap was only about -0.52, which is quite negligible, considering the reduced bit precision.

Evan: Could you break down the specific results for some of the tasks?

Ashley: Certainly.

On the WikiText-2 perplexity task, they reported PPL scores of 7.67 at 4K tokens and 10.84 at 32K tokens for Minima.

This was compared to 6.95 and 10.35, respectively, for the BF16 model.

Evan: What about the other benchmarks?

Ashley: Sure.

For the MMLU-Pro task, Minima scored 79.7% compared to 80.4% for BF16.

On GSM8K, both models scored 95.5%.

In AIME’25, Minima and BF16 both scored 86.7%.

For GPQA-Diamond, the scores were 85.1% for Minima and 86.5% for BF16.

Evan: Those are tight margins.

How did Minima perform on the LiveCodeBench v6?

Ashley: On LiveCodeBench v6, Minima scored 78.5%, slightly behind BF16 at 79.0%.

These results across diverse benchmarks confirm Minima's robustness.

Evan: Let’s discuss the efficiency metrics.

What improvements did the authors report?

Ashley: Minima was found to be 2.9 times smaller than BF16 in terms of VRAM usage, taking only 17.5 GiB compared to 50.13 GiB.

This reduction is significant for scaling and deploying large models cost-effectively.

Evan: And the speed metrics?

Ashley: Minima outperformed others in prefill operations, completing 32K-token prefill tasks in 4.03 seconds, compared to 6.90 seconds for BF16.

It also showed similar prompt throughput improvements.

Evan: That’s quite an improvement.

What about their analysis of quantization impact over token positions?

Ashley: Interestingly, the quantization impact diminished over longer contexts.

They observed that Minima’s perplexity gap with BF16 narrowed from +0.72 at 4K tokens to +0.49 at 32K tokens.

This indicates that the quantization cost is absorbed as the context grows.

Evan: And how did Minima handle activation and weight errors?

Ashley: The authors found that activation errors were confined within 15-16% due to block scaling, preventing degradation across the entire layer.

Weight errors were slightly higher but still maintained within the acceptable range of 10.5-11.9%.

Evan: Were there any findings around robustness?

Ashley: Yes, their results showed that the gate projections, despite being quantized to 4 bits, handled errors very well.

The projected gate outputs had minimal impact on overall output errors, demonstrating great resilience.

Evan: Do they provide any insights into practical deployment?

Ashley: Indeed.

They highlighted the importance of per-layer calibrated FP8 scales for the KV-cache, which when applied, recovered 83% of the model's long-context performance penalties.

This calibration is performance-free and highly recommended for deployment.

Evan: That’s very useful information for practitioners.

And with that, we've come to the end of the Experiment section.

Evan: Now we'll move on to the Related Work section where the paper situates itself within the existing research landscape.

Ashley: The authors start by discussing the previous work on linear attention and hybrid models.

Notably, they reference Gated DeltaNet's combination of the parallelizable delta rule with Mamba2-style gating, deployed as the main mixer in Qwen3.8-27B.

Evan: What makes Gated DeltaNet typical within these hybrid models?

Ashley: Gated DeltaNet is significant for its ability to leverage the efficiency of the delta rule in parallelizing layers while incorporating gating mechanisms to improve state management.

This combination results in enhanced performance for large language models.

Evan: And how does this intersect with low-bit quantization work?

Ashley: The authors reference work on low-bit weight-only and weight-activation quantization methods, which have demonstrated that handling outliers is a critical challenge in W4-W8 inference.

They mention NVFP4 as a hardware-native format that integrates this management directly into the datatype.

Evan: NVFP4 seems central to their work.

How does it fit in with the existing quantization techniques?

Ashley: Indeed, NVFP4 is a specialized format supporting native 4-bit tensor operations, which builds on earlier methods for outlier management in quantization.

This format ensures that even extreme values are contained within their block scales, simplifying implementation on current accelerators.

Evan: Are there any other directly related approaches mentioned by the authors?

Ashley: Yes, the authors discuss GPTQ for accurate post-training quantization of transformers and SmoothQuant for post-training quantization of large models, both of which informed their approach.

Additionally, they mention AWQ, which emphasizes activation-aware weight quantization for model compression.

Evan: Sounds like there’s a rich landscape of quantitative techniques being applied.

Do the authors note any recent checkpoints or projects?

Ashley: Yes, they mention QUASAR, which released a concurrent checkpoint of Qwen3.8-27B using quantization-aware training.

QUASAR's model also quantizes all 496 projections, similar to this paper's approach, but utilizes learned weights under quantization instead of post-training quantization.

Evan: That's fascinating.

How do they view their method in comparison?

Ashley: The authors argue that while QUASAR adds an extra layer of complexity with its quantization-aware training, their calibration-only post-training quantization (PTQ) achieves comparable accuracy.

They emphasize that the mechanisms inherent in GDN’s architecture support robust performance even with straightforward PTQ.

Evan: Do they discuss any practical implementations of their approach?

Ashley: They reference their concurrent engineering work which extends the Minima quantization approach to the full model, including embeddings and the language model head, showing that broader application is feasible and beneficial for efficient deployment.

Evan: How does their work interact with other methods for KV-cache optimization?

Ashley: The paper references methods like KVQuant that target long-context inference.

Specifically, hybrid models, which inherently reduce KV-cache size, benefit from the nuanced approach to scaling and calibration in the context of 4-bit quantization, as illustrated by their practical guidance on KV-cache precision.

Evan: What about future directions or ongoing challenges?

Ashley: The authors acknowledge limitations, such as the scope of evidence being constrained to one model family and the need for further testing on longer contexts.

They also recognize the potential need for different parameters on other architectures, especially where delta and correction mechanisms vary.

Evan: Clearly, there's a complex web of developments in this field.

And that's the end of the Related Work section.

Evan: We've now covered the main sections of today's paper.

Let's summarize the key contributions and takeaways.

Ashley: The authors introduced Minima, a fully quantized Gated DeltaNet model using NVFP4 W4A4.

They demonstrated that all 496 linear layers, including the gate projections, can be quantized to 4 bits without compromising performance.

Evan: Specifically, Minima matched the performance of a BF16 model across a range of benchmarks including perplexity, language understanding, problem-solving, and code generation tasks.

Ashley: Indeed.

They also showed that the quantization introduced significant efficiency improvements.

Minima was 2.9 times smaller in VRAM usage and delivered faster prefill operation times compared to the BF16 model.

Evan: A key takeaway from their mechanism study was that the gate projections, initially thought to be fragile under quantization, actually handle errors very well due to the architecture's inherent robustness.

Ashley: They also highlighted the importance of calibrated scales for the KV-cache.

This practical recommendation ensures long-context performance remains robust under 4-bit quantization.

Evan: In summary, this paper not only pushes the boundaries of 4-bit quantization but also provides detailed insights into practical deployment for hybrid large language models.

Ashley: That wraps up today's discussion on the Daily Paper Cast.

We hope you found this episode insightful.

Evan: Be sure to join us again for more deep dives into the latest AI research.

Thanks for listening, and see you next time!