LatentPress: Context Compression Beyond Text and Vision

🤗 Upvotes: 52 | cs.LG, cs.AI

Authors:
Zhengze Zhou, Hejian Sang

Title:
LatentPress: Context Compression Beyond Text and Vision

Arxiv:
http://arxiv.org/abs/2609.01507v2

Abstract:
Compressed context is usually carried as human-readable text or as rendered images that must be decoded, even when its consumer is a language model. We introduce LatentPress, which writes conversational histories and long documents into a third representation: continuous memory tokens that a frozen decoder reads directly through its input-embedding interface, with no text reconstruction at inference. A small reader-matched writer compresses $4$-$16\times$ while training only an adapter (4.2M-26.2M parameters, $\sim\!0.1\%$ of the decoder). On LongMemEval, LatentPress reaches $0.504$ accuracy at $7.70\times$ compression versus $0.490$ for uncompressed evidence, outperforming text summaries (0.184) and OCR-based compression (0.426 to 0.312). On LongBench-QA, in-domain writers match or exceed raw-context reading at $4$-$8\times$ compression, while $16\times$ trails raw. Writing takes 43ms per conversation, roughly an order of magnitude faster than text summarization or OCR reconstruction, and reading is $5$-$9\times$ faster than raw context or cached OCR. We validate the interface under two transfer settings, zero-shot from UltraChat to LongMemEval memory QA and from LongMemEval-derived QA to unseen LongBench document domains, establishing direct soft tokens as a practical machine-facing context interface beyond text and vision. The implementation of the experiments could be found at: https://github.com/HJSang/LatentPress .

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast, everyone!

Ashley: Today's paper comes from the Hugging Face daily paper list of September 4, 2026, and it's already garnered 52 upvotes.

Evan: The title of the paper is 'LatentPress: Context Compression Beyond Text and Vision.'

Ashley: The first two authors are Zhengze Zhou from Cornell University and Hejian Sang from Iowa State University, with correspondence to Zhengze Zhou.

Evan: Alright, Ashley, let's dive into the Introduction.

What problem is this paper addressing?

Ashley: The paper tackles the challenge of efficiently managing and processing large amounts of conversational histories and long documents that AI models often accumulate.

Evan: Why is this a problem that needs addressing?

Ashley: Well, long-running assistants and agents, for example, gather more history than they can afford to process completely anew every time.

This history can include instructions, dialogue, plans, tool usage, observations, and feedback.

Similarly, when models read long documents, they often need only small, relevant pieces of information to answer specific questions.

Evan: I see.

It's about efficiency in context usage.

How do systems currently handle this issue?

Ashley: Most systems use discrete text as the machine-facing interface.

They retrieve, summarize, prune, or reconstruct text from other formats.

Though text is convenient for human understanding and interoperability, a language model doesn't require human-readable text to utilize stored or compressed context.

Evan: So, the question is if we can go beyond human-readable formats for this compressed context, right?

Ashley: Exactly.

This paper introduces LatentPress, which compresses context into continuous memory tokens.

These tokens are read directly by a frozen language model decoder, without needing to reconstruct text at the inference stage.

Evan: Interesting.

How does LatentPress approach this context compression?

Ashley: LatentPress splits the use of context into two operations: WRITE and READ.

The WRITE operation maps text to a compact representation using continuous vectors, while the READ operation feeds this representation into a frozen decoder for tasks like question answering.

Evan: That sounds clever.

What's unique about LatentPress compared to previous methods in this space?

Ashley: LatentPress differs from prior methods by keeping the downstream decoder entirely frozen and training only a small reader-matched adapter.

It aims for efficient, near-real-time forward passes for compression, making it faster and potentially more scalable.

Evan: And how practical is this approach in real-world applications?

Ashley: The practical choices LatentPress incorporates include determining how much to compress each segment of context and training the writer adapter using relevant supervision to retain essential information.

Evan: So, it's about smart compression rates and effective training for the writer.

What are the main contributions the authors highlight?

Ashley: The key contributions are threefold: First, introducing LatentPress as an interface for directly-read soft-token memory that doesn't rely on text reconstruction.

Second, demonstrating competitive accuracy and faster speeds in context reading tasks compared to uncompressed or text-summary-based approaches.

Lastly, validating this method's transferability across different domains and tasks, such as conversations in UltraChat and document questions in LongBench-QA.

Evan: That's quite a significant set of capabilities.

It's exciting to think about the implications for practical AI systems.

Ashley: Indeed.

And that wraps up the Introduction section of the paper.

Evan: Alright, Ashley, let’s dig into the Method section of the paper.

How does LatentPress actually work?

Ashley: The methodology of LatentPress centers on creating a direct-read soft-token interface.

It comprises a reader-matched writer that compresses text segments into continuous vectors.

These vectors are then directly fed into a frozen language model decoder through its input-embedding interface.

Evan: Okay, that sounds technical.

Can you unpack the components of this interface and how they function together?

Ashley: Sure.

LatentPress starts by defining a context sequence composed of segments, which could be dialogue turns or document chunks.

The frozen decoder is tasked with answering a question based on a compact representation of this context.

Evan: So the context is being represented in a compact form?

What about the training for this compact representation?

Ashley: Exactly.

The compact representation is generated by a small trainable writer.

This writer takes the sequence of context segments, transforms them into continuous vectors, and injects these into the decoder through its input-embedding interface.

The training for this writer involves mapping these text segments to a shorter sequence of continuous vectors that retain the necessary context for the decoder to understand and answer the question.

Evan: Interesting.

What determines the compression rates for these segments?

Ashley: The compression rate for each segment is determined by a simple hand-specified rule, represented as π = (k1, . . . , kT).

This rule decides how many neighboring token positions are pooled into each soft token.

Although the rates are currently hand-specified, the authors suggest that future work could involve learning these rates automatically.

Evan: And what choices do they make to supervise the writer in retaining the key information?

Ashley: For supervision, they use two approaches.

One is for generic representation learning, combining a reconstruction loss and a forward-KL loss to distill information into the compressed context.

The other approach is for task-specific adaptation, training the writer on question-answering examples from the target or different domains.

Evan: Got it.

Now, the paper mentioned that LatentPress also differentiates between roles in conversational turns and handles long documents.

How does this work in practice?

Ashley: For conversations, LatentPress employs a role-based compression schedule.

This means user turns are often kept lossless, with kuser set to 1, whereas assistant turns are significantly compressed, with kassistant set to values like 8, 16, or 32.

This approach ensures that critical user-provided information remains detailed, while less essential assistant responses are more compressed.

Evan: That seems clever.

How does it handle long documents where there's less clear structure than in conversations?

Ashley: For long documents, a uniform compression rate is applied across the whole document.

This simplifies the process but ensures that detail is retained uniformly, allowing the system to generalize beyond structured conversational data to more free-form text.

Evan: You mentioned the interface does not require reconstructing the text at the inference stage.

How is this beneficial?

Ashley: Avoiding text reconstruction during inference means the decoder can read the context and answer questions much faster.

This increases efficiency since the compact representation is directly used, skipping the laborious step of turning it back into human-readable text before processing.

Evan: Let's talk about practical application.

How fast is this method in practice?

Ashley: The authors report that encoding tokens with LatentPress takes about 43 milliseconds per conversation, which is substantially quicker than text summarization or OCR-based methods.

Reading from the compressed context is also 5 to 9 times faster than reading from raw context or OCR caches.

Evan: What about the footprint of the trainable parts?

How large is the writer?

Ashley: The trainable footprint is intentionally kept small, only involving the writer head, which ranges from about 4.2 million to 26.2 million parameters depending on the model used.

This is approximately 0.1 percent of the frozen decoder's parameters, making it very lightweight.

Evan: Is this approach validated across different setups or models?

Ashley: Yes, the method is validated under two transfer settings.

One is zero-shot transfer from generic UltraChat conversations to LongMemEval memory-QA, where the writer trained on UltraChat dialogues is tested on LongMemEval.

The second is transferring from LongMemEval-derived QA to other unseen LongBench document domains.

These tests illustrated that LatentPress could effectively generalize across different types of data and tasks.

Evan: Seems like they've covered the practical aspects pretty thoroughly.

How do they ensure the efficacy of the compression rate choices?

Ashley: The authors chose a structured, heuristic approach to compression rates based on the input structure.

For instance, in conversation, keeping user inputs more detailed than assistant inputs, and for documents, applying a uniform compression rate.

They leave dynamic, learned compression policies to future work, suggesting that reinforcement learning could further improve the method.

Evan: That makes sense.

Is there anything else they discuss in terms of the LatentPress mechanism?

Ashley: To summarize, LatentPress focuses on the key components of creating this compact representation, setting compression rates appropriately, and ensuring the trainable parts are kept minimal and interfacing directly without reconstructing text.

This streamlined process is what allows LatentPress to achieve the impressive results reported.

Evan: Alright, that wraps up our discussion of the Method section.

Evan: Now, Ashley, let's move on to the Experiment and Results section.

How did the authors validate LatentPress?

Ashley: The authors conducted several experiments to test LatentPress across multiple dimensions: accuracy, write cost, read cost, and trainable footprint.

They used two main benchmarks: LongMemEval and LongBench-QA.

Evan: Let's start with LongMemEval.

What did they aim to achieve with this benchmark?

Ashley: LongMemEval was used to evaluate the accuracy and transfer behavior for conversational memory.

The writers trained on generic UltraChat conversations were tested zero-shot on unseen memory-QA labels using frozen readers.

Evan: And what were the results?

Ashley: The results showed LatentPress achieving approximately 0.48 to 0.50 accuracy at compression rates of 4.6 to 7.7 times.

This performance was comparable to, and in some settings better than, uncompressed oracle evidence.

LatentPress significantly outperformed alternative compression methods such as text summaries and OCR-based compression.

Evan: That's impressive.

What about LongBench-QA?

Ashley: LongBench-QA was utilized to test the interface on long documents without the constrained role structure of conversations.

Here, LatentPress demonstrated matching or exceeding raw-context baselines at 4 to 8 times compression rates.

However, at the most aggressive compression rate of 16 times, performance did degrade.

Evan: It seems like it's quite robust.

Were there any additional insights from these experiments?

Ashley: Yes, they found that cross-domain transfer was only partially successful.

For example, on Qwen2.5-7B, the compressed reader exceeded the raw baseline at 4 times compression but dropped at higher rates.

In-domain training, however, lifted the sliced compression rates, showing that training directly on the target domain splits was more effective.

Evan: How about efficiency metrics?

How did LatentPress perform in terms of write and read costs?

Ashley: The efficiency section demonstrated significant speed advantages.

Generating encoded tokens took about 43 milliseconds per conversation, substantially faster than OCR-based or text summarization methods.

Read times from the compressed context were 5 to 9 times faster than reading from raw context or OCR caches.

Evan: What about the overall job completion time?

Ashley: In end-to-end job time comparisons, LatentPress was shown to be 6 to 13.7 times faster than the cold-cache OCR pipeline.

The training, prediction, and evaluation processes were notably more efficient.

Evan: Were there any findings on how the method generalizes across different frozen readers?

Ashley: Yes, they repeated the zero-shot comparison across different model scales and families, such as Qwen2.5-7B, Qwen3-8B, and Qwen3-1.7B.

LatentPress consistently outperformed uniform pooling methods and often surpassed the visual compression baseline.

This showed that the method’s effectiveness is not model-specific.

Evan: It sounds like LatentPress brings both efficiency and accuracy improvements.

Were there any specific cases where it struggled?

Ashley: One key struggle was at very high compression rates, such as 16 times where information loss impacts accuracy.

Other failure modes included unanswerable collapse in the absence of data, format artifacts, and repetition loops at high compression settings.

Evan: How do these findings impact practical applications of LatentPress?

Ashley: The practical impact is clear: LatentPress provides a scalable, efficient method for managing extensive contextual data, making it highly valuable for deployment in real-world AI systems that deal with long and complex histories or documents.

Evan: Any last points on the experiments before we wrap up this section?

Ashley: To sum up, the experiments demonstrated that LatentPress not only handles compression effectively, retaining crucial context for accurate question answering, but also significantly reduces the time and computational resources required for both writing and reading compressed context.

Evan: That wraps up the Experiment section of the paper.

Evan: Let's move on to the Related Work section.

What context does the paper provide in terms of previous research in this field?

Ashley: The paper situates LatentPress within a broader landscape of research on context compression for language models.

They compare and contrast their method with a variety of existing approaches according to several dimensions.

Evan: Can you give us an overview of these existing methods?

Ashley: Certainly.

Firstly, the paper discusses prior work on compressing context into continuous vectors.

Some well-known methods include Gist, AutoCompressor, and ICAE, which adapt entire large language model-scale readers or encoders.

For example, Gist focuses on masked attention mechanisms, while AutoCompressor employs recursive summary techniques.

Evan: How is LatentPress different from these approaches?

Ashley: LatentPress is unique in that it leaves the downstream decoder entirely frozen, training only a small reader-matched adapter.

This is in contrast to other methods that often require extensive training and adaptation of the language model itself.

Evan: What about visual compression methods?

Are they mentioned?

Ashley: Yes, the paper covers methods like DeepSeek-OCR, which involve rendering text as images and using optical character recognition for context compression.

These methods reconstruct text from images before feeding it into a language model, which can be slower and less efficient compared to LatentPress's direct-read approach.

Evan: Are there any other key distinctions made between LatentPress and earlier methods?

Ashley: LatentPress emphasizes a direct-read interface with variable-length context compression, allowing it to assign different compression rates based on the structure of the input.

Other methods like xRAG compress one retrieved passage into a single token, which limits their capability in dealing with multi-turn histories and long documents.

Evan: Are there any discussions on strategies for integrating compressed contexts into broader memory systems?

Ashley: Yes, the paper discusses how LatentPress can be integrated into long-term memory systems that manage extensive histories.

It complements systems like Generative Agents and MemoryBank by providing a learned soft-token representation to be used inside pipelines that handle retrieval, reflection, and conflict resolution.

Evan: Sounds like there's a lot of complementary research here.

What about token pruning methods?

Are they mentioned?

Ashley: Indeed.

The paper mentions token pruning techniques such as Selective Context and the LLMLingua family, which selectively drop or retain tokens based on their importance.

The idea here is similar to LatentPress in that it seeks to streamline the input for efficient processing, although the mechanisms differ.

Evan: How does LatentPress align with or differ from latent reasoning methods?

Ashley: Latent reasoning methods like Coconut and CRISP compress a model's reasoning into continuous states or distill its reasoning process.

LatentPress, however, focuses on compressing the input history rather than the internal reasoning trace, making it complementary to these latent reasoning approaches.

Evan: So, LatentPress is both distinctive and complementary depending on the context of usage.

Were there any specific limitations or future directions that connected LatentPress with related work?

Ashley: Yes, the authors acknowledge the limitations related to the fixed compression schedules and hint at the potential for dynamic compression policies driven by reinforcement learning.

This would place LatentPress within a growing body of work focusing on adaptive, learned compression techniques.

Evan: It's fascinating to see how LatentPress interacts with and contributes to existing research.

Any final thoughts from the Related Work section?

Ashley: Overall, the Related Work section highlights how LatentPress innovates by offering a lightweight, efficient method for context compression that stands out through its direct-read approach.

It opens up several avenues for further integration with existing techniques and systems, ensuring its relevance in the broader landscape of context compression research.

Evan: That's the end of the Related Work section.

Evan: Alright, Ashley, let's wrap things up.

What are the key contributions and takeaways from the LatentPress paper?

Ashley: The primary contributions of the LatentPress paper are threefold.

First, it introduces a novel direct-read soft-token interface that allows for efficient context compression without the need for text reconstruction.

This distinct approach enables faster and more effective reading of compressed context by frozen language model decoders.

Evan: That's a significant development.

What about their experimental results?

Ashley: The experimental results are compelling.

LatentPress demonstrates competitive accuracy, outperforming or matching raw context baselines at compression rates between 4 to 8 times.

It also shows superior efficiency, reducing token generation times and read latencies significantly compared to prior methods.

Evan: And how about its practical applications and future directions?

Ashley: LatentPress is highly practical for applications requiring long-term context management, such as conversational agents.

It also sets the stage for future work on dynamic compression policies, potentially leveraging reinforcement learning to optimize compression rates further.

Evan: In summary, LatentPress represents an innovative approach to context compression, balancing efficiency, and accuracy, with significant implications for the practical deployment of AI systems.

Ashley: Evan.

And with that, we conclude today's episode on Daily Paper Cast.

Thank you for joining us.

Evan: We hope you found the discussion insightful and enjoyable.

Make sure to tune in for our next episode, where we'll dive into more cutting-edge research in AI and NLP.

Ashley: See you next time, and keep exploring the fascinating world of AI research!

Evan: Goodbye!

Ashley: Bye-bye!