NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction

🤗 Upvotes: 98 | cs.CL

Authors:
Intern-NCP Team, Jiaqi Cao, Chiyu Chen, Shuang Cheng, Xu Cheng, Beiya Dai, Yufan Feng, Kewen Ge, Ruijun Ge, Jiayi Huang, Yang Jiao, Dahua Lin, Zhouhan Lin, Yifan Liu, Yuliang Liu, Biqing Qi, Mowen Ruan, Junzhe Shen, Yunchong Song, Hao Sun, Zhongbo Tian, Yixuan Wang, Rubin Wei, Jiaxin Xiong, Kangyu Yang, Qian Yao, Qi Zhang, Bowen Zhou

Title:
NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction

Arxiv:
http://arxiv.org/abs/2609.10715v1

Abstract:
We introduce NCP-ArchPreview, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP). Alongside NTP, the model learns through Next Concept Prediction (NCP) to predict discrete concepts that span multiple tokens, introducing an explicit and more challenging concept-level objective while preserving standard token-level autoregressive generation. NCP-ArchPreview builds a latent space by constructing a product-quantized concept vocabulary directly from its hidden states, and subsequently learns to predict future concepts via a dedicated Concept Module. These predicted concepts are then fed back to the token level to guide subsequent generation, with NTP and NCP trained jointly end-to-end. We scale this architecture to 8.9B parameters and train it on 5.73T tokens from the Dolma-3 dataset, marking the largest demonstration of a latent-space language model to date. Remarkably, by consuming only 51.3% of the total training tokens, NCP-ArchPreview achieves the final pretraining loss of OLMo-3-7B. Following full pretraining, it outperforms OLMo-3-7B by 2.45 points on the downstream macro-average, including a notable 5.99-point gain on GSM8K. Controlled experiments isolate a clear progression of performance gains stemming from both the latent architecture and the NCP objective. Furthermore, utilizing only 85% of the standard computation, NCP-ArchPreview approaches the training loss of a strictly parameter-aligned 8.9B baseline. The learned latent space remains highly valuable after the pretraining stage: updating just the 17M-parameter VQ module yields a novel, lightweight interface for domain adaptation, while a simple injection of concept representations into a DFlash2 drafter improves the mean accepted length by 4.17% with negligible overhead.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast, where we dive into the latest research papers in AI, NLP, Computer Vision, and more.

Ashley: Today, we're discussing a paper from the Hugging Face daily paper list of September 11, 2026, which has garnered 98 upvotes.

Evan: The paper is titled 'Moving towards Latent Space Language Models through Next Concept Prediction.' And leading this research are The Intern-NCP Team.

Ashley: Right.

The first two authors are from Shanghai AI Lab and LUMIA Lab at Shanghai Jiao Tong University.

Additionally, correspondence for this paper can be directed to [e-mail].

Evan: Let's dive into the introduction of this fascinating work.

Recent progress in generative AI has shown that the representation space where a model learns can be just as crucial as its parameter scale.

Ashley: Exactly.

In visual synthesis, latent diffusion models shift generation from raw pixels to compact continuous representations.

This has dramatically improved modeling efficiency and scalability.

Evan: And similarly, modern language models induce high-level abstractions, like semantic concepts and latent world representations, within their hidden states.

Ashley: However, under the standard Next Token Prediction or NTP paradigm, these abstractions emerge as a byproduct.

Supervision is limited to granular tokens without explicit objectives guiding how semantic structures unfold over multiple tokens.

Evan: So to address this gap, this paper introduces NCP-ArchPreview.

Could you explain what that is?

Ashley: NCP-ArchPreview is a novel latent-space foundation model architecture.

It incorporates the direct prediction of discrete concepts spanning multiple tokens into the pretraining stage, effectively combining traditional NTP with Next Concept Prediction, or NCP.

Evan: The way I understand it, this joint pretraining with both NTP and NCP allows NCP-ArchPreview to scale latent-space language modeling up to 5.73 trillion pretraining tokens.

What does this mean for the field of language modeling?

Ashley: It means establishing the architectural viability and scalability of this paradigm at a frontier scale.

Essentially, it's a significant step toward more efficient and robust language modeling.

Evan: Can we break down how the latent representations are formed and what they are trained to represent?

Ashley: Certainly.

Previous research efforts, such as the Hourglass Transformer and MegaByte models, have utilized fixed hierarchical structures with predefined resolutions or patches.

On the other hand, models like Byte Latent Transformer, DLCM, and H-Net use dynamic or input-adaptive chunking.

This allows the size of latent boundaries or patches to change with the input.

Evan: And how does NCP-ArchPreview differ from those approaches?

Ashley: NCP-ArchPreview differs by not only forming latent representations dynamically but also assigning them meaningful predictions.

Specifically, it predicts future concepts directly in a learned concept vocabulary using a product-quantized approach.

Evan: Interesting.

So, it integrates latent predictions with concept-level objectives.

Does this impact the overall model performance?

Ashley: Yes, and in a positive way.

NCP-ArchPreview uses these predicted concepts to enhance token-level autoregressive generation, leading to better optimization efficiency.

It achieves the final pretraining loss of a comparable model, OLMo-3-7B, by consuming only 51.3% of the training tokens.

Evan: That's impressive.

What other performance metrics stand out from this introduction?

Ashley: Interestingly, NCP-ArchPreview not only matches but also outperforms the OLMo-3-7B on certain downstream macro-average metrics.

For example, it shows a significant 5.99-point gain on the GSM8K dataset, which is used for benchmarking mathematical problem-solving.

Evan: These improvements sound promising.

Could you elaborate on the implications for future model training and domain adaptation?

Ashley: One of the key implications is that the learned latent space remains valuable beyond pretraining.

By updating just the 17M-parameter Vector Quantization module, the model can efficiently adapt to specific domains.

Additionally, concepts can be injected into a drafter model to improve generation efficiency, such as extending the mean accepted length by 4.17%.

Evan: It sounds like NCP-ArchPreview could redefine efficient model training strategies.

Is this the paper's main contribution?

Ashley: In essence, yes.

By proving the viability of joint token and concept modeling at scale, NCP-ArchPreview presents latent-space prediction not just as an auxiliary objective but as a highly efficient and scalable architectural blueprint for future foundation models.

Evan: That wraps up the introduction section of the paper.

Let's delve deeper into the model architecture in the next part of our discussion.

Evan: Alright, let's now dive into the methods employed in this paper, notably the architecture of NCP-ArchPreview.

Ashley: NCP-ArchPreview introduces a latent-space language model that combines token-level modeling with concept-level prediction in a discrete latent space.

This method fundamentally changes how language models are traditionally trained.

Evan: So, how exactly is this architecture organized?

Ashley: The backbone comprises three core modules: a Token Encoder, a Concept Module, and a Token Decoder.

The Token Encoder is responsible for producing token-level representations.

Evan: And what happens after that?

Ashley: The model then compresses contiguous groups of tokens into continuous concept representations.

These are processed by the Concept Module, which predicts the next concept.

Evan: How does the model deal with predicted concepts?

Ashley: Good question.

The Token Decoder uses these predicted concepts, along with token-level representations, to predict the next token.

To ensure smooth flow of information, the architecture incorporates hierarchical residual connections both within and across the modules.

Evan: That sounds like a complex system.

Can you elaborate more on the concept vocabulary construction?

Ashley: Of course.

The model uses Vector Quantization (VQ) to construct a discrete concept vocabulary.

This involves partitioning each concept vector into several feature segments, which are then quantized using smaller codebooks.

Evan: So, each concept is represented by a combination of codebook entries?

Ashley: Exactly.

For instance, each concept vector is divided into segments, each of which is mapped to its nearest codeword.

These segments are then concatenated to form the complete quantized concept.

Evan: Interesting.

How is the next concept predicted?

Ashley: The Concept Module predicts the next concept autoregressively within the learned structured latent space.

It produces a probability distribution over codebook entries for each segment, forming a differentiable predicted concept via a weighted combination of these entries.

Evan: So the predicted concepts are then integrated into the token stream?

Ashley: Precisely.

The predicted concepts are causally shifted and repeated to match the token sequence length.

They are then added to the token states, forming fused representations for the Token Decoder to predict the next token.

Evan: This hierarchical residual connection mechanism within and across modules, what are its benefits?

Ashley: Hierarchical residual connections enable the model to blend information across depths and modules effectively.

This consists of Intra-Module Residual Connections (IRC), which combine representations from different layers within the same module, and Cross-Module Residual Connections (CRC), which transfer information between modules.

Evan: How do these residual connections work within a module?

Ashley: For IRC, each residual state is a weighted combination of representations from current and earlier layers, initialized to emphasize recent layers but trainable to balance historical layer contributions dynamically.

These weights are produced by a lightweight multi-layer perceptron (MLP).

Evan: And CRC?

Ashley: CRC transfers information between modules aligned at different sequence granularities.

The concept sequence is aligned using chunking or repetition and integrated into the target module through another MLP, producing normalized coefficients to combine source-depth representations.

Evan: That explains the architecture.

What about the training objectives and the optimization strategies?

Ashley: The model is jointly trained using the Next Token Prediction (NTP) and Next Concept Prediction (NCP) objectives.

The NTP objective supervises token-level predictions via standard causal next-token prediction, while the NCP objective supervises the Concept Module to predict the next concept within the learned concept vocabulary.

Evan: How are these objectives combined in training?

Ashley: The total loss function is a weighted sum of the NTP loss, the NCP loss, and a VQ loss that ensures the codebook entries fit the distribution of continuous concept representations.

Specifically, the overall loss is LNTP + αLNCP + βLVQ.

Evan: Which optimizers are employed for this training?

Ashley: The model uses the Moonlight Muon optimizer, a variant of the Muon optimizer, for matrix-valued parameters.

Other parameters, like embeddings and biases, use AdamW.

The model applies a learning rate of 6 x 10^-5 and follows a cosine scheduler, similar to OLMo-3-7B.

Evan: That covers the architecture and training.

What datasets are used for pretraining this model?

Ashley: NCP-ArchPreview is pretrained on the Dolma-3 dataset, consisting of 5.73 trillion tokens.

The training follows a staged data curriculum, initially training on Dolma 3 Mix, followed by further training on Dolma 3 Dolmino.

Evan: After pretraining, how is the model evaluated?

Ashley: The evaluation follows the OLMo evaluation protocol, encompassing 30 benchmark families.

These benchmarks assess various dimensions such as factual knowledge, mathematical reasoning, code generation, commonsense reasoning, reading comprehension, and language modeling.

Evan: Could you give examples of these benchmarks?

Ashley: Sure, some of the benchmarks include MMLU, GSM8K, MATH-500, HumanEval, MBPP, ARC, and HellaSwag.

Each group includes tasks focused on specific domains, ensuring comprehensive evaluation.

Evan: It sounds like they have covered a lot of ground with their evaluations.

Any specific adaptations for different domains?

Ashley: Indeed.

For example, the VQ training adapts the concept vocabulary for specific domains such as code, mathematics, and knowledge.

They performed separate stages of domain-specific training for Magicoder, Orca-Math, and TriviaQA-RC datasets.

Evan: And how are the general capabilities measured?

Ashley: General capabilities are evaluated using benchmarks like MMLU, ARC-Easy, HellaSwag, Natural Questions, and SciQA.

Together, this thorough evaluation framework provides a complete picture of model performance across various domains.

Evan: That concludes the Method section.

It's clear that NCP-ArchPreview has a robust and innovative architecture.

Ashley: Let's move forward with the empirical validations and results in the next segment.

Evan: Let's delve into the experimental setup and results of the NCP-ArchPreview model.

How did they design their experiments?

Ashley: For their experiments, they pre-trained NCP-ArchPreview using the Dolma-3 dataset and evaluated it based on a series of benchmarks.

The setup is structured to compare NCP-ArchPreview with OLMo-3-7B across various stages of training.

Evan: Did they perform any ablation studies to analyze the contributions of different components?

Ashley: Yes, they did extensive ablation studies to isolate the contributions of the Concept Module, the hierarchical residual connections, and the NCP loss.

They compared results against standard and size-aligned baselines of OLMo-3-7B.

Evan: So, what were their findings in terms of training loss performance?

Ashley: NCP-ArchPreview displayed a lower training loss compared to OLMo-3-7B throughout both Stage-1 and Stage-2.

In Stage-1, the loss gap widened progressively, achieving a final gap of 0.091.

NCP-ArchPreview reached the final training loss of OLMo-3-7B after consuming only 51.3% of the training tokens.

Evan: That's almost twice as efficient in terms of convergence speed.

Did Stage-2 show similar improvements?

Ashley: Indeed, it did.

In Stage-2, NCP-ArchPreview continued to show a lower final loss, achieving it with only 66.2% of the training tokens, indicating a 1.51× convergence speedup compared to OLMo-3-7B.

Evan: How did the model perform on downstream tasks?

Ashley: NCP-ArchPreview outperformed OLMo-3-7B on almost all evaluated benchmarks at both stages.

For instance, in Stage-1, it achieved an average score 2.45 points higher across multiple downstream datasets.

The improvements were especially notable in MATH, Code, and MC-Non-STEM categories.

Evan: Was there a similar trend observed in Stage-2?

Ashley: Yes, though the magnitude varied.

In Stage-2, NCP-ArchPreview achieved a 0.59-point higher macro-average, with improvements seen primarily in mathematics, MC-Non-STEM, and general question answering benchmarks.

Evan: Let's talk specifics.

Which downstream tasks showed the largest gains?

Ashley: The GSM8K dataset, which evaluates mathematical problem-solving, showed the most significant improvement with a 5.99-point gain in Stage-1.

This indicates that the concept-level prediction provides substantial benefits for complex reasoning tasks.

Evan: And how did the component ablations affect the training loss?

Ashley: The ablation studies were insightful.

Introducing the Concept Module alone improved the training loss over the vanilla OLMo-3-7B baseline.

Adding the hierarchical residual connections further reduced the loss.

Finally, incorporating the NCP loss yielded the most significant improvement, showing that each component of NCP-ArchPreview contributes to its efficiency.

Evan: Did they also explore different configurations for residual connections?

Ashley: Yes, they experimented with various hierarchical residual connection designs.

They found that combining intra-module and cross-module residual connections provided the largest loss reduction with minimal computational overhead.

Evan: Scaling laws and model analysis were also part of their investigation.

What did they find?

Ashley: In their scaling experiments, NCP-ArchPreview showed a 1.74x computational efficiency compared to OLMo-3.

The scaling laws revealed that NCP-ArchPreview maintains its performance advantages across various compute budgets.

Evan: That sounds impressive.

Were there any stability issues during training?

Ashley: Initially, they observed numerical instabilities with the OLMo-3-7B configuration, notably persistent growth in attention logits and gradient-norm spikes.

By employing per-head Q/K normalization, these instabilities were effectively mitigated, especially in the Muon optimizer baseline.

Evan: So, what's the takeaway from all these experiments?

Ashley: The experiments robustly validate the efficacy of NCP-ArchPreview's architecture.

Joint token and concept modeling prove to be highly efficient, enhancing both pretraining convergence and downstream performance.

Furthermore, the exploration into domain adaptation through lightweight VQ training suggests practical applications for specialized tasks without significant overhead.

Evan: It's clear that these results could pave the way for future advancements in language modeling.

And that wraps up the Experiment section.

Ashley: Exactly.

Up next, we'll dive into the limitations, implications, and future directions mentioned in the paper.

Evan: Moving on, let's discuss the Related Work section of the paper, which offers a comprehensive review of previous efforts and methodologies that laid the groundwork for NCP-ArchPreview.

Ashley: The paper situates its contributions within three primary research domains: abstract-level prediction, hierarchical and latent-space language modeling, and residual connections in deep transformers.

Evan: Starting with abstract-level prediction, what are the key points?

Ashley: Abstract-level prediction involves models that learn representations by predicting latent targets rather than reconstructing fine-grained inputs.

Joint-embedding predictive architectures, or JEPA, are a significant development in this field.

These models learn to predict latent, high-level representations that retain semantic structure while discarding unnecessary input details.

Evan: That's intriguing.

How has this approach been applied in previous work?

Ashley: For instance, the Large Concept Models map complete sentences into a continuous embedding space, applying abstract-level autoregressive modeling to language.

By targeting these higher-level abstractions, these models move beyond merely predicting surface tokens, though they typically rely on a separate encoder to define the prediction space.

Evan: And how does NCP-ArchPreview build on these methods?

Ashley: NCP-ArchPreview extends this paradigm by integrating concept-level objectives into the language model itself.

This allows for a more seamless prediction space that aligns directly with token-level predictions, providing a more cohesive and efficient model architecture.

Evan: Next, let's delve into hierarchical and latent-space language modeling.

What's the background there?

Ashley: Hierarchical language models compress sequences into higher-level representations, which are then expanded back into fine-grained outputs.

Models like the Hourglass Transformer and MegaByte exemplify this approach.

They operate by first compressing token sequences and then performing global modeling over these compressed representations.

Evan: How do they differ in their handling of input structures?

Ashley: The Hourglass Transformer uses fixed hierarchical structures, whereas models like Byte Latent Transformer and H-Net employ dynamic chunking.

This means they adjust latent boundaries based on input, offering adaptability to varying data structures.

Evan: NCP-ArchPreview seems to follow a different route.

How does it manage these representations?

Ashley: Instead of fixed structures, NCP-ArchPreview constructs a dynamic, product-quantized concept vocabulary directly from token representations.

This allows the model to maintain a balance between high-level concept prediction and traditional token-level next-token prediction.

Evan: Sounds like a sophisticated mechanism.

What about the related work in residual connections?

Ashley: Residual connections ensure smooth information flow across network layers, and have been standard in deep learning models.

Various enhancements have been proposed, such as DenseFormer, which averages previous block outputs, and DeepCrossAttention, which mixes layer outputs using depth-wise cross-attention.

Evan: And how do these fit into the context of NCP-ArchPreview?

Ashley: NCP-ArchPreview employs hierarchical residual connections inspired by MUDDFormer, incorporating both intra-module and cross-module connections.

This design allows the model to dynamically incorporate historical information from different depths and modules, facilitating a highly efficient learning process.

Evan: Could you give an example of a model with depth-wise cross-layer connections?

Ashley: Sure.

Attention Residuals apply softmax attention over preceding layer outputs, and the blockwise variant scales this for larger models.

Another example is Depth-Attention, which mixes earlier value states through the existing QKV pathway within self-attention modules.

Evan: How does NCP-ArchPreview's approach compare to these methods?

Ashley: NCP-ArchPreview's implementation of hierarchical residual connections goes a step further by combining both types of connections—within modules and across modules.

This offers a more integrated approach to reusing earlier representations while maintaining efficient data flow.

Evan: That wraps up an extensive look at the related work.

It's clear that NCP-ArchPreview builds on a robust foundation of prior research, enhancing and integrating various advanced concepts effectively.

Ashley: Indeed.

With this strong basis, NCP-ArchPreview stands as a significant advancement in language modeling.

That concludes the Related Work section.

Evan: Let's wrap up today's episode with a summary of the key contributions and takeaways of the NCP-ArchPreview paper.

Ashley: Firstly, NCP-ArchPreview introduces a novel architecture, integrating Next Token Prediction and Next Concept Prediction.

This approach leverages discrete concept vocabularies formed through vector quantization, enhancing the model's ability to predict high-level concepts and improving token-level autoregressive generation.

Evan: The architecture utilizes a dynamic, product-quantized concept vocabulary and hierarchical residual connections both within and across modules.

This design allows for efficient information flow and integrates multiple levels of abstraction within the model.

Ashley: In terms of performance, experiments showed that NCP-ArchPreview achieves faster convergence rates and outperforms the OLMo-3-7B model on various benchmarks.

It reaches the final training loss of OLMo-3-7B using only 51.3% of the training tokens in Stage-1 and 66.2% in Stage-2, indicating higher efficiency.

Evan: Additionally, the model demonstrated significant improvements in downstream tasks, particularly in mathematical problem-solving and code generation, showcasing a clear advantage in complex reasoning tasks.

Ashley: The paper also highlights the model's adaptability through lightweight domain-specific training using the VQ module, which maintains general capabilities while enabling efficient domain adaptation.

Evan: In summary, NCP-ArchPreview offers a highly efficient and scalable architecture that advances the field of language modeling substantially.

It's a noteworthy contribution that sets the stage for future research and development.

Ashley: That's it for today's episode of Daily Paper Cast.

We hope you found this discussion insightful.

Evan: Be sure to join us next time as we explore more groundbreaking research.

Don't forget to subscribe and share with your colleagues.

Ashley: Thanks for listening, and we'll catch you in the next episode!