All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts

🤗 Upvotes: 25 | cs.CV

Authors:
Xingsong Ye, Yongkun Du, Jiaxin Zhang, Zhixian Li, Chong Sun, Chen Li, Jing Lyu, Lianwen Jin, Zhineng Chen

Title:
All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts

Arxiv:
http://arxiv.org/abs/2609.24058v1

Abstract:
Multilingual scene text recognition (STR) remains challenging due to the scarcity of training data for most languages and the difficulty of serving diverse scripts within a single model. Existing solutions either deploy one recognizer per language, inflating cost and introducing error accumulation, or rely on massive vision-language models (VLMs) that are expensive and still inaccurate on many scripts. In this work, we pursue an all-in-one multilingual recognizer that is simpler than per-language experts, lighter than VLMs, and more accurate than both. First, we construct TextMuSS-10M, a large-scale synthetic scene text dataset spanning 10 scripts and 229 languages. It provides balanced and sufficient supervision where real data is unavailable. Second, we propose ScriptMoE, a script-aware Mixture-of-Experts (MoE) architecture. It shares a single visual encoder and replaces the dense decoder with a sparse MoE block, which consists of an image-level router dispatches each image to the top-2 script-aligned experts and a shared expert absorbs cross-script knowledge. Extensive experiments on our assembled TextMuSS-Bench (10 scripts, 10,899 images) show that ScriptMoE achieves the highest accuracy of 82.06%, outperforming the strongest STR baseline by 1.31%. On the CC-OCR end-to-end multilingual task, replacing only the recognizer in PP-OCRv5 with ScriptMoE lifts F1 score from 65.71% to 80.89%, slightly surpassing the best VLM (80.73%) at a fraction of the parameter count.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast!

Evan: Today's paper is from the Hugging Face daily paper list of September 23, 2026, with 25 upvotes.

Ashley: The title is 'All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts'.

Evan: The first two authors are Xingsong Ye and Yongkun Du.

The corresponding author is Zhineng Chen from Fudan University.

Ashley: Great, let's dive into the Introduction section.

Evan: So, Ashley, what's the background behind this research on multilingual scene text recognition?

Ashley: Scene text recognition, or STR, is a subfield of optical character recognition, aiming to decode text from complex images.

Although accuracy for high-resource languages like English and Chinese has improved dramatically, the challenge remains for many other scripts and languages, which lack sufficient training data.

Evan: Right, so what's the problem with using existing methods for these lower-resource languages?

Ashley: There are mainly two approaches—expert OCR systems and massive vision-language models, or VLMs.

Expert systems use multiple models, often one per language, which increases cost and complexity.

VLMs unify many languages into a single model but come with high parameter counts and still struggle with certain scripts.

Evan: So, the paper proposes an all-in-one solution.

Can you explain their objectives?

Ashley: The authors aim to create a unified multilingual recognizer that is simpler than per-language expert systems, lighter than VLMs, and more accurate than either.

To achieve this, they introduce two main contributions: a new dataset called TextMuSS-10M and a novel architecture, ScriptMoE.

Evan: Interesting!

So TextMuSS-10M is a large-scale synthetic scene text dataset that covers 10 scripts and 229 languages, addressing the scarcity of training data.

Ashley: Exactly.

The dataset ensures balanced and sufficient data for scripts where real data is lacking.

Moving to their architecture, ScriptMoE stands for Script-aware Mixture-of-Experts.

It combines a single visual encoder with a sparse mixture-of-experts decoder designed to route images to the appropriate experts based on the script.

Evan: How does this ScriptMoE architecture exactly work?

Ashley: Basically, ScriptMoE uses an image-level router to dispatch each image to the top two script-aligned experts while sharing a common expert to absorb cross-script knowledge.

This design balances the model capacity across scripts efficiently.

Evan: So, it improves accuracy while staying lightweight.

That's impressive.

Were there any specific results that stood out?

Ashley: Yes, their extensive experiments show that ScriptMoE achieves an accuracy of 82.06% on their developed TextMuSS-Bench, outperforming the best existing STR models.

Moreover, when integrated into an end-to-end OCR pipeline, it raised the F1 score significantly on a challenging multilingual task.

Evan: Wow, those metrics definitely underscore the potential of this approach.

Any final contributions worth mentioning?

Ashley: Besides the dataset and architecture, they also provide extensive evaluations and ablations to validate their claims, systematically comparing ScriptMoE against various strong baselines and generalist systems.

Evan: That's the end of the Introduction section.

Evan: Alright, let's move on to the Method section.

Ashley, how do the authors tackle the challenge of creating a unified multilingual recognizer?

Ashley: The authors take a dual approach, combining a novel dataset and an innovative model architecture to address the challenges.

First, let's talk about the dataset, TextMuSS-10M.

Evan: Sure, what makes TextMuSS-10M unique?

Ashley: TextMuSS-10M is a synthetic dataset designed to provide balanced training data for 10 scripts, covering 229 languages.

The authors created it to bridge the gap in available training data for low-resource languages.

Evan: Synthetic as in computer-generated data, right?

How did they ensure it's useful?

Ashley: Exactly.

They leveraged a strong STR synthesis engine called UnionST to generate one million samples per script.

Each script's samples were carefully constructed to include a mix of real words, shuffled characters, rare characters, and sentences drawn from news corpora.

This way, the data isn't just large-scale but also diverse and representative.

Evan: Got it.

And how do they evaluate the performance of their model using this data?

Ashley: They constructed a benchmark called TextMuSS-Bench, which includes real scene text images for evaluation.

This benchmark has 10,899 images spanning the same 10 scripts used in TextMuSS-10M.

They've included both existing datasets and newly collected images to ensure a comprehensive evaluation.

Evan: That sounds thorough.

So, what's the next part of their method?

Ashley: Next, they introduce their innovative architecture, ScriptMoE.

This stands for Script-aware Mixture-of-Experts.

It couples a robust visual encoder with a sparse mixture-of-experts decoder to manage the complexity of multiple scripts efficiently.

Evan: How does ScriptMoE distribute its workload when processing an image?

Ashley: ScriptMoE uses an image-level routing mechanism.

An image-level router looks at the visual features of each image and then dispatches it to the top two most relevant script-aligned experts, while also engaging a shared expert that handles cross-script knowledge.

Evan: So, it's like having specialists for each script and a generalist to handle common elements?

Ashley: Exactly.

By assigning specific experts to different scripts, the model achieves a higher level of accuracy without expanding into an overly large and inefficient system.

Evan: Interesting.

Can you break down how this architecture is implemented?

Ashley: Sure.

The backbone of the model is the SVTRv2 encoder, which is a hierarchical visual encoder.

It's followed by a script-aware Transformer decoder, where the traditional dense feed-forward network is replaced with the MoE-based feed-forward network.

This network comprises multiple Transformer blocks with attention mechanisms and gated mixture-of-experts FFNs.

Evan: What makes the mixture-of-experts FFNs special here?

Ashley: Well, during the training, an image is routed to the top two experts based on script alignment, meaning only a subset of the parameters (those of the selected experts) are activated per image.

This selective activation helps in managing computational costs while ensuring specialized processing for different scripts.

Evan: And how do they ensure these experts learn the right thing?

Ashley: They use a script-classifier head during training to guide the experts.

This classifier supervises the learning by ensuring that the routing aligns well with the scripts, providing script-aware specialization.

Evan: That's quite clever.

So, it's the balance of specialization and generalization that makes this work.

Ashley: Exactly.

They also divide the 10 scripts into four groups based on character morphology similarities to optimize expert capacity.

For example, the alphabet group includes Latin and Cyrillic scripts, while the CJK group includes Chinese, Japanese, and Korean.

Evan: Fascinating!

And how does the model handle training and inference?

Ashley: Training involves using cross-entropy loss for token-level autoregressive learning, along with a regularization term to guide the script classification.

During inference, the model uses greedy decoding to predict characters, balancing accuracy and efficiency.

Evan: What are the key metrics they use to evaluate the model's performance?

Ashley: They focus on word accuracy as the primary metric, emphasizing exact string matches, which are crucial for downstream applications.

They also report normalized edit distance as a complementary metric to capture character-level discrepancies.

Evan: Got it.

So, it's a well-rounded evaluation.

This structure seems quite robust.

Anything else notable in their method?

Ashley: They conducted extensive ablation studies to isolate the impact of each component and fine-tune hyperparameters, further validating the architecture.

They also compared their model against various competitive baselines to highlight its advantages.

Evan: It seems they've covered all bases methodologically.

Well, that wraps up the Method section.

Evan: Let's now dive into the Experiment and Results section.

Ashley, what did the paper find in terms of performance?

Ashley: Well, the authors conducted extensive experiments to validate ScriptMoE's effectiveness across different scripts.

They mainly compare its performance against other state-of-the-art multilingual STR models and general OCR systems.

Evan: Alright.

How do they frame these comparative results initially?

Ashley: The core evaluation is based on their benchmark, TextMuSS-Bench, which includes 10,899 images spanning the 10 scripts covered by TextMuSS-10M.

They benchmark ScriptMoE against 15 representative STR models and 9 general OCR systems.

Evan: That's quite comprehensive.

What were the key findings from these comparisons?

Ashley: ScriptMoE showed an average accuracy of 82.06% across all scripts in TextMuSS-Bench, outperforming the strongest baseline, which achieved 80.75%.

The improvements were especially noticeable in low-resource scripts like Arabic, Thai, and Tibetan, where it showed a 2-3 percentage point increase.

Evan: Those improvements definitely sound significant.

How does ScriptMoE compare to these models in terms of activated parameters?

Ashley: Interestingly, ScriptMoE activates only 45.85 million parameters per image, one to two orders of magnitude fewer than many vision-language models, making it both efficient and accurate.

Evan: Okay, so it's efficient in terms of computational cost as well.

What about the performance on an end-to-end multilingual OCR task?

Ashley: They evaluated ScriptMoE in an end-to-end pipeline on the CC-OCR multilingual task, where it was integrated with PP-OCRv5 as the detector.

The F1 score of this combined system jumped from 65.71% to 80.89%, slightly outperforming the best vision-language model, which scored 80.73%, but at a fraction of the parameter count.

Evan: That's quite a leap in performance.

Did they highlight any other interesting findings from their experiments?

Ashley: Yes.

They noted a cross-script trade-off where optimizing aggregate accuracy sometimes sacrifices peak performance for individual scripts.

Additionally, they found ScriptMoE effectively mitigates the Latin-Cyrillic character confusion issues, which other models struggled with.

Evan: Did they provide any qualitative comparisons or case studies?

Ashley: They did.

ScriptMoE consistently produced correct results where other models often failed, especially on complex multilingual text images.

Examples in the paper show how it successfully identified text in scripts like Bangla, Korean, and Arabic with minimal errors.

Evan: That's impressive.

Did they conduct any additional ablation studies to explore the impact of specific components?

Ashley: Indeed.

Their ablation studies dissected the role of both real and synthetic data, finding that while synthetic data is crucial for low-resource scripts, combining it with real data provided the best performance.

They also explored the importance of the shared expert and script-classification signal, confirming both are key to the model's efficiency and accuracy.

Evan: Were there any extended ablations reported?

Ashley: Yes, they performed fine-grained sweeps over several hyperparameters like script-classification loss weight and router jitter.

These detailed experiments helped them validate and fine-tune the model configuration, ensuring robust performance across scripts.

Evan: It seems like they've left no stone unturned.

Any specific limitations noted in their findings?

Ashley: Yes, they pointed out some remaining issues like the domain gap between synthetic and real data and the challenge of Cyrillic-Latin homoglyph confusion.

They suggest future work focusing on real-world data collection and stronger text detectors to address these gaps.

Evan: Fantastic.

So it sounds like ScriptMoE has strong potential but still has areas for further improvement.

Ashley: Precisely.

And with that, we've covered the Experiment section.

Evan: Now that we've unpacked the experiments, let's connect this study to the broader context of existing research.

Ashley, what does the Related Work section tell us?

Ashley: The Related Work section covers several key areas that are relevant to the study.

The authors categorize them primarily into scene text recognition models, multilingual OCR approaches, and script-aware techniques.

Evan: Okay, let's break these areas down.

First, what's discussed about scene text recognition models?

Ashley: Scene text recognition models, or STR models, are generally divided into autoregressive and non-autoregressive methods.

Autoregressive models, like those based on transformers, decode text token-by-token and are known for their accuracy but slower speed.

Non-autoregressive models, including those using Connectionist Temporal Classification, or CTC, and parallel decoding techniques, show faster performance but often lack the expressiveness of language priors found in autoregressive methods.

Evan: And how does this distinction impact multilingual text recognition?

Ashley: Well, both families were initially designed for monolingual settings, predominantly English and Chinese.

When these models are used for multilingual text recognition, their flaws become more apparent.

Non-autoregressive models, for instance, degrade significantly on morphologically rich scripts like Hindi and Tibetan due to their vision-centric approach and lack of language priors.

In contrast, autoregressive models handle joint multi-script recognition better but face imbalances due to a shared dense decoder.

Evan: Interesting, so the paper builds on these insights.

What about the multilingual OCR aspect?

Ashley: The landscape of multilingual OCR has evolved primarily through competitions that established their data and evaluation protocols.

Early approaches like E2E-MLT, introduced a systematic pipeline for multilingual OCR.

Later methods, such as Multiplexed TextSpotter, deployed script-specific recognition heads, increasing complexity by requiring script identification for each word.

Similarly, SARN enhanced recognizers with script information to refine character features.

Evan: So, many of these approaches either need script classification steps or specific dataset designs, which can be cumbersome.

How does this study streamline these efforts?

Ashley: Exactly.

The paper by Ye et al.

innovates with ScriptMoE, which uses a single visual encoder and a sparse decoder managed by an image-level router, circumventing the need for intermediate script classification steps.

This ensures a unified model capable of handling various scripts more efficiently.

Evan: How about script-aware methods?

Were there notable advancements mentioned?

Ashley: Yes, script-aware methods have indeed seen advancements.

Incremental multilingual text recognition, or IMLTR, explored cross-language knowledge sharing within multiplexed structures.

MRN's work maintained separate feature extractors per language and activated language-specific experts using distinct character classifiers.

However, their parameter counts scaled linearly with the number of languages, which our paper's ScriptMoE aims to address more efficiently by utilizing a script-aware mixture-of-experts model.

Evan: So, ScriptMoE integrates these principles by activating expert parameters on demand, similar to large language models and VLMs, but without the linear parameter growth seen in past models.

Ashley: Exactly.

By leveraging a script-aware, sparse mixture-of-experts approach, ScriptMoE achieves high accuracy and efficiency in multilingual text recognition, presenting a significant step forward based on the insights and limitations of earlier works.

Evan: It's fascinating to see how ScriptMoE builds upon and diverges from previous models.

Were there any other related studies that stood out?

Ashley: Among other notable mentions, CLI-STR highlighted the decisive role of data scale over linguistic similarity in multilingual text recognition, which further motivated the synthetic dataset approach used in our paper.

Additionally, general multilingual OCR systems and recent VLMs, such as PP-OCRv5, have pushed the boundaries but still lag behind specialized recognizers, reinforcing the need for the proposed all-in-one model.

Evan: A comprehensive look at prior models definitely grounds our understanding of ScriptMoE's innovations.

That's the end of the Related Work section.

Evan: Alright, we're nearing the end of this episode.

Let's summarize the key contributions and takeaways from today's paper.

Ashley: This paper made significant strides in multilingual scene text recognition by addressing two core challenges: the scarcity of training data for many scripts and the need for a unified model that handles diverse scripts efficiently.

Evan: One of their main contributions is the TextMuSS-10M dataset, right?

Ashley: Exactly.

TextMuSS-10M is a synthetic dataset covering 10 scripts and 229 languages.

It's designed to provide balanced and comprehensive training data where real-world data is lacking, ensuring robust training for diverse scripts.

Evan: And their innovative model, ScriptMoE, also shines in this study.

Ashley: That's right.

ScriptMoE, which stands for Script-aware Mixture-of-Experts, employs an image-level router to dispatch images to the top script-aligned experts, optimizing both performance and efficiency.

By balancing specialized experts with a shared common expert, it achieves high accuracy without expanding into an overly large system.

Evan: In terms of real-world application, ScriptMoE demonstrated significant accuracy improvements in their experiments, particularly for low-resource scripts like Arabic, Thai, and Tibetan.

It also boosted the performance of end-to-end multilingual OCR tasks.

Ashley: The ablation studies provided deeper insights into how each component contributes to the model's overall performance, validating their design choices.

And while there are remaining challenges, like domain gaps and character confusion issues, the study offers concrete steps towards future improvements.

Evan: Well, that wraps up our discussion on the paper 'All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts.' Thanks for tuning in to Daily Paper Cast!

Ashley: We hope you found today's episode insightful.

Don't forget to join us next time for more exciting research discussions.

Evan: Until then, stay curious and keep exploring!