Multilingual support

[decadance-dance]

🚀 The feature

Support of multiple languages (accordingly VOCABS["multilingual"]) by pretrained models.

Motivation, pitch

It would be great to use models which supports multiple languages because it significantly improve user experience in various cases.

Alternatives

No response

Additional context

No response

[felixdittrich92]

Hi @decadance-dance 👋,

Have you already tried:
docTR: https://huggingface.co/Felix92/doctr-torch-parseq-multilingual-v1
OnnxTR: https://huggingface.co/Felix92/onnxtr-parseq-multilingual-v1
? :)

Depends a bit if there is any data from mindee we could use.
Question goes to @odulcy-mindee ^^

[decadance-dance]

Hi, @felixdittrich92
I used docTR more than half year but have never faced this multilingual model, lol.
So, I am gonna try it, thanks.

[felixdittrich92]

Ah let's keep this issue open there is more todo i think :)

[felixdittrich92]

Hi, @felixdittrich92 I used docTR more than half year but have never faced this multilingual model, lol. So, I am gonna try it, thanks.

Happy about an feedback how it works for you :)
The model was fine tuned only on synth data.

[odulcy-mindee]

Depends a bit if there is any data from mindee we could use.
Question goes to @odulcy-mindee ^^

Unfortunately, we don't have such data

[felixdittrich92]

@decadance-dance
For training such recognition models i don't see a problem.. we can generate synth train data and need in a best case only real val samples.
But for detection we would need real data that's the main issue.

In general we would need the help of the community to collect documents (newspaper, receipt photos, etc.) in divers langauges (can be unlabeled). / This would need a license to sign that we can freely use this data.
With enough divers data we could use Azure Doc AI for example to pre-label this data.
Later on i wouldn't see an issue to open source this dataset.

But not sure how to trigger such "event" 😅 @odulcy-mindee

[nikokks]

Hello =)
I found some public dataset for various tasks
english documents
mathematics documents
latex ocr
latex ocr
chinese ocr
chinese ocr
chinese ocr

[nikokks]

Moreover it should be interesting for Chinese detection models to add multiple recognition data in the same image without intersection. This should help for a Chinese detection model to perform better without real detection data.
Anyone interested in creating random multilingual data for detection models (hindi, chinese, etc.) ?

[felixdittrich92]

Hi @nikokks 😃
Recognition should not be such a big deal i found already a good way to generate such data for fine tuning.

To collect multilingual data for detection is troublesome because it should be real data (or if possible really good generated ones / for example with a fine tuned FLUX model maybe !?)
We need different kinds of layouts/documents (newspapers, invoices, receipts, cards, etc.) so the data should come close to real use cases (not only scans also document photos etc.)
:)

[decadance-dance]

To collect multilingual data for detection is troublesome because it should be real data

Do you can estimate how much data we need to provide multilingual capabilities on the same level as only english ocr is?

[felixdittrich92]

Hi @decadance-dance 👋,

I think if we could collect ~100-150 different types of documents for each language we would have a good starting point (at the end the language doesn't matter it's more about the different char sets / fonts / text sizes) - for example:

is super useful because it captures a lot of different fonts / text sizes
or something "in the wild":

At the end it's more critical to take care that we really can use such images legally.

The tricky part is the detection because we need complete real data .. if we have this it should be much easier for the recognition part we could create some synth data and eval on the already collected real data.

I think if we are able to collect the data up to end of january i could provide pre-labeling via Azure's Document AI.

Currently missing parts are:

• handwritten (for the detection model - recognition is another story)
• chinese (symbols)
• hindi
• bulgarian/ukrainian/russian/serbian (cyrillic)
• special symbols (bullet points, etc.)
• more latin based (spanish, czech, ..)
• ...

CC @odulcy-mindee

Lang list: https://github.com/eymenefealtun/all-words-in-all-languages

[decadance-dance]

@felixdittrich92, thank you for a detailed answer.
I'd help to collect data. It would be great if we can populate this initiative to our community. I think if everyone provides at least a couple of samples, then a good amount of data can be collected.
BTW, Is there any flow or established process for collecting and submitting data?