text analyzer stemmer filter language support

[faileon]

Hello,
are there any plans to include more languages for the stemming filter step in text analyzer? It would greatly improve keyword/hybrid search in my local language (czech). Will it eventually be possible to add custom stemmers?

The roadmap has the following section for CY25:
Analyzer Enhancement
Enhance Analyzer with expanded tokenizer support and improved observability

Perhaps it will be included with this feature?

[yhmo]

Wait @aoiasd to comment.

[yhmo]

I just discussed with @aoiasd. So far, we don't have the stemming filter for Czech. The milvus analyzer is mainly powered by Tantivy, and the snowball project provides stemmers for tantivy. Czech is not in the list of the stemmers of snowball: https://github.com/snowballstem/snowball/blob/master/libstemmer/modules.txt

[faileon]

While not on the GitHub list, the snowball project mentions Czech stemmer on their site https://snowballstem.org/algorithms/czech/stemmer.html

There is also one directly for tantivy https://github.com/testuj-to/tantivy-czech-stemmer

But perhaps the most important would be this PR in the snowball repo snowballstem/snowball#151

[yhmo]

Looks like the https://github.com/testuj-to/tantivy-czech-stemmer is a customized stemmer based on snowball, not an official snowball lib.

https://snowballstem.org/algorithms/czech/stemmer.html This page mentioned "In March 2012 Jim O’Regan sent us an implementation of Ljiljana Dolamic's Czech stemmer." There should be a file "czech.sbl" in this folder https://github.com/snowballstem/snowball/tree/master/algorithms, but no such file.

Seems the pr snowballstem/snowball#151 is submitted by another developer since the implementation is different. This pr has been pending since 2021, and marked as milestone 3.1.0 recently. It is not available in the current version of tantivy. Once it is released, we can plan to upgrade tantivy to new version to support Czech.

[ojwb]

To correct a minor detail here, snowballstem/snowball#151 is actually derived from the version on the website (though it has evolved to the point where that may not be obvious).

If you compare the outputs from the Snowball Czech stemmer on the website with Dolamic's original Java stemmer (linked in comments at the end), there are significant differences. I started pulling at that thread and also found some clear bugs in the Java versions (e.g. incorrect lengths in checks mean some suffixes can never match, such as buffer.substring( len- 2 ,len).equals("\u0161ti")), then while resolving that I noticed more threads to pull at, and so on. A lot of unravelling later...

The version in the PR works better than either the website version (which is functionally the same as the tantivy version) or the Java original, but there are a few points still to resolve and I'd certainly appreciate input from Czech speakers, especially those working on searching Czech text. I've just summarised one of the two open points over in the Snowball PR to try to make that easier:

snowballstem/snowball#151 (comment)

[faileon]

hi @yhmo, the czech stemmer got merged in snowball recently snowballstem/snowball#151
is there a possibility to get it into milvus as is or do you guys need to wait for the 3.1.0 release? thanks

[faileon]

hello good folks from Milvus, snowballstemmer is now in v3.1.1, which should include the new czech stemmer. From my understanding the follwing changes must be made in the follwing order:

1. update milvus/rust-stemmers with new snowball version, regenerate stemmers, add the new czech stemmer, PR
2. update zilliztech/tantivy with czech support and dependency bump of milvus/rust-stemmers
3. update milvus-io/milvus with czech support and dependency bump of zilliztech/tantivy

i will send PRs if its OK with you

[xiaofan-luan]

hello good folks from Milvus, snowballstemmer is now in v3.1.1, which should include the new czech stemmer. From my understanding the follwing changes must be made in the follwing order:

1. update milvus/rust-stemmers with new snowball version, regenerate stemmers, add the new czech stemmer, PR
2. update zilliztech/tantivy with czech support and dependency bump of milvus/rust-stemmers
3. update milvus-io/milvus with czech support and dependency bump of zilliztech/tantivy

i will send PRs if its OK with you

Hi Faileon, happy to take a pr.

[faileon]

hello good folks from Milvus, snowballstemmer is now in v3.1.1, which should include the new czech stemmer. From my understanding the follwing changes must be made in the follwing order:

1. update milvus/rust-stemmers with new snowball version, regenerate stemmers, add the new czech stemmer, PR
2. update zilliztech/tantivy with czech support and dependency bump of milvus/rust-stemmers
3. update milvus-io/milvus with czech support and dependency bump of zilliztech/tantivy

i will send PRs if its OK with you

Hi Faileon, happy to take a pr.

perfect, thanks. i started with the rust-stemmers repo, it should be the largest - milvus-io/rust-stemmers#2
i had to regenerate other stemmers while updating the snowball to new version, as well as update the snowball test data in order to fix tests, hence the PR size, hope its OK.

[faileon]

hi, @xiaofan-luan , is the PR for rust-stemmers OK? can someone take a look at it please so I can proceed with the next steps? thank you

[yhmo]

hi, @xiaofan-luan , is the PR for rust-stemmers OK? can someone take a look at it please so I can proceed with the next steps? thank you

Wait @aoiasd to confirm.