Parser fixing - #437
Conversation
|
ざまあみやがれ should parse as https://jiten.moe/vocabulary/2868161/1 , not split into components |
|
Good catch, I'll address this. |
| yield return ["１０度って言われた", "度", 1445160, (byte)0]; | ||
| // A suffix-tagged kanji inside a plain compound must keep its noun homograph: counter | ||
| // senses need numeric material before them (石=いし stone, not せき the jewel counter) | ||
| yield return ["結晶石", "石", 1382440, (byte)0]; |
There was a problem hiding this comment.
https://nadeshiko.co/en/search/%E7%B5%90%E6%99%B6%E7%9F%B3
1 example I could find and it's pronounced せき here. せき is used for word coinage, but jmdict doesn't note this like kokugos do, so not sure how to deal with this. I would remove manual parsing of this for now.
There was a problem hiding this comment.
Hmm, I see what you're saying. I'll change to a better test (石 specifically wasn't the aim of the fix, it was just related), but as you mention the core issue here is that jmdict itself doesn't see せき as anything but a counter. So there is no clean solution other than that getting added to jmdict, unless we carve it out so it will stay せき, but I don't think thats a good idea because it would say it is a counter when that isn't how it is operating here anyways.
There was a problem hiding this comment.
Yeah. Maybe it should just lead to kanji page in such cases.
Also, AFAIK it's a policy of jmdict to not add coinage senses to entries since "it's a job of a kanwa dictionary".
In any case, it's issue for @Sirush to solve.
There was a problem hiding this comment.
There's 2 solutions: leave it unparsed or link it to the right reading (even if the entry is wrong for the sense), I think it makes more sense to still link it to せき in this case.
There's no mechanic that can possibly link to a kanji page currently, and this seems to be a bit of an edge case
3 participants
Parser fixes:
New stage for repairing bad splits like 流+しっ+ぱなし.
New stage for remerging shredded katakana words. Example: コーポレイテッド -> コーポ + レイ + テッド. An unknown katakana name like this would previously get split into incorrect words.
New stage fixing compounds being broken up by certain adjacent words, like 滑走路上 was being parsed to 滑走+路上, now 滑走路+上. The 上 was stealing 路 from the compound.
New stage repairing a kanji verb being split from its tail. Example: 眼を覚ませって言われた was splitting 覚+ませ.
New stage splitting 返る from being part of a verb compound if no entry exists for that, to stop misparses.
Improvements for counters. Examples: 第二話=わ, 十分後=じっぷん.
SFX gate improvements
りゃ conditional. Example: すりゃ
ませぬ fix
Lots of disambiguation
って/っつー fixing
あんだ/すんだ (contracted ある / する + んだ) fix. Example: 何すんだよ！ was parsing as 済んだ instead of す(する)+んだ
... and a lot more
The rewrite engine rule pins can now be marked HardPin, for pins that must not be absorbed into other things (needed for す+んだ).
Additionally, four of the new stages are candidate gated, the existing token feature scan records candidate positions and the stages only visit those, to save on performance. Mutating stages return a fresh list, which causes a rescan.
All form / parser tests pass.