Parser fixing

CLAUDE.md

@@ -112,7 +112,7 @@ dotnet run --project Jiten.Cli -- --search-lookup "そうする"    # Lookups ta
 **Key files for parser fixes:**
 
 - `Jiten.Parser/MorphologicalAnalyser.cs`: `SpecialCases2`/`SpecialCases3` (hardcoded token combinations), `MisparsesRemove` (tokens to filter), `Combine*` methods (merging logic), `PreprocessText()` (forced splits), `RepairNTokenisation()` (ん-form fixes)
-- `Jiten.Parser/Stages/MorphologicalAnalyser.RewriteRules.cs`: declarative token-rewrite table + engine. **A new single-surface lexical fix (pin a WordId, or split/merge a specific surface into fixed tokens) should be a `RewriteRule` row here, not a hand-rolled `if`-block.** The engine owns offsets, readings-from-templates, cloning, and pin/conjugation recovery. Phases: `Cleanup` (runs before `FilterMisparse`), `Late` (mora-theft position), `Early` (ProcessSpecialCases position). Add a new pipeline **stage** only for a pattern-general mechanism, never a single lexical pattern. Context-heavy logic (window scans, dynamic readings, POS-section edits, order-dependent-on-another-block) legitimately stays as code.
+- `Jiten.Parser/Stages/MorphologicalAnalyser.RewriteRules.cs`: declarative token-rewrite table + engine. **A new single-surface lexical fix (pin a WordId, or split/merge a specific surface into fixed tokens) should be a `RewriteRule` row here, not a hand-rolled `if`-block.** The engine owns offsets, readings-from-templates, cloning, and pin/conjugation recovery. Phases: `Cleanup` (runs before `FilterMisparse`), `Late` (mora-theft position), `Early` (ProcessSpecialCases position). A rule pin is soft by default (lookup-time compound matching may absorb the token into a longer attested span); set `HardPin` when the pin is a final word decision that compound matching must not swallow. Add a new pipeline **stage** only for a pattern-general mechanism, never a single lexical pattern. Context-heavy logic (wider/dynamic window scans, dynamic readings, POS-section edits, multi-token mutation, order-dependent-on-another-block) legitimately stays as code.
 - `Shared/resources/deconjugator.json`: Rule-based deconjugation (~1500 rules). Types: `stdrule`, `rewriterule`, `onlyfinalrule`, `neverfinalrule`, `contextrule`. Fields: `dec_end`/`con_end` (endings), `dec_tag`/`con_tag` (grammar tags). Search for specific endings or `"detail": "past"` etc.
 - `Shared/resources/user_dic.xml`: Custom Sudachi dictionary entries. Format: `surface,leftId,rightId,cost,display,pos1,pos2,pos3,pos4,conjType,conjForm,reading,normalised,dictFormId,splitType,splitA,splitB,unused`. Regenerate after editing: `sudachi ubuild "Y:\CODE\Jiten\Shared\resources\user_dic.xml" -s "S:\Jiten\sudachi.rs\resources\system_full.dic" -o "Y:\CODE\Jiten\Shared\resources\user_dic.dic"`
 

Jiten.Core/Data/PosMapper.cs

@@ -350,6 +350,10 @@ public static bool IsJmDictCompatibleWithSudachi(
 
         // Sudachi 接尾辞 (Suffix) should match JMDict n-suf (NounSuffix) and suf (Suffix).
         // E.g. だらけ is n-suf in JMDict but 接尾辞 in Sudachi; 達 (たち) is suf in JMDict.
+        // Counter (ctr) entries are deliberately NOT compatible here: counter eligibility is
+        // contextual (needs a preceding numeral), which this context-free check cannot see —
+        // a suffix-tagged kanji inside a plain compound (結晶|石) must not prefer a counter
+        // homograph. Numeral contexts select counters in adjacent rescoring instead.
         if (sudachiPos == PartOfSpeech.Suffix &&
             (convertedPosList.Contains(PartOfSpeech.NounSuffix) || convertedPosList.Contains(PartOfSpeech.Suffix)))
             return true;

Jiten.Core/JapaneseTextHelper.cs

@@ -35,6 +35,11 @@ public static bool IsAllKatakana(ReadOnlySpan<char> s)
         return true;
     }
 
+    /// <summary>Katakana letter (ァ–ヺ) or the long-vowel mark ー — the characters a katakana
+    /// word is spelled with, excluding middle dots and iteration marks.</summary>
+    public static bool IsKatakanaWordChar(char c) =>
+        c is (>= 'ァ' and <= 'ヺ') or 'ー';
+
     /// <summary>Fullwidth (Ａ-Ｚ/ａ-ｚ) or halfwidth (A-Z/a-z) Latin letter.</summary>
     public static bool IsLatinLetter(char c) =>
         c is (>= '\uFF21' and <= '\uFF3A') or (>= '\uFF41' and <= '\uFF5A')  // fullwidth Ａ-Ｚ / ａ-ｚ
@@ -86,4 +91,14 @@ public static bool IsKanji(Rune r)
             (>= 0xF900 and <= 0xFAFF) or   // Compatibility Ideographs
             (>= 0x2F800 and <= 0x2FA1F);   // Compatibility Supplement
     }
+
+    /// <summary>
+    /// A numeral character: an ASCII or full-width digit, a kanji digit (一〜九), a kanji place
+    /// marker (十百千万億兆), or a kanji zero (〇零). Covers the counting/quantity vocabulary shared by
+    /// the numeral-context guards; contexts that need a narrower kanji-only set test IsKanji separately.
+    /// </summary>
+    public static bool IsNumeralChar(char c) =>
+        c is (>= '0' and <= '9') or (>= '０' and <= '９')
+          or '一' or '二' or '三' or '四' or '五' or '六' or '七' or '八' or '九'
+          or '十' or '百' or '千' or '万' or '億' or '兆' or '〇' or '零';
 }

Jiten.Parser/Data/WordInfo.cs

@@ -38,6 +38,23 @@ public class WordInfo
     /// ProcessWord escalation chain performs.
     public bool IsKanaExclamation { get; set; }
 
+    /// Set when Sudachi originally tagged this all-katakana token as a noun — a name/loanword
+    /// shape. Deconjugation of its hiragana conversion must not land on kanji-primary words
+    /// (ハガナ → はが+な → 剥ぐ fabricates vocabulary out of a name). Survives the POS rewrites
+    /// that the ProcessWord escalation chain performs.
+    public bool IsKatakanaNounSurface { get; set; }
+
+    /// Set when a rewrite-rule template pinned this token — a deliberate lexical decision the
+    /// misparse gates must not overrule (え|っつった). PreMatchedWordId alone can't carry this:
+    /// parser-level machinery (compound matches, fallbacks) reuses it for ordinary tokens.
+    public bool PinnedByRewriteRule { get; set; }
+
+    /// Set when a gate's pin is a final word decision that compound formation must not absorb
+    /// or override (the ぶん of a fraction frame, the ordinal 目). Soft pins — reading defaults
+    /// like the する-family — stay absorbable: an attested expression spanning them (そうした,
+    /// 臆病風に吹かれる) is the better parse.
+    public bool HardPinned { get; set; }
+
     /// Sudachi lattice segmentation margin: extra cost of the cheapest competing lattice path
     /// crossing one of this token's boundaries (clamped to 99999 = no competitor).
     /// Null when margin output was not requested. Low values = uncertain segmentation.
@@ -70,6 +87,9 @@ public WordInfo(WordInfo other)
         ResolvedWordId = other.ResolvedWordId;
         SudachiBoundaryMargin = other.SudachiBoundaryMargin;
         IsKanaExclamation = other.IsKanaExclamation;
+        IsKatakanaNounSurface = other.IsKatakanaNounSurface;
+        PinnedByRewriteRule = other.PinnedByRewriteRule;
+        HardPinned = other.HardPinned;
     }
 
     public WordInfo(string sudachiLine)

Jiten.Parser/Grammar/TransitionRuleEngine.cs

@@ -156,15 +156,20 @@ or PartOfSpeech.PrenounAdjectival or PartOfSpeech.Counter
                 MatchCondition.NextIsNotQuotative =>
                     w.Next == null || !w.Next.Text.StartsWith("という"),
 
-                // Valid hosts for a sentence-final particle: auxiliaries/particles (だな, はね) plus —
-                // for な only — plain-form verbs/i-adjectives (prohibitive えぐるな, exclamatory
-                // 欲しいな), which stay valid mid-sentence because run-on speech has no punctuation.
-                // Other SFPs keep the merge: ない+か must still form ないか.
+                // Valid hosts for a sentence-final particle: auxiliaries/particles (だな, はね), plain
+                // predicates before な (prohibitive えぐるな, exclamatory 欲しいな), and plain
+                // predicates before よ/ね only when the next token looks vocative
+                // (やべえよ母ちゃん). The vocative gate keeps ordinary run-on text such as
+                // 食べるよ明日 subject to the existing merge/remove repair.
                 MatchCondition.PrevIsSfpValidHost =>
                     w.Prev?.PartOfSpeech is PartOfSpeech.Auxiliary or PartOfSpeech.Particle
                     || (w.Current.Text == "な"
                         && w.Prev is { PartOfSpeech: PartOfSpeech.Verb or PartOfSpeech.IAdjective }
-                        && w.Prev.Text == w.Prev.DictionaryForm),
+                        && w.Prev.Text == w.Prev.DictionaryForm)
+                    || (w.Current.Text is "よ" or "ね"
+                        && w.Prev is { PartOfSpeech: PartOfSpeech.Verb or PartOfSpeech.IAdjective }
+                        && w.Prev.Text == w.Prev.DictionaryForm
+                        && IsVocativeFollower(w.Next)),
 
                 _ => false
             };
@@ -173,6 +178,17 @@ or PartOfSpeech.PrenounAdjectival or PartOfSpeech.Counter
         return true;
     }
 
+    private static bool IsVocativeFollower(WordInfo? next)
+    {
+        if (next?.PartOfSpeech is PartOfSpeech.Name or PartOfSpeech.Pronoun)
+            return true;
+        if (next?.PartOfSpeech is not (PartOfSpeech.Noun or PartOfSpeech.CommonNoun))
+            return false;
+
+        return TransitionRuleSets.HonorificSuffixes.Any(
+            suffix => next.Text.EndsWith(suffix, StringComparison.Ordinal));
+    }
+
     private static void ApplyViolation(
         TransitionRule rule,
         List<(WordInfo word, int pos, int len)> words,

Jiten.Parser/Grammar/TransitionRuleSets.cs

@@ -160,10 +160,12 @@ internal static class TransitionRuleSets
             [ScoringCondition.NextIsExplanatoryN],
             25),
 
+        // High enough that a numeral+counter pair outbids a top-priority noun homograph together
+        // with its ruby prior (第二話: the counter わ must beat はなし).
         new("numeral-counter-cohesion",
             [ScoringCondition.CandidateIsCounter],
             [ScoringCondition.PrevIsNumeral],
-            40),
+            60),
 
         new("orphan-counter-penalty",
             [ScoringCondition.CandidateIsCounter, ScoringCondition.CandidateIsNotNounLike],

Jiten.Parser/Helpers/MorphologicalAnalyser.Diagnostics.cs

@@ -30,12 +30,13 @@ private static List<SudachiToken> ParseSudachiOutputToDiagnosticTokens(string ra
         return tokens;
     }
 
-    private static List<WordInfo> TrackStage(TokenStage stage, List<WordInfo> input, ParserDiagnostics? diagnostics)
+    private static List<WordInfo> TrackStage(TokenStage stage, List<WordInfo> input, ParserDiagnostics? diagnostics,
+                                             TokenFeatureScan? scan = null)
     {
         var inputSnapshot = diagnostics != null ? input.Select(TokenSnapshot.From).ToList() : null;
         var sw = diagnostics != null ? Stopwatch.StartNew() : null;
 
-        var result = stage.Apply(input);
+        var result = stage.Apply(input, scan);
 
         if (diagnostics == null)
             return result;

Jiten.Parser/Helpers/MorphologicalAnalyser.RuleData.cs

@@ -111,6 +111,7 @@ public partial class MorphologicalAnalyser
         ("すぐ", "に", PartOfSpeech.Adverb),
         ("たしか", "に", PartOfSpeech.Adverb),
         ("確か", "に", PartOfSpeech.Adverb), // 確かに 1205770 — kanji variant of the たしか row above
+        ("直", "に", PartOfSpeech.Adverb), // 直に 1430690 (じかに, "directly/in person") — the bare 直 otherwise reads ちょく
         ("ゆえ", "に", PartOfSpeech.Conjunction), // 故に 1267130 ("therefore") — literary connective Sudachi splits
         ("故", "に", PartOfSpeech.Conjunction), // kanji variant of the ゆえ row above
         ("とう", "に", PartOfSpeech.Adverb), // 疾うに 1633370 ("long ago") — the bare とう otherwise strands on 塔

Jiten.Parser/Misparse/MisparseGates.cs

@@ -37,7 +37,9 @@ internal static class MisparseGates
     /// digits, or a long mixed-kana surface — are excluded before the caller assembles flags,
     /// neighbour frames and blob context. The character test accepts a superset of the kana the
     /// gates accept (full/half-width kana blocks plus stretch marks), so it can only skip tokens
-    /// no gate would touch.
+    /// no gate would touch. The ≤4 cap excludes 6-character reduplications such as ギュウギュウ;
+    /// those overlap attested on-mim adverbs such as ごちゃごちゃとして and require separate
+    /// mimetic preference handling.
     public static bool MayBeKanaFragment(string s)
     {
         if (s.Length == 0) return false;
@@ -95,7 +97,14 @@ private static bool IsSfxMimeticFragment(in MisparseGateContext ctx)
             ? ctx.SelectedWord.OriginalText
             : ctx.Token.Text;
 
-        if (surface.Length == 0 || surface.Length > 4 || !WanaKana.IsKana(surface)) return false;
+        // Pure reduplication (コクコク) is mimetic orthography — it extends the local length
+        // window and counts as a positive marker below. Six-character reduplications
+        // (ギュウギュウ) only arrive when a shorter resolved token carries the longer
+        // OriginalText — MayBeKanaFragment's ≤4 cap excludes them as whole tokens upstream.
+        bool reduplicated = surface.Length is 4 or 6
+                            && surface[..(surface.Length / 2)] == surface[(surface.Length / 2)..];
+        if (surface.Length == 0 || (surface.Length > 4 && !reduplicated) || surface.Length > 6
+            || !WanaKana.IsKana(surface)) return false;
 
         // A neighbour already discarded as a misparse shred is part of the same burst; so is an
         // unresolved kana scrap. Both make the current token frame-transparent on that side.
@@ -117,17 +126,24 @@ private static bool IsSfxMimeticFragment(in MisparseGateContext ctx)
         // unsegmentable blob. A case particle or content word on either side means real syntax,
         // which the gate must not touch. When a sokuon is clipped straight onto the token (ず|がんっ),
         // a short kana neighbour is part of the same burst even if it happened to resolve.
+        // A reduplicated surface heading the mimetic-adverb frame 〜と+verb (コクコクと振る) is
+        // phonetic wherever it sits — the construction itself is the isolation.
+        bool reduplicatedToFrame = reduplicated
+                                   && ctx.Next is { Text: "と", PartOfSpeech: PartOfSpeech.Particle };
+
         bool frameBefore = ctx.Prev == null || ctx.SymbolsBefore.Length > 0 || ctx.IsSentenceInitial
                            || ctx.Prev.PartOfSpeech == PartOfSpeech.Interjection
                            || (ctx.Prev.PartOfSpeech == PartOfSpeech.Particle
                                && ctx.Prev.Text is "な" or "よ" or "ね" or "ぞ" or "ぜ" or "わ" or "さ")
                            || (exclamatoryClip && ctx.Prev.Text.Length <= 2 && WanaKana.IsKana(ctx.Prev.Text))
-                           || prevShard;
+                           || prevShard
+                           || reduplicatedToFrame;
         if (!frameBefore) return false;
 
         bool nextIsQuotativeTo = ctx.Next is { Text: "と", PartOfSpeech: PartOfSpeech.Particle }
                                  && ctx.SymbolsAfter.Length > 0;
-        bool frameAfter = ctx.Next == null || ctx.SymbolsAfter.Length > 0 || nextShard;
+        bool frameAfter = ctx.Next == null || ctx.SymbolsAfter.Length > 0 || nextShard
+                          || reduplicatedToFrame;
         if (!frameAfter) return false;
 
         // Require a positive mimetic signal; isolation alone describes any one-word answer (「梨」).
@@ -138,6 +154,7 @@ private static bool IsSfxMimeticFragment(in MisparseGateContext ctx)
                       || ctx.SymbolsAfter.IndexOfAny(GapMimeticChars) >= 0
                       || nextIsQuotativeTo
                       || prevShard || nextShard
+                      || reduplicated
                       // An in-surface sokuon is ordinary orthography when the entry spells the
                       // surface — literally (おっさん) or through the expressive-deformation check
                       // in GetWordFlags (バカッ/ほんっと) — but phonetic evidence when it does not
@@ -174,9 +191,15 @@ private static bool IsSfxMimeticFragment(in MisparseGateContext ctx)
         // うぜえ、と, ごくり、と, ハルさんっ): attested as written, usually-kana, or reached
         // through conjugation. A non-noun invented by text mutation (うぅ→うん) stays in scope
         // as noise.
+        // An entry with an adverbial sense used in the adverbial と frame is that sense, whatever
+        // POS happens to be listed first (ぽつぽつ [n, adv, adv-to] + と + 灯す) — first-POS alone
+        // must not pull an attested mimetic back into the plain-noun scope.
+        bool adverbialInToFrame = ctx.Next is { Text: "と", PartOfSpeech: PartOfSpeech.Particle }
+            && ctx.SelectedWord.PartsOfSpeech.Any(p => p is PartOfSpeech.Adverb or PartOfSpeech.AdverbTo);
         if ((ctx.SelectedWord.PartsOfSpeech.Count == 0
              || ctx.SelectedWord.PartsOfSpeech[0] is not (PartOfSpeech.Noun
-                 or PartOfSpeech.CommonNoun or PartOfSpeech.Numeral or PartOfSpeech.NaAdjective))
+                 or PartOfSpeech.CommonNoun or PartOfSpeech.Numeral or PartOfSpeech.NaAdjective)
+             || adverbialInToFrame)
             && (ctx.SurfaceAttestsListedForm || ctx.IsUsuallyKana
                 || (ctx.SelectedWord.Conjugations.Count > 0
                     && ctx.SelectedWord.Conjugations[0] is not ("(stem)" or "(infinitive)"
@@ -367,6 +390,23 @@ public static (bool isUsuallyKana, bool hasKanjiSpelling, bool readingIsIchi, bo
 
         bool surfaceAttestsLiterally = surfaceAttestsForm;
 
+        // Mimetic adverbs and interjections are written in either kana script interchangeably and
+        // JMdict lists most of them hiragana-only — a katakana spelling (スタスタと) is ordinary
+        // orthography for the same word, not a deformation. Restricted to the adverbial/
+        // interjection class so a katakana SFX cannot claim an unrelated hiragana word's identity
+        // through the script fold.
+        if (!surfaceAttestsForm && surface is { Length: > 0 }
+            && word.PartsOfSpeech.Any(p => p is "adv" or "adv-to" or "int" or "on-mim")
+            && JapaneseTextHelper.IsAllKatakana(surface))
+        {
+            var folded = WanaKana.ToHiragana(surface);
+            if (folded != surface && word.Forms.Any(f => f.Text == folded))
+            {
+                surfaceAttestsForm = true;
+                surfaceAttestsLiterally = true;
+            }
+        }
+
         // Expressive spelling deforms a word without changing it: emphatic gemination writes a
         // sokuon in (ほんっと, バカッ, マジッす), a chōonpu stretches a mora (おーっと), and a
         // trailing stretch elongates the final one (そっかー, だってぇ, なんだとぉ). Such a

Jiten.Parser/MorphologicalAnalyser.cs

@@ -16,6 +16,7 @@ public partial class MorphologicalAnalyser
     public Func<string, int?>? GetNonNameCompoundWordId { get; set; }
     public Func<string, int?>? GetNonNameCompoundFrequencyRank { get; set; }
     public Func<string, bool>? HasVerbOrAdjectiveLookup { get; set; }
+    public Func<string, bool>? HasCounterSenseLookup { get; set; }
 
     // Captured per Parse call for RetokeniseOovBlobs; an instance must not serve two parses with
     // different Sudachi configs concurrently (all production paths construct one instance per parse).