Skip to content

Complete incomplete Viterbi lattice tests#2081

Open
eunseo9311 wants to merge 1 commit into
huggingface:mainfrom
eunseo9311:test/complete-lattice-viterbi-tests
Open

Complete incomplete Viterbi lattice tests#2081
eunseo9311 wants to merge 1 commit into
huggingface:mainfrom
eunseo9311:test/complete-lattice-viterbi-tests

Conversation

@eunseo9311
Copy link
Copy Markdown
Contributor

The test_viterbi test in lattice.rs was marked XXX: incomplete
it only checked .len() without verifying node IDs or token content.

This PR completes it and adds three new test cases:

  • test_viterbi: verify node IDs and tokens returned by viterbi(),
    and confirm score-based path selection picks the higher-score path
  • test_viterbi_multibyte: lattice over a multi-byte UTF-8 string
    ("テスト"), including a multi-character spanning node
  • test_viterbi_empty: empty string edge case
  • test_sample_smoke: structural validity check for sample()
    sampled tokens must always concatenate back to the original input

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant