fix(md): keep the last cell of table rows without a trailing pipe#3817
fix(md): keep the last cell of table rows without a trailing pipe#3817chuenchen309 wants to merge 1 commit into
Conversation
The leading and trailing pipes of a GFM table row are both optional, and
the backend's own row detector only requires a leading one:
is_table_row = "|" in snippet_text and (
self.in_table or original_text.lstrip().startswith("|")
)
But the parser split each row with [1:-1], which assumes both pipes are
there. On a row written without the trailing pipe, that slice drops the
last cell:
| Character | Name in German -> | Character |
|---|--- |----------------|
| Scrooge McDuck | Dagobert Duck | Scrooge McDuck |
"Name in German" and "Dagobert Duck" are gone, silently, with no warning.
The input is legal GFM: marko, docling's own markdown dependency, renders
that same table with two <th> columns.
The sibling asciidoc backend already gets this right -- _split_row there
does `line.split("|")[1:]` and comments that it is removing the leading
empty string only.
Split off an empty field only when it comes from a pipe at the very edge
of the row, so both spellings parse the same. Rows that do carry the
trailing pipe are unaffected, and an intentionally empty cell in the
middle of a row still survives.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: chuenchen309 <48723787+chuenchen309@users.noreply.github.com>
|
✅ DCO Check Passed Thanks @chuenchen309, all your commits are properly signed off. 🎉 |
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
Thanks @chuenchen309 for suggesting this fix.
|
|
Thanks @ceberam! I checked both against the current backend: Tables without a leading pipe — the split side already handles this: So this PR alone doesn't cover it. Enabling it means relaxing that detector, which is riskier — any prose line containing GFM cell-count mismatch (4.10) — currently not normalized; the grid comes out ragged rather than padded/truncated: Happy to extend this PR to cover both. They're fairly independent of the trailing-pipe drop, though, and the detector change carries the false-positive risk above — so would you prefer one combined PR, or keep this one scoped to the trailing-pipe fix and track full GFM 4.10 coverage as a follow-up? Either works for me. |
Thanks @chuenchen309 for the detailed analysis. I think it would be better if you could tackle all those edge cases in a single PR, to ensure consistency in the code. It would be about dealing with markdown tables without leading/trailing pipes. |
The leading and trailing pipes of a GFM table row are both optional, and the backend's own row detector only requires a leading one (
md_backend.py:488):But the parser split each row with
[1:-1](:215,:220), which assumes both pipes are present. On a row written without the trailing pipe, that slice drops the last cell:main| Character | Name in German|---|---| Scrooge McDuck | Dagobert Duck| Character ||----------------|| Scrooge McDuck |"Name in German" and "Dagobert Duck" are gone, silently, with no warning. Directly against the backend: the trailing-pipe form gives
num_cols=2and four cells; the same table without it givesnum_cols=1and two.The input is legal GFM — that isn't my reading of the spec:
marko, docling's own markdown dependency, renders that same table with two<th>columns.The sibling asciidoc backend already gets this right —
_split_row(asciidoc_backend.py:373) doesline.split("|")[1:]and comments that it removes the leading empty string only.Fix
Split off an empty field only when it comes from a pipe at the very edge of the row, so both spellings parse identically. Rows that do carry the trailing pipe are unaffected, and an intentionally empty cell in the middle of a row still survives.
Reachable from the public
DocumentConverter().convert()(document_converter.py:152) and also fromvlm_pipeline.py:302, so a VLM emitting a table without trailing pipes loses columns too.Test Plan
Added
test_convert_table_without_trailing_pipesnext to the existingtest_convert_table_has_no_duplicate_cells, asserting both spellings produce the same two-column table. It fails onmainand passes with this change; the existing table test passes on both.tests/test_backend_markdown.py— 9 passed, 1 skippedtests/test_backend_asciidoc.py tests/test_backend_csv.py tests/test_backend_html.py— 44 passed-k "backend or markdown or md"sweep — 334 passed, 2 failed; both failures aretest_backend_webp.py::test_e2e_webp_conversionsraisingImportErrorfor a missing optional OCR engine, and they reproduce identically on a pristine checkout with no changes applied.ruff check/ruff format --checkclean.Checklist
Disclosure: written with Claude Code. I verified the reproduction, the marko comparison, the sibling backend, and the test red/green myself.