feat(pdf): recover compound list-item markers missed by the marker processor

[DanielNg0729]

Summary

Hi maintainers, i saw this as a #TODO in the codebase so i try to implement it!
The PDF/image pipeline labels list regions as LIST_ITEM but records no ordered-vs-unordered distinction that has to be inferred from the item text. At item-creation time the reading-order stage already runs docling-ibm-models' ListItemMarkerProcessor, which strips simple leading markers (1., a), [3], • …) into ListItem.marker and sets ListItem.enumerated. This covers the common cases behind the recurring # TODO: Infer if this is a numbered or a bullet list item.

What it still misses are compound / hierarchical markers — 9a., 3.a., 1.2.1. These stay fused inside text with an empty marker. When such an item sits inside an otherwise-enumerated list, the Markdown serializer sees the empty marker and prepends a position-based number of its own, producing a doubled, wrong marker. On tests/data/pdf/.../2203.01017v2 (the Table-Transformer paper) the algorithm list currently renders:

3. Use a carefully selected IOU threshold ...
4. 3.a. If all IOU scores ... discard ...        <- doubled + wrong number
4. Find the best-fitting content alignment ...
...
7. 9a. Compute the top and bottom boundary ...   <- doubled + wrong number
8. 9b. Intersect the orphan's bounding box ...

This PR adds an opt-in ListNormalizationModel that runs right after the reading-order model and, per list group, recovers those compound markers into marker (stripping them from text and setting enumerated). The same list then renders:

3. Use a carefully selected IOU threshold ...
- 3.a. If all IOU scores ... discard ...
4. Find the best-fitting content alignment ...
...
- 9a. Compute the top and bottom boundary ...
- 9b. Intersect the orphan's bounding box ...

Recovery is deliberately conservative:

• It only touches items the marker processor left with an empty marker; items already handled are untouched.
• It runs only in a numbered context — the group already has a recognized numbered marker, or at least two items look compound-ordered, so a lone dotted-decimal value in prose (1.1 million units …) inside a bullet list is never mistaken for a marker.
• It never adds, removes, or reorders items.

Behavior / compatibility

• Gated behind PdfPipelineOptions.list_normalization_options.enabled, default False: no change to existing output, so no reference data is regenerated.
• Wired into both StandardPdfPipeline and LegacyStandardPdfPipeline, mirroring the existing heading-hierarchy stage.
• ListNormalizationModel.normalize operates on a bare DoclingDocument, so the core is reusable outside the pipeline.

How to enable

opts = PdfPipelineOptions()
opts.list_normalization_options.enabled = True

Checklist:

• Documentation has been updated, if necessary.
• Examples have been added, if necessary.
• Tests have been added, if necessary.

[github-actions]

✅ DCO Check Passed

Thanks @DanielNg0729, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[codecov]

Codecov Report

❌ Patch coverage is 90.90909% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ges/list_normalization/list_normalization_model.py	89.47%	6 Missing ⚠️

📢 Thoughts on this report? Let us know!

[cau-git]

@DanielNg0729 thanks for taking the effort to improve this aspect of list marker identification. Since the docling-ibm-models package already has partial logic, would anything speak against improving the code there instead of adding another layer into docling?

See https://github.com/docling-project/docling-ibm-models/

[DanielNg0729]

Thank you very much Dr @cau-git I didnt know there was already work for this in docling-ibm-models. I'm happy to move this into the processor instead of adding a stage here.

Two things I'd like to check before I open the PR there:

• Should the new patterns be unconditional, or gated (similar to the existing infer_enumerated)? Since this changes default conversion output, I'd plan to regenerate the affected reference docs
• the process_list_item works per item without group context, so it can't use the "is this item inside an otherwise-numbered list?" guard I used to avoid treating a stray 1.1 million … as a marker. I'll keep the added patterns conservative; let me know if you'd prefer a stricter shape.

If that sounds good, I'll open a PR against docling-ibm-models adding the compound-marker patterns, and close this one. Thank you!

[cau-git]

@DanielNg0729 I see that you opened a new PR on docling-ibm-models for this, thanks. The final pass to apply now is:

• Make a fresh docling PR which pins in the unmerged docling-ibm-models branch as a git-dependency (edit tool.uv.sources section in pyproject.toml), and rerun the test suite.

DOCLING_GEN_TEST_DATA=1 CI=1 uv run pytest tests/ - we need to see which files in the test GT of docling will change through this.

[DanielNg0729]

Hi Dr @cau-git, thank you for taking the time to review this. I've opened #3849 to pin the docling-ibm-models branch, and left a couple of comments there walking through the approach and the ground-truth impact. Whenever you have a moment, I'd be grateful for your thoughts. Thanks again!

[dolfim-ibm]

@DanielNg0729 does it mean we can close this PR in favor of docling-project/docling-ibm-models#169 + #3849?

[DanielNg0729]

Hi Dr @dolfim-ibm, yes this is superseded by that 2 PR, thank you very much!