✨ Render full RST need directives from @rst blocks

[chrisjsewell]

Summary

Enable users to embed complete RST need directives inside source code comments delimited by @rst ... @endrst markers.
These blocks are parsed and rendered as real Sphinx-Needs nodes during the Sphinx build, complementing the existing one-line marker support.

Motivation

One-line markers (@req{ID}) are convenient for simple needs but cannot express options, content bodies, or arbitrary directive fields. By allowing full directive syntax inside @rst blocks, users can write rich need items directly in source comments — including :links:, :status:, content text, and any other field that NeedDirective accepts — while still getting automatic source-tracing URLs.

Changes

analyse/utils.py

• ParsedDirective TypedDict — structured return type capturing a directive's name, argument, options, content, line offsets, and whether extra content exists outside the directive body.
• parse_single_directive() — regex-based parser that extracts the first directive from an RST text block. Returns None when the first non-blank line is not a directive.

sphinx_extension/directives/src_trace.py

• generate_str_link_name() widened to accept Metadata (the base class) instead of OneLineNeed, so it works for both one-line needs and marked RST blocks.
• render_marked_rst_needs() — new method on SourceTracingDirective that iterates src_analyse.marked_rst, parses each block, injects local/remote URL options, constructs a NeedDirective instance, and calls .run() to produce docutils nodes.
• Called from run() after render_needs().

tests/test_analyse_utils.py

• Parametrised tests for parse_single_directive covering: minimal directives, options, content bodies, multi-line content, leading/trailing blanks, extra content detection, no-argument directives, namespaced directive names, and None-return cases.

tests/test_src_trace.py + tests/doc_test/

• rst_basic — integration test with a Sphinx project that uses only get_rst = true. Source file contains a single /* @rst … @endrst */ block with an .. impl:: directive (RST_IMPL_1). Verifies the need node appears in the doctree snapshot.
• rst_mixed — integration test combining get_oneline_needs and get_rst in the same build. Uses [[ ]] one-line markers to avoid clashing with the @rst prefix. Source file contains both a one-line need (OL_IMPL_1) and an RST block need (RST_IMPL_2). Verifies both needs appear in the doctree snapshot.

Design decisions

Why instantiate NeedDirective directly?

NeedDirective uses DummyOptionSpec — a dummy spec that accepts all options and keeps them as strings
(sphinx-needs source). DummyOptionSpec was introduced in sphinx-needs v6 (commit d09332d); earlier versions use an explicit option_spec dict, so passing arbitrary raw-string options would fail there.
We should consider raising the minimum dependency to sphinx-needs>=6 (currently >=5,<9 in pyproject.toml).

NeedDirective.run() itself does its own key-by-key validation (via a match key: block). This means:

• No option validation/conversion is needed before instantiation.
• Passing raw dict[str, str | None] is exactly what the directive expects.

Using NeedDirective directly (rather than add_need()) gives full directive feature support: content body, arbitrary options, and internal NeedDirective logic for title_from_content, delete, etc.

Why a custom regex parser instead of docutils parsing?

The parse_single_directive function is a purposefully simple regex parser scoped to the single-directive-per-block use case. Full docutils RST parsing would be heavier and harder to control for this constrained input.

Comparison with MyST-Parser's directive handling

MyST-Parser's
run_directive follows a more general pipeline:

1. Directive class lookup via docutils.parsers.rst.directives.directive() — resolves any registered directive and warns on unknowns.
2. parse_directive_text() — a dedicated parser that validates options against the directive's option_spec (type converters, unknown-key detection), validates arguments against required_arguments / optional_arguments / final_argument_whitespace, and supports both YAML-delimited and RST-style (:key: value) option blocks.
3. Mocked state / state_machine (MockState, MockStateMachine) — because MyST renders from markdown-it tokens, not from within a real docutils state machine, it must mock these so directives can call nested_parse().
4. Error wrapping — DirectiveError and MockingError are caught and converted to clean error nodes.

What we don't need from this approach:

• Option validation — NeedDirective uses DummyOptionSpec and validates internally, so pre-validation would be redundant.
• Directive class lookup — we always target NeedDirective.
• Mocked state — we run inside a real SphinxDirective.run(), so full docutils state / state_machine are already available.

What we could adopt (potential TODOs):

• content_offset fallback — when content_line_offset is None the code falls back to self.content_offset (the enclosing .. src-trace:: directive's offset). This is semantically incorrect (though harmless when there's no content). Consider using 0 or adding a clarifying comment.
• Bump minimum sphinx-needs to v6 — DummyOptionSpec (which lets us pass arbitrary string options) was added in v6 (d09332d). The current constraint is sphinx-needs>=5,<9; without bumping it, render_marked_rst_needs will break on sphinx-needs <6 where the directive uses a fixed option_spec.
• Per-line source tracking on StringList — the current code creates StringList(content_lines, source=src_file) which sets one source for all lines. For richer error messages pointing to exact lines within the RST block, per-line offset info could be added (docutils StringList supports this via the items parameter).
• Marker clashes between one-line parser and @rst blocks — when both get_oneline_needs and get_rst are enabled, comments containing @rst ... @endrst may also be matched by the one-line parser if its start sequence overlaps with @rst (e.g. the default @ prefix). Currently the analysis pipeline processes every comment node through both extractors independently; there is no mutual exclusion. This can produce spurious one-line needs or validation errors from the @rst block's content being misinterpreted as one-line fields. Consider adding a skip/guard so that comments already claimed by extract_marked_rst are not also fed to extract_oneline_needs, or document that users must choose non-overlapping marker sequences.

Key finding: @rst blocks require block comments

RST blocks must use C-style block comments (/* @rst ... @endrst */) rather than // line comments in C/C++. Tree-sitter parses each // line as a separate comment node, and extract_rst() needs both @rst and @endrst within a single comment node's text. This is an inherent constraint of the current tree-sitter based extraction and should be documented for users.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 86.30137% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.86%. Comparing base (cdf9e58) to head (4e4266f).

Files with missing lines	Patch %	Lines
...codelinks/sphinx_extension/directives/src_trace.py	61.53%	10 Missing and 5 partials ⚠️
src/sphinx_codelinks/analyse/utils.py	93.82%	3 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #66      +/-   ##
==========================================
- Coverage   90.06%   89.86%   -0.21%     
==========================================
  Files          29       31       +2     
  Lines        2628     2772     +144     
  Branches      306      327      +21     
==========================================
+ Hits         2367     2491     +124     
- Misses        165      178      +13     
- Partials       96      103       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[chrisjsewell]

Investigation: Source mapping and directive parsing in docutils — implications for render_marked_rst_needs

Background

I investigated how docutils' Include directive (in docutils/parsers/rst/directives/misc.py) and Sphinx's Include override (in sphinx/directives/other.py) handle parsing content from external sources, and what that means for the approach taken in this PR — specifically render_marked_rst_needs() and parse_single_directive().

The goal was to understand whether we could delegate RST parsing to docutils instead of using a custom regex parser, and whether source mapping (pointing error messages and node source attributes back to the correct line in the original .c/.cpp file) can be made to work correctly.

How docutils source mapping actually works

Docutils has two parallel line-number systems, and understanding both is critical:

System 1: Absolute line numbers

StateMachine.abs_line_number() returns self.line_offset + self.input_offset + 1 — a flat counter within the state machine's coordinate space. This is what gets stored as lineno in directives and passed to self.reporter.warning(..., line=lineno).

System 2: StringList.items per-line metadata

StringList (a subclass of ViewList) stores a parallel items list of (source_filename, offset) tuples — one per line. When you construct StringList(['line0', 'line1'], source='myfile.c'), the items default to [('myfile.c', 0), ('myfile.c', 1)].

The bridge: get_source_and_line(lineno)

This method converts an absolute lineno back into a StringList index and looks up the real (source, line) via self.input_lines.info(offset). It only works correctly when called on the same state machine whose input_lines contain the relevant content.

How insert_input works (and why Include uses it)

StateMachine.insert_input() splices lines directly into the live input_lines of the currently running state machine, right after the current position:

def insert_input(self, input_lines, source) -> None:
    self.input_lines.insert(self.line_offset + 1, '',
                            source='internal padding after '+source,
                            offset=len(input_lines))
    self.input_lines.insert(self.line_offset + 1, '',
                            source='internal padding before '+source,
                            offset=-1)
    self.input_lines.insert(self.line_offset + 2,
                            StringList(input_lines, source))

It inserts two blank padding lines as sentinels, then splices the new StringList between them. The state machine's main loop simply walks through input_lines one index at a time, so it naturally processes the inserted lines next, then resumes the original document.

The original document's source mapping is not corrupted because insert_input only adds new entries to the data and items arrays — existing entries shift position in the array but their (source, offset) values are unchanged.

Additionally, docutils' Include appends ['', '.. end of inclusion from "source"'] after the content. The RST parser's Body.comment() method recognises this special comment and pops the include_log entry — this is how circular inclusion detection unwinds.

Bugs in docutils' line-number handling

Bug 1: The reporter is bound to the root state machine only

In RSTState.runtime_init():

if not hasattr(self.reporter, 'get_source_and_line'):
    self.reporter.get_source_and_line = self.state_machine.get_source_and_line

The not hasattr guard means this is set exactly once on the root RSTStateMachine. Every NestedStateMachine skips it. So when any code does self.reporter.warning('...', line=some_lineno), the reporter resolves that line number using the root SM's get_source_and_line(), which looks up the root SM's input_lines. If the content being parsed is in a separate StringList (as happens with nested_parse), the lookup hits the wrong line or a completely unrelated line from the main document.

The docutils source itself has a TODO confirming this discrepancy (states.py line ~2908):

# TODO: why is abs_line_number() == srcline+1
# if the error is in a table (try with test_tables.py)?

Bug 2: insert_input masks this, but nested_parse doesn't

With insert_input, lines are spliced into the root SM's input_lines, so the root SM's get_source_and_line() can find them — which is why Include uses insert_input rather than nested_parse.

With nested_parse, content lives in a separate StringList given to a new nested SM. The nested SM's own get_source_and_line() works correctly for producing node source attributes, but the reporter doesn't use it.

Bug 3: Implications for the current PR

The PR currently does:

need_directive = NeedDirective(
    lineno=directive_lineno,          # derived from source .c file line numbers
    content_offset=content_offset,
    state=self.state,                 # parent's state
    state_machine=self.state_machine, # parent's state machine
)

When NeedDirective.run() internally calls self.state_machine.get_source_and_line(self.lineno), it's using the parent's state machine with a lineno derived from the C source file. The parent SM then does:

offset = directive_lineno - self.input_offset - 1
src, srcoffset = self.input_lines.info(offset)

But self.input_lines is the main RST document. directive_lineno is a line number in the C source file — it has nothing to do with the RST document's length. This lookup will return the wrong line from the RST document, or raise an IndexError.

Bug 4: The StringList items are wrong

The PR currently creates:

content = StringList(content_lines, source=src_file)

The StringList constructor generates items as [(src_file, 0), (src_file, 1), ...] — offsets starting from 0, not the actual line numbers in the source file. Even within the nested SM, get_source_and_line() would report line 1, 2, 3... rather than the true lines.

Summary: what works and what doesn't

Code path	Works correctly?
node.source, node.line = sm.get_source_and_line() inside nested SM	✅ If StringList.items are set correctly
self.reporter.warning(..., line=lineno)	❌ Reporter uses root SM; lineno is in wrong coordinate space
NeedDirective.run() calling self.state_machine.get_source_and_line(self.lineno) with parent SM	❌ Uses parent SM with a lineno from the source file
Nodes produced by nested_parse carrying source info	✅ If StringList.items are correct

Options

Option A: Fix StringList items, accept reporter limitation (recommended)

The most pragmatic approach. Fix the StringList construction so that nodes in the output have correct source attribution:

base_offset = marked_rst.source_map["start"]["row"] + parsed["content_line_offset"]
items = [(src_file, base_offset + i) for i in range(len(content_lines))]
content = StringList(content_lines, items=items)

The reporter bug is a long-standing docutils issue that affects everyone (including Include in edge cases). For build-time warnings from NeedDirective, this is acceptable — especially since NeedDirective uses DummyOptionSpec and does its own validation, making reporter messages from option parsing unlikely.

Pros: Minimal change, fixes the most user-visible issue (node source in output), doesn't fight docutils internals.
Cons: Reporter warnings during build will have wrong line numbers for @rst block content.

Option B: Use insert_input with RST text injection

Instead of parsing the directive ourselves, inject the raw RST text (with URL options pre-injected as RST field-list syntax) into the parent state machine via insert_input. This means the lines live in the root SM's input_lines and the whole parse happens in the normal docutils flow:

# Hypothetical approach:
rst_text = marked_rst.rst
# Inject options directly into the RST text before parsing
rst_text = inject_option_line(rst_text, local_url_field, local_link_name)
rst_text = inject_option_line(rst_text, remote_url_field, remote_link_name)

textlines = statemachine.string2lines(rst_text)
self.state_machine.insert_input(textlines, src_file)
return []  # nodes inserted into doctree by state machine, not returned

Pros: Eliminates the custom regex parser entirely. Full docutils parsing (multi-line arguments, complex indentation, etc.). Reporter line numbers work correctly. Source mapping works for everything.
Cons: You lose direct control over which directive class handles the text (it goes through normal directive resolution). Injecting options as RST text requires a small helper to insert :field: value lines at the right indentation. The return value is [] (nodes are inserted into the doctree by the state machine, not returned), which changes the control flow. Harder to enforce "only a single directive per block". The StringList constructor used by insert_input generates sequential offsets from 0 — to get correct source-file line offsets you'd need to bypass insert_input and splice a custom StringList with correct items directly into self.state_machine.input_lines.

Option C: Temporarily monkey-patch the reporter

Similar to how Sphinx's Include patches state_machine.insert_input to emit include-read events — temporarily redirect the reporter's get_source_and_line:

original = self.state.document.reporter.get_source_and_line
self.state.document.reporter.get_source_and_line = my_nested_sm.get_source_and_line
try:
    result = need_directive.run()
finally:
    self.state.document.reporter.get_source_and_line = original

Pros: Reporter messages point to correct source lines.
Cons: Fragile. The reporter method would need to handle both nested content AND the main document simultaneously. Not worth the complexity for build-time warnings.

Recommendation

Option A — fix the StringList items and document the limitation. The PR's approach of directly instantiating NeedDirective is sound for this use case. The reporter line-number bug is a docutils-wide issue, not something we should try to work around. The important thing is that output nodes carry correct source attribution, which the items fix achieves.

If the custom regex parser ever becomes a maintenance burden (multi-line arguments, complex indentation edge cases), Option B is the path forward — but it requires rethinking the control flow since insert_input doesn't return nodes.

Other takeaways from Sphinx's Include

• statemachine.string2lines() should be used for text normalization (tab expansion, whitespace conversion) instead of raw splitlines() — a small improvement regardless of the larger approach.
• self.env.note_included() — Sphinx's Include calls this for rebuild tracking. The PR already covers this via note_dependency on the whole source file, which is sufficient.
• Sphinx's include-read event — an extensibility hook for transforming included text before parsing. Not needed now, but a good pattern if users ever want to preprocess @rst blocks.

Here's the section to append:

---

Source code references

docutils

• StateMachine.insert_input() and ViewList/StringList: docutils/statemachine.py — insert_input at L385, get_source_and_line at L358, ViewList.__init__ (items construction) at L1071
• docutils Include directive: docutils/parsers/rst/directives/misc.py — insert_into_input_lines at L249, custom_parse at L219
• RST parser states (reporter binding bug, nested_parse, run_directive): docutils/parsers/rst/states.py — runtime_init reporter binding at L237, nested_parse at L282, run_directive at L2243

Sphinx

• Sphinx Include override (path resolution, include-read event patching): sphinx/directives/other.py L371–L419
• Sphinx LiteralInclude (file reading, dependency tracking): sphinx/directives/code.py L414–L469

This PR

• parse_single_directive() and StringList construction: analyse/utils.py
• render_marked_rst_needs() and NeedDirective instantiation: sphinx_extension/directives/src_trace.py L339–L435

[chrisjsewell]

Plan: Replace NeedDirective with add_need in render_marked_rst_needs

Summary

Replace the direct NeedDirective instantiation in render_marked_rst_needs
with a call to add_need() — the public API already used by render_needs.
This removes coupling to NeedDirective's constructor/internals and aligns
both code paths.

What NeedDirective.run() does that add_need doesn't

These are the things that would be missing if you call add_need directly:

1. Title derivation from content — NeedDirective supports
   :title_from_content: which extracts the title from the first sentence of
   the body, with max_title_length trimming. add_need expects title as a
   pre-resolved string. For @rst blocks the title is almost always the
   directive argument, so this is low risk.

2. :delete: option — NeedDirective checks for :delete: true and
   returns early. add_need has no delete parameter. Trivial to handle with
   an early continue.

3. Option classification — NeedDirective splits options into core params
   (id, status, tags, hide, collapse, style, layout, template,
   pre_template, post_template, constraints, jinja_content) vs extra
   fields vs link fields, and warns on unknowns. You'd need to pop recognized
   keys and pass the rest as **kwargs. add_need/generate_need validates
   unknown kwargs internally (raises InvalidNeedException).

4. Boolean coercion — NeedDirective coerces :title_from_content: and
   :jinja_content: from strings to bools. add_need handles hide/collapse
   coercion internally but expects jinja_content as bool | None.

5. Warning style — NeedDirective uses log_warning() with sphinx-needs
   warning subtypes. add_need raises InvalidNeedException for errors. Slight
   difference in user-facing messages.

What transfers cleanly (no loss)

• Content as StringList — add_need accepts content: str | StringList,
  preserving source mapping.
• Jinja/templates — add_need natively supports jinja_content, template,
  pre_template, post_template.
• Source info — Can construct
  NeedItemSourceDirective(docname=..., lineno=..., lineno_content=...) and pass
  via need_source=.
• Extra fields + links — Passed as **kwargs, validated by generate_need.

Implementation steps

Step 1: Pop and handle :delete:

From parsed["options"] — if truthy, continue to next block.

Step 2: Pop and handle :title_from_content:

If set, derive title from first sentence of parsed["content"]; otherwise use
parsed["argument"].

Step 3: Pop core add_need parameters

From parsed["options"]: id, status, tags, hide, collapse, style,
layout, template, pre_template, post_template, constraints,
jinja_content (coerce to bool).

Step 4: Construct NeedItemSourceDirective

With docname=self.env.docname, lineno=directive_lineno,
lineno_content=content_offset + 1.

Step 5: Call add_need()

With named params + remaining options as **kwargs, wrapped in
try/except InvalidNeedException.

Step 6: Remove NeedDirective import

If unused elsewhere.

Verification

1. Run tox -e py312-sphinx8-needs5 -- tests/test_src_trace.py — specifically
   the rst_basic and rst_mixed test cases.
2. Doctree snapshots should remain identical.
3. Verify warnings match or are acceptably different.

Relevant files

• src/sphinx_codelinks/sphinx_extension/directives/src_trace.py —
  render_marked_rst_needs to modify; render_needs as the reference pattern.
• sphinx_needs/directives/need.py — NeedDirective.run() and _get_title for
  reference.
• sphinx_needs/api/need.py — add_need full signature.
• sphinx_needs/need_item.py — NeedItemSourceDirective.

Design considerations

1. Coupling trade-off: add_need is the intended public API and more
   stable long-term. NeedDirective constructor coupling risks breakage on
   sphinx-needs upgrades.

2. :title_from_content: replication: The _get_title function in
   sphinx-needs is private. If needed, replicate ~10 lines of logic.
   Alternatively, just don't support that option for @rst blocks (warn if
   present).

3. need_source vs legacy params: The newer API prefers
   need_source=NeedItemSourceDirective(...). Use this if targeting
   sphinx-needs v6+; otherwise keep the legacy docname/lineno params for
   backward compatibility.