bids assistant: add refs to BEPs

[yarikoptic]

re: https://develop.osa-demo.pages.dev/bids

they are present in beps.yaml on website

ATM I asked for

what do you know about formatting of neuropixels data?

and it did well for starters -- lead to BEP032 but pointed to just generic beps page and not specific bep PR or ideally even rendered for it website for the spec, potentially with a link to search for neuropixels within that website.

Also I wonder how we could potentially make it comprehend referencing specific file formats here as relevant for specific question, like here would have been awesome if it also knew about NWB

also attn @CodyCBakerPhD and @bendichter

[neuromechanist]

Implementation Steps

1. Pull and index BEPs metadata from bids-standard/bids-website

The BIDS website repo has structured YAML files under data/beps/:

• beps.yml - active BEPs (number, title, google_doc, pull_request, html_preview, leads, status, blocking)
• beps_completed.yml - merged BEPs (metadata only, content already in spec)
• beps_other.yml - other/inactive BEPs

Action: During knowledge sync, fetch beps.yml and build a BEP lookup. Each entry includes:

• BEP number, title, status
• Direct link to PR and rendered HTML preview (if exists)
• Google Doc link (for active BEPs without a PR)
• Leads and maintainers

Completed BEPs (beps_completed.yml) do NOT need content indexing since their spec content is already merged and indexed through the existing documentation sources.

Watch for changes: Sync should monitor beps.yml for newly added BEPs and index their content automatically.

Data source: https://github.com/bids-standard/bids-website/tree/main/data/beps

2. Index active BEP specification content (not just metadata)

The metadata alone is not enough; the assistant needs the actual BEP content to answer questions like "how do I format neuropixels data?" from context rather than hardcoded mappings.

For BEPs with PRs (preferred, structured content):

• Use the GitHub API to list files changed in the PR (/pulls/{number}/files)
• BEP PRs typically add specification markdown under src/modality-specific-files/ or src/derivatives/ (e.g., PR #1705 adds src/modality-specific-files/microelectrode-electrophysiology.md)
• They also add schema YAMLs under src/schema/ (entities, metadata, suffixes, rules)
• Action: For each BEP PR, fetch the .md files from the PR branch via raw GitHub content. Index these as retrievable documents for the assistant, tagged with the BEP number.
• Example: BEP032 PR #1705 -> fetch microelectrode-electrophysiology.md from the bep032 branch

For BEPs with only Google Docs (no PR yet):

• Need a systematic Google Doc -> Markdown conversion pipeline
• Options:
  • Use Google Docs export API (/export?mimeType=text/plain or text/markdown via docs API)
  • Use pandoc to convert the exported HTML to markdown
  • Use a library like google-docs-to-markdown
• Action: For each BEP Google Doc URL, export and convert to markdown during sync. Store alongside the BEP metadata. This will need a Google API key or service account.
• Consider caching/rate-limiting since Google Docs content changes less frequently

3. Add a lookup_bep tool to the BIDS assistant

A tool that takes a BEP number or search query and returns:

• BEP metadata (title, status, leads, links)
• Relevant specification content snippets from the indexed markdown
• Direct links to the PR, HTML preview, or Google Doc

This lets the assistant answer from the actual BEP specification text rather than generic pointers.

4. Enhance system prompt for BEP-aware responses

Update the BIDS config.yaml system prompt to instruct the assistant to:

• When a topic maps to a BEP, use lookup_bep to get the spec content and direct links
• Link to the specific BEP PR or rendered spec preview, not just the generic BEPs page
• Cite the BEP number explicitly (e.g., "This is covered by BEP032 - Microelectrode Electrophysiology")
• When the BEP content mentions related formats or tools, surface those too (e.g., NWB for neuropixels emerges naturally from the BEP032 spec text)

---

Notes

PR file patterns observed:

• BEP032 (PR #1705): src/modality-specific-files/microelectrode-electrophysiology.md + schema YAMLs
• BEP044 (PR #2022): src/modality-specific-files/stimuli.md + src/modality-agnostic-files/events.md + schema YAMLs
• Pattern: main spec content is in src/**/*.md, schema definitions in src/schema/**/*.yaml

Active BEPs with PRs (can index directly): 011, 012, 016, 020, 021, 028, 032, 034, 042, 044, 045, 047
Active BEPs with only Google Docs (need conversion): 004, 014, 017, 023, 024, 026, 033, 035, 036, 037, 038, 039, 040, 041, 043, 046

[neuromechanist]

Also, please note that the demo pages now moved to a permanent location at:

https://demo.osc.earth/

Abd develop and other feature branches can be found at:

https://develop-demo.osc.earth

{brancb}-demo.osc.earth (auto generated by CI)

[neuromechanist]

If you ask this question again from https://develop-demo.osc.earth/:

what do you know about formatting of neuropixels data?

I'll search for information about neuropixels data formatting in BIDS, as this is likely covered by a BIDS Extension Proposal (BEP).Let me search more broadly for microelectrode electrophysiology, which is the BIDS extension that covers neuropixels and similar high-density probe data.Excellent! I found BEP032: Microelectrode electrophysiology, which covers neuropixels and other high-density microelectrode array data.

Key Information

Status: Closed (this BEP has been finalized and integrated into the BIDS specification)

Leads: Sylvain Takerkart, Benjamin Dichter, Yaroslav O. Halchenko, Lyuba Zehl, Andrew Davison

Resources:

• Full specification preview: https://bids-specification--1705.org.readthedocs.build/en/1705/modality-specific-files/microelectrode-electrophysiology.html
• GitHub PR: [ENH] microelectrode electrophysiology specification (BEP032) bids-standard/bids-specification#1705
• Collaborative document: https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/

What This Covers

BEP032 defines BIDS conventions for microelectrode electrophysiology data, including:

• Neuropixels probes and other high-density electrode arrays
• Spike sorting outputs and unit information
• Electrode positions and geometry
• Metadata requirements for probe specifications and recording parameters
• File naming and organization for extracellular recordings

Since this BEP is closed, the microelectrode electrophysiology specification is now part of the official BIDS standard. I recommend consulting the preview link above for the complete formatting guidelines, file naming conventions, required metadata fields, and examples for organizing neuropixels data in BIDS format.

Would you like help with a specific aspect of formatting neuropixels data, such as file naming, required metadata, or electrode coordinate specifications?

Note that it says wrongly that this is merged, as the PR is closed and moved to another PR, but PR1705 is still mentioned as the PR for the BEP in beps.yml:
https://github.com/bids-standard/bids-website/blob/main/data/beps/beps.yml#L293

I think this issue is completed by #169 and will soon be available on the production as well.

[yarikoptic]

FWIW, trying on https://develop-demo.osc.earth/bids looked great to me.