[ENH] BEP047 - Add audio/video recordings to behavioral experiments by bendichter · Pull Request #2231 · bids-standard/bids-specification

bendichter · 2025-10-25T16:42:26Z

Add comprehensive support for audio and video recordings in behavioral experiments: - Add audio file extensions (mp3, wav) and video file extensions (mp4, mkv, avi) with corresponding _audio and _video suffixes - Document usage of audio/video recordings in beh directory for capturing vocalizations, speech, facial expressions, and body movements - Add metadata schema for audio/video device information and stream properties - Include privacy warnings about personally identifiable information in human subject recordings - Update behavioral experiments title to remove "with no neural recordings" restriction, clarifying data can be stored with or without neural recordings - Add examples for file organization including multi-angle recordings and split files - Define optional entities: task, acquisition, run, recording, split

for more information, see https://pre-commit.ci

@yarikoptic

…ee macros - Change section title from 'Behavioral experiments' to 'Behavioral recordings' - Convert file tree examples to use MACROS___make_filetree_example for consistent rendering - Address review comments from @yarikoptic in PR #2231

…asets

…umentation

effigies

Overall this makes sense to me. It would be good to get some feedback from contributors to related BEPs, such as eye-tracking (20), motion (29), stimuli (44) and physio (45). Even if this PR doesn't propose adding this as an associated file to those data types, the potential is there and it's worth getting opinions and identifying potential conflicts.

cc @bids-standard/bep029 @bids-standard/bep044
cc @mszinte @julia-pfarr @oesteban (BEP020)
cc @m-miedema @smoia @SouravKulkarni (?) (BEP045)

codecov · 2025-10-28T13:29:50Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.07%. Comparing base (4235ae7) to head (07ca171).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2231   +/-   ##
=======================================
  Coverage   83.07%   83.07%           
=======================================
  Files          22       22           
  Lines        1696     1696           
=======================================
  Hits         1409     1409           
  Misses        287      287

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Co-authored-by: Chris Markiewicz <markiewicz@stanford.edu>

@CodyCBakerPhD

Per PR review discussion on thread r2989972681 (h-mayorquin proposed, @CodyCBakerPhD raised the image-vs-video tension on r3349547943): ImagePixelFormat (FFmpeg pix_fmt) deterministically encodes bit depth for any FFmpeg-readable file, so ImageBitDepth is redundant for video. However: - Common PIL modes (L, RGB, RGBA, P, ...) are implicitly 8-bit-per- channel and do not encode bit depth in the mode name. Image-domain tooling (Pillow, libtiff, PNG library) surfaces bit depth as a first-class integer rather than as part of a pix_fmt string. - For image-only sidecars whose producing tools do not naturally go through FFmpeg, ImagePixelFormat may be absent and bit depth is the only color-precision field available. - An integer is more directly discoverable for the typical researcher than the FFmpeg pix_fmt naming convention. Added as OPTIONAL with an explicit "redundant with ImagePixelFormat when both present; the two MUST agree" note in the description, so the redundancy is acknowledged rather than hidden. Co-authored-by: Heberto Mayorquin <h.mayorquin@gmail.com> Co-authored-by: Cody Baker <CodyCBakerPhD@gmail.com> Co-Authored-By: Claude Code 2.1.161 / Claude Opus 4.7 <noreply@anthropic.com>

…definitions) This branch builds off PR #2367 ("Add common media file definitions for BEP044/BEP047") and layers PR #2231 ("BEP047 - Add audio/video recordings to behavioral experiments") on top. Conflict resolutions (preferred #2367's canonical media definitions for all shared objects, kept #2231's unique behavioral additions): - extensions.yaml: kept #2367 generic descriptions for mkv/mp3/mp4/ogg/wav/avi; dropped #2231 duplicate avi/wav entries. - suffixes.yaml: kept #2367 generic audio/audiovideo/image/video suffixes; dropped #2231 duplicate behavioral-specific versions. NOTE: #2367 defines 'video' as video-only (no audio); #2231 had treated 'video' as possibly containing audio. Resolved in favor of #2367. - metadata.yaml: dropped #2231 duplicate AudioChannelCount/AudioSampleRate (kept #2367's richer versions); retained #2231-only fields (AudioDuration, AudioBitDepth, CameraPosition, Duration, FrameRate, Height, Width). - behavioral-experiments.md: kept #2367's updated examples-page URL. NOTE: #2231 uses generic metadata names (FrameRate/Width/Height/Duration) while #2367 uses prefixed names (VideoFrameRate/VideoWidth/...). Both sets coexist after this merge and should be reconciled by maintainers. Schema loads cleanly; no duplicate keys remain. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Per review on the mediafiles PR: remove redundant generic metadata and sidecar rules that re-declare what the common media file definitions (media.yaml / #2367) already provide. - metadata.yaml: drop Duration, AudioDuration, FrameRate, Height, Width (superseded by RecordingDuration, VideoFrameRate, ImageWidth/ImageHeight). Keep behavioral-specific Device, AudioBitDepth, CameraPosition. - rules/sidecars/beh.yaml: drop AudioVideoDuration/AudioStreams/VideoStreams/ ImageProperties groups (covered by media.yaml's MediaDuration/ MediaAudioProperties/MediaVideoProperties/MediaImageProperties). Keep only behavioral additions (Device/License, AudioBitDepth, CameraPosition) and fix the stale 'behimage' suffix -> 'image'. - behavioral-experiments.md: point sidecar tables at the common media groups and update JSON examples to the canonical field names. Schema loads; behavioral example validates with 0 errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

These are generic media properties usable by any modality that stores audio, so define them alongside the other common media file definitions: - flac (.flac) lossless audio extension - AudioBitDepth metadata, added as an optional field of MediaAudioProperties Moved here from the BEP047 behavioral PR per review discussion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

flac (extension) and AudioBitDepth (metadata) are generic media properties, now defined in the common media definitions (mediafiles PR, yarikoptic#3). Remove them from the BEP047 layer and the behavioral AudioBitDepthProperties sidecar group; AudioBitDepth is now an optional field of MediaAudioProperties. Device stays here as a behavioral-specific field. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add flac extension and AudioBitDepth to common media definitions

yarikoptic · 2026-06-05T15:59:02Z

A great usecase came from @xiaonansun is video recording of epileptic patients along with EEG or iEEG or other (single units) recording.

yarikoptic · 2026-06-05T16:06:29Z

and @mvdm 's lab records rodent faces for eye tracking + potential whiskers analysis etc!

Per the July 1 BEP047 discussion, generalize the camera-position field so it also covers audio recording devices (for example, microphone arrays): - Rename the `CameraPosition` metadata object to `DevicePosition` and broaden its description to any recording device. - Move it into the `AudioVideoImageDevice` sidecar group so it applies to the `audio`, `video`, `audiovideo`, and `image` suffixes, and drop the separate `CameraProperties` group. - Update the behavioral-experiments tables and example sidecar accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

# Conflicts: # src/schema/objects/extensions.yaml

The BEP047 metadata field was generalized from `CameraPosition` to `DevicePosition` (bids-standard/bids-specification#2231) so it applies to any recording device. Update the example sidecars and README accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

bendichter · 2026-07-01T16:06:59Z

Consolidation update

This PR now carries the complete BEP047 proposal in one place. It combines what were previously three separate PRs:

Add common media file definitions for BEP044/BEP047 #2367 (Yarik's common media file definitions for BEP044/BEP047) — now closed
Add BEP047 audio/video behavioral recordings on top of common media definitions yarikoptic/bids-specification#2 (BEP047 behavioral layer on top of those definitions) — now closed
the original BEP047 work on this branch

Both upstream PRs have been closed with pointers here, so #2231 is the single canonical PR to review. The branch has been merged onto current master.

What's included (12 files)

src/appendices/media-files.md — shared appendix defining media file naming, accepted formats/codecs, and common stream metadata (duration, audio/video/image properties). Applies to both BEP044 (stimuli) and BEP047 (recordings).
src/modality-specific-files/behavioral-experiments.md — the BEP047 behavioral layer: _audio, _video, _audiovideo, and _image recordings in beh/, with the recording/run/split entities and behavioral-specific device metadata.
Schema objects/rules (extensions, suffixes, metadata, rules/files/raw/beh, rules/sidecars/{beh,media}) plus the mkdocs/macro plumbing for the metadata tables.

bendichter · 2026-07-01T16:07:26Z

bids-standard/bids-examples#523 is up do date with this branch

yarikoptic

I think this is in a good shape and would allow also BEP042 to start making use of the metadata fields and macros

yarikoptic · 2026-07-07T18:39:59Z

Since this would relate to the software which would use such recordings, I would love to invite @talmo @MMathisLab @niksirbi @roaldarbol as relating to

Motion trajectories & pose extracted data for animal beh research #2057

Please have a look at this PR changes and/or rendered version at https://bids-specification--2231.org.readthedocs.build/en/2231/modality-specific-files/behavioral-experiments.html . It would be great to hear your opinion about the suggested BIDS file naming conventions for audiovideo recordings and metadata to come along in the sidecars.

edit: note -- you are welcome to submit an "official" review of the PR

niksirbi · 2026-07-16T15:45:11Z

Thanks for tagging me @yarikoptic! I'm quite busy at the moment, but would definitely want to review this. Is there a time horizon / deadline for the review period?

roaldarbol · 2026-07-19T12:25:38Z

Currently on holidays, I'll have a look in early August. :-)

bendichter added 2 commits October 25, 2025 12:31

fix(metadata): remove invalid enum reference for DatasetType

ea2e5d1

bendichter requested a review from erdalkaraca as a code owner October 25, 2025 16:42

bendichter and others added 3 commits October 25, 2025 12:59

Merge branch 'master' into audio-video-clean

e1d95e8

[pre-commit.ci] auto fixes from pre-commit.com hooks

ffccafc

for more information, see https://pre-commit.ci

Apply suggestion from @bendichter

4707a33

bendichter commented Oct 25, 2025

View reviewed changes

Comment thread src/schema/objects/metadata.yaml Outdated

Apply suggestion from @bendichter

38310aa

bendichter commented Oct 25, 2025

View reviewed changes

Comment thread src/schema/objects/metadata.yaml Outdated

Apply suggestion from @bendichter

fa98d09

bendichter commented Oct 25, 2025

View reviewed changes

Comment thread src/schema/objects/metadata.yaml Outdated

Apply suggestion from @bendichter

8884985

yarikoptic reviewed Oct 25, 2025

View reviewed changes

Comment thread src/modality-specific-files/behavioral-experiments.md Outdated

yarikoptic reviewed Oct 25, 2025

View reviewed changes

Comment thread src/modality-specific-files/behavioral-experiments.md Outdated

yarikoptic changed the title ~~SCHEMA: Add audio video~~ SCHEMA: Add audio video behavioral data support Oct 25, 2025

yarikoptic added the schema Issues related to the YAML schema representation of the specification. Patch version release. label Oct 25, 2025

bendichter added 4 commits October 26, 2025 12:55

fix(docs): update link to BIDS examples repository for behavioral dat…

7cacffd

…asets

refactor(beh): streamline description of behavioral recordings in doc…

9b262a3

…umentation

Merge branch 'master' into audio-video-clean

7bc990a

effigies reviewed Oct 27, 2025

View reviewed changes

Comment thread src/modality-specific-files/behavioral-experiments.md

effigies reviewed Oct 28, 2025

View reviewed changes

Comment thread src/modality-specific-files/behavioral-experiments.md Outdated

Comment thread src/modality-specific-files/behavioral-experiments.md Outdated

Comment thread src/schema/rules/sidecars/beh.yaml Outdated

effigies changed the title ~~SCHEMA: Add audio video behavioral data support~~ [ENH] Add audio/video recordings to behavioral experiments Oct 28, 2025

Merge branch 'master' into audio-video-clean

e8a8bf3

Update src/modality-specific-files/behavioral-experiments.md

3ee2c6c

Co-authored-by: Chris Markiewicz <markiewicz@stanford.edu>

bendichter commented Oct 28, 2025

View reviewed changes

Comment thread src/modality-specific-files/behavioral-experiments.md

Apply suggestion from @bendichter

826e25a

bendichter commented Oct 28, 2025

View reviewed changes

Comment thread src/schema/rules/files/raw/beh.yaml

Apply suggestion from @bendichter

00e3e26

yarikoptic and others added 3 commits June 3, 2026 12:35

Merge branch 'master' into mediafiles

6bf8f12

This was referenced Jun 4, 2026

Merge BEP047 audio/video (#2231) onto common media file definitions (#2367) bendichter/bids-specification#2

Closed

Add BEP047 audio/video behavioral recordings on top of common media definitions yarikoptic/bids-specification#2

Closed

bendichter and others added 2 commits June 4, 2026 16:12

bendichter mentioned this pull request Jun 4, 2026

Add flac extension and AudioBitDepth to common media definitions yarikoptic/bids-specification#3

Merged

bendichter and others added 6 commits June 4, 2026 16:21

Merge branch 'media-extra-formats' into mediafiles-bep047

1ef6197

List flac in the media-files appendix audio formats table

1330ad9

Merge branch 'media-extra-formats' into mediafiles-bep047

31c95b0

Merge pull request #3 from bendichter/media-extra-formats

811b8fc

Add flac extension and AudioBitDepth to common media definitions

Merge branch 'mediafiles-updated' into mediafiles-bep047

96a3386

bendichter and others added 2 commits July 1, 2026 10:48

Merge remote-tracking branch 'origin/master' into pr2-mediafiles-bep047

07ca171

# Conflicts: # src/schema/objects/extensions.yaml

bendichter force-pushed the audio-video-clean branch from 9418f93 to 07ca171 Compare July 1, 2026 14:58

bendichter mentioned this pull request Jul 1, 2026

[BLOG] BEP047 audio/video behavioral recordings community review bids-standard/bids-website#863

Draft

yarikoptic reviewed Jul 1, 2026

View reviewed changes

Comment thread tools/mkdocs_macros_bids/macros.py

yarikoptic approved these changes Jul 6, 2026

View reviewed changes

yarikoptic mentioned this pull request Jul 7, 2026

Potentially unify listing (table) of extensions #2461

Open

yarikoptic mentioned this pull request Jul 12, 2026

bids: needs knowledge about how to validate against BEP OpenScience-Collective/osa#346

Open

Uh oh!

Conversation

bendichter commented Oct 25, 2025 • edited by yarikoptic Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

effigies left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

yarikoptic commented Jun 5, 2026

Uh oh!

yarikoptic commented Jun 5, 2026

Uh oh!

bendichter commented Jul 1, 2026

Consolidation update

What's included (12 files)

Uh oh!

bendichter commented Jul 1, 2026

Uh oh!

Uh oh!

yarikoptic left a comment

Choose a reason for hiding this comment

Uh oh!

yarikoptic commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

niksirbi commented Jul 16, 2026

Uh oh!

roaldarbol commented Jul 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

bendichter commented Oct 25, 2025 •

edited by yarikoptic

Loading

codecov Bot commented Oct 28, 2025 •

edited

Loading

yarikoptic commented Jul 7, 2026 •

edited

Loading