Add BEP047 audio/video behavioral recordings on top of common media definitions

src/modality-specific-files/behavioral-experiments.md

@@ -1,4 +1,4 @@
-# Behavioral experiments (with no neural recordings)
+# Behavioral recordings
 
 !!! example "Example datasets"
 
@@ -15,19 +15,14 @@ and a guide for using macros can be found at
 -->
 {{ MACROS___make_filename_template("raw", datatypes=["beh"]) }}
 
-In addition to logs from behavioral experiments
-performed alongside imaging data acquisitions,
-one MAY also include data from experiments
-performed with no neural recordings.
-The results of those experiments MAY be stored in the `beh` directory
-using the same formats for event timing (`_events.tsv`),
-metadata (`_events.json`),
-physiological (`_physio.tsv.gz`, `_physio.json`)
-and other continuous recordings (`_stim.tsv.gz`, `_stim.json`)
-as for tasks performed during MRI, electrophysiological or other neural recordings.
-Additionally, events files
-that do not include the mandatory `onset` and `duration` columns
-MAY be included,
+The `beh` directory MAY store behavioral recordings such as audio (`_audio.*`), video (`_video.*`), combined audio-video (`_audiovideo.*`), and still image (`_image.*`) recordings, physiological (`_physio.*`) recordings, and other continuous recordings (`_stim.tsv.gz`, `_stim.json`).
+Audio, video, audio-video, and image recordings MAY be of subjects performing tasks, resting-state behavior, or recordings of stimuli being presented to the subject.
+Audio/video recordings MAY occur simultaneously with other recordings, such as BOLD or EEG.
+Relative timing between files may be determined by consulting the `scans.tsv` file.
+If no `scans.tsv` file is present, the alignment is undefined.
+The `beh` directory MAY also contain event timing files (`_events.tsv`) and their associated metadata (`_events.json`) for behavioral experiments that do not have corresponding neuroimaging or functional data.
+
+Additionally, events files that do not include the mandatory `onset` and `duration` columns MAY be included,
 but MUST be labeled `_beh.tsv` rather than `_events.tsv`.
 
 The following OPTIONAL columns are pre-defined for behavioral data files:
@@ -76,6 +71,234 @@ A guide for using macros can be found at
 -->
 {{ MACROS___make_sidecar_table("beh.BEHInstitutionInformation") }}
 
+## Audio, video, and audio-video recordings and images
+
+Audio and video recordings of behaving subjects MAY be stored in the `beh` directory
+using the `_audio`, `_video`, and `_audiovideo` suffixes.
+The `_audio` suffix is for audio-only recordings, `_video` for video-only recordings,
+and `_audiovideo` for recordings that contain both audio and video streams.
+These recordings are typically used to capture vocalizations, speech, facial expressions,
+body movements, or other behavioral aspects during experimental tasks or rest periods.
+
+Still images captured during behavioral experiments MAY be stored in the `beh` directory
+using the `_image` suffix.
+These images are typically used for training frames for pose estimation,
+snapshots of behavioral setups, or individual frames extracted from video recordings.
+
+!!! warning "Privacy and personally identifiable information"
+
+    Audio and video recordings and images of human subjects often contain personally identifiable
+    information (PII) such as faces, voices, and other identifying features.
+    Data curators MUST take special care to ensure compliance with applicable privacy
+    regulations (such as HIPAA in the United States, GDPR in the European Union, or other
+    local data protection laws) when handling these recordings.
+
+    These recordings are generally more suitable for internal use or for sharing
+    non-human subject data, unless appropriate privacy protections are implemented.
+
+### File formats
+
+Audio recordings MUST use one of the following extensions:
+
+-   `.flac` - Free Lossless Audio Codec
+-   `.mp3` - MPEG Audio Layer III
+-   `.ogg` - Ogg Vorbis
+-   `.wav` - Waveform Audio File Format
+
+Video and audio-video recordings MUST use one of the following extensions:
+
+-   `.mp4` - MPEG-4 Part 14
+-   `.mkv` - Matroska video container
+-   `.avi` - Audio Video Interleave
+
+Image files MUST use one of the following extensions:
+
+-   `.jpg` - JPEG image
+-   `.png` - Portable Network Graphics
+
+### Entities
+
+Audio and video files MAY use the following entities:
+
+-   `task` - OPTIONAL for audio and video recordings
+-   `acq` - OPTIONAL, can distinguish different recording setups
+-   `run` - OPTIONAL, for multiple recordings with identical parameters
+-   `recording` - OPTIONAL, to differentiate simultaneous recordings from different angles, locations, or devices
+-   `split` - OPTIONAL, for continuous recordings split into multiple files
+
+### Examples
+
+<!-- This block generates a file tree.
+A guide for using macros can be found at
+ https://github.com/bids-standard/bids-specification/blob/master/macros_doc.md
+-->
+{{ MACROS___make_filetree_example(
+   {
+   "sub-01": {
+      "beh": {
+         "sub-01_task-rest_video.mp4": "",
+         "sub-01_task-rest_video.json": "",
+         "sub-01_task-interview_audiovideo.mp4": "",
+         "sub-01_task-interview_audiovideo.json": "",
+         "sub-01_task-stroop_recording-face_video.mp4": "",
+         "sub-01_task-stroop_recording-face_video.json": "",
+         "sub-01_task-stroop_recording-room_video.mp4": "",
+         "sub-01_task-stroop_recording-room_video.json": "",
+         "sub-01_task-rest_image.jpg": "",
+         "sub-01_task-rest_image.json": "",
+         "sub-01_task-vocalization_audio.wav": "",
+         "sub-01_task-vocalization_audio.json": "",
+         },
+      },
+   }
+) }}
+
+For continuous recordings split into multiple files:
+
+<!-- This block generates a file tree.
+A guide for using macros can be found at
+ https://github.com/bids-standard/bids-specification/blob/master/macros_doc.md
+-->
+{{ MACROS___make_filetree_example(
+   {
+   "sub-01": {
+      "ses-01": {
+         "beh": {
+            "sub-01_ses-01_task-freeplay_run-01_split-001_video.mp4": "",
+            "sub-01_ses-01_task-freeplay_run-01_split-002_video.mp4": "",
+            "sub-01_ses-01_task-freeplay_run-01_split-003_video.mp4": "",
+            "sub-01_ses-01_task-freeplay_run-01_video.json": "",
+            },
+         },
+      },
+   }
+) }}
+
+### Sidecar JSON for audio, video, audio-video recordings, and images
+
+The following metadata fields are available for audio, video, audio-video recordings, and images:
+
+<!-- This block generates a metadata table.
+These tables are defined in
+  src/schema/rules/sidecars
+The definitions of the fields specified in these tables may be found in
+  src/schema/objects/metadata.yaml
+A guide for using macros can be found at
+ https://github.com/bids-standard/bids-specification/blob/master/macros_doc.md
+-->
+{{ MACROS___make_sidecar_table("beh.AudioVideoImageDevice") }}
+
+!!! note "Licensing for recordings containing participants"
+
+    Audio, video, and image recordings of participants may have different licensing
+    restrictions than the main dataset due to privacy considerations. The optional
+    `License` field can be used to specify different terms for individual recordings
+    that contain identifiable participant data. If not specified, the recording
+    inherits the license from `dataset_description.json`.
+
+{{ MACROS___make_sidecar_table("beh.AudioVideoDuration") }}
+
+The following fields are available for audio recordings (`_audio`) and audio-video recordings (`_audiovideo`):
+
+{{ MACROS___make_sidecar_table("beh.AudioStreams") }}
+
+The following fields are available for video recordings (`_video`) and audio-video recordings (`_audiovideo`):
+
+{{ MACROS___make_sidecar_table("beh.VideoStreams") }}
+
+The following fields are available for image files (`_image`):
+
+{{ MACROS___make_sidecar_table("beh.AudioVideoImageDevice") }}
+
+{{ MACROS___make_sidecar_table("beh.ImageProperties") }}
+
+### Example audio-video sidecar JSON
+
+For an audio-video file containing both video and audio streams:
+
+```JSON
+{
+  "TaskName": "RestingState",
+  "Device": "Sony FDR-AX53",
+  "AudioChannelCount": 2,
+  "AudioSampleRate": 48000,
+  "FrameRate": 30.0,
+  "Height": 1080,
+  "Width": 1920,
+  "Duration": 600.5
+}
+```
+
+### Example video sidecar JSON
+
+For a video-only recording:
+
+```JSON
+{
+  "TaskName": "RestingState",
+  "Device": "Sony FDR-AX53",
+  "FrameRate": 30.0,
+  "Height": 1080,
+  "Width": 1920,
+  "Duration": 600.5
+}
+```
+
+### Example audio sidecar JSON
+
+For an audio-only recording:
+
+```JSON
+{
+  "TaskName": "Vocalization",
+  "Device": "Zoom H6 Handy Recorder",
+  "AudioChannelCount": 2,
+  "AudioSampleRate": 44100,
+  "Duration": 300.2
+}
+```
+
+### Example image sidecar JSON
+
+For a still image:
+
+```JSON
+{
+  "TaskName": "Reaching",
+  "Device": "GoPro Hero 10",
+  "Height": 1080,
+  "Width": 1920,
+  "CameraPosition": "overhead"
+}
+```
+
+### Annotations and events
+
+Behavioral annotations or event markers for audio and video recordings
+SHOULD be stored in accompanying `_events.tsv` files following the standard
+[events file format](../modality-agnostic-files/events.md).
+These events files use the same filename entities as the audio/video file they describe,
+but with the `_events` suffix.
+
+For example:
+
+<!-- This block generates a file tree.
+A guide for using macros can be found at
+ https://github.com/bids-standard/bids-specification/blob/master/macros_doc.md
+-->
+{{ MACROS___make_filetree_example(
+   {
+   "sub-01": {
+      "beh": {
+         "sub-01_task-speech_audio.wav": "",
+         "sub-01_task-speech_audio.json": "",
+         "sub-01_task-speech_events.tsv": "",
+         "sub-01_task-speech_events.json": "",
+         },
+      },
+   }
+) }}
+
 ## Example `_beh.tsv`
 
 ```tsv

src/schema/objects/extensions.yaml

@@ -126,6 +126,12 @@ fif:
   display_name: Functional Imaging File Format
   description: |
     An MEG file format used by Neuromag, Elekta, and MEGIN.
+flac:
+  value: .flac
+  display_name: Free Lossless Audio Codec
+  description: |
+    A [FLAC](https://en.wikipedia.org/wiki/FLAC) audio file.
+    This format is commonly used for behavioral audio recordings.
 jpg:
   value: .jpg
   display_name: Joint Photographic Experts Group Format

src/schema/objects/metadata.yaml

@@ -959,6 +959,13 @@ Descriptors:
     - type: array
       items:
         type: string
+Device:
+  name: Device
+  display_name: Device
+  description: |
+    Free-form description of the device used to record the data
+    (for example, `"iPhone 12"`, `"Canon EOS R5"`).
+  type: string
 DeviceSerialNumber:
   name: DeviceSerialNumber
   display_name: Device Serial Number
@@ -4762,3 +4769,65 @@ iEEGReference:
     this field should have a general description and the channel specific
     reference should be defined in the `channels.tsv` file.
   type: string
+
+AudioDuration:
+  name: AudioDuration
+  display_name: Audio Duration
+  description: |
+    Duration of the audio recording in seconds.
+  type: number
+  exclusiveMinimum: 0
+  unit: s
+
+AudioBitDepth:
+  name: AudioBitDepth
+  display_name: Audio Bit Depth
+  description: |
+    Number of bits per sample in the audio recording.
+
+    Common values include `16`, `24`, or `32`.
+  type: integer
+  minimum: 1
+
+CameraPosition:
+  name: CameraPosition
+  display_name: Camera Position
+  description: |
+    Free-form description of the camera placement relative to the subject or scene.
+
+    Examples include "front", "profile-left", "ceiling", "room-corner", or "overhead".
+  type: string
+
+Duration:
+  name: Duration
+  display_name: Duration
+  description: |
+    Total duration of the audio or video recording in seconds.
+  type: number
+  exclusiveMinimum: 0
+  unit: s
+
+FrameRate:
+  name: FrameRate
+  display_name: Frame Rate
+  description: |
+    Frame rate of the video recording in frames per second (for example, `30.0`).
+  type: number
+  exclusiveMinimum: 0
+  unit: Hz
+
+Height:
+  name: Height
+  display_name: Video Height
+  description: |
+    Height of the video in pixels (for example, `1080`).
+  type: integer
+  minimum: 1
+
+Width:
+  name: Width
+  display_name: Video Width
+  description: |
+    Width of the video in pixels (for example, `1920`).
+  type: integer
+  minimum: 1