[BEP028] provenance - study dataset + manual annotation

.github/workflows/validate_datasets.yml

@@ -139,6 +139,10 @@ jobs:
             exit 1;
         fi
 
+    - name: Skip legacy validation for post-legacy datasets
+      run: for DS in mrs_* dwi_deriv pet006 pheno004 volume_timing provenance_*; do touch $DS/.SKIP_VALIDATION; done
+      if: matrix.bids-validator == 'legacy'
+
     - name: Skip stable validation for datasets with unreleased validator features
       # Replace ${EMPTY} with dataset patterns, when this is needed
       # Reset to "for DS in ${EMPTY}; ..." after a validator release
@@ -148,7 +152,7 @@ jobs:
     - name: Skip main validation for datasets with unreleased spec features
       # Replace ${EMPTY} with dataset patterns, when this is needed
       # Reset to "for DS in ${EMPTY}; ..." after a spec release
-      run: for DS in eyetracking_* atlas-* emg_*; do touch $DS/.SKIP_VALIDATION; done
+      run: for DS in dwi_deriv pheno004 provenance_* eyetracking_* atlas-* emg_*; do touch $DS/.SKIP_VALIDATION; done
       if: matrix.bids-validator != 'dev'
 
     - name: Set BIDS_SCHEMA variable for dev version
@@ -157,7 +161,7 @@ jobs:
       # Update this URL to the schema.json from PRs to the spec, when needed.
       # If this variable is unset, dev will generally track the latest development
       # release of https://jsr.io/@bids/schema
-      run: echo BIDS_SCHEMA=https://bids-specification.readthedocs.io/en/latest/schema.json >> $GITHUB_ENV
+      run: echo BIDS_SCHEMA=https://bids-specification--2099.org.readthedocs.build/en/2099/schema.json >> $GITHUB_ENV
 
     - name: Set BIDS_SCHEMA variable for dev-prs version
       if: matrix.bids-validator == 'dev-prs' && env.SPEC_PR != ''

provenance_manual/README.md

@@ -0,0 +1,80 @@
+# Provenance of manual segmentations
+
+## Goal
+
+This example aims at showing provenance metadata for a study dataset in which several experts performed different types of manual segmentations from the same T1w file.
+
+* dataset `raw` contains the input T1w file;
+* derived dataset `seg-brain` contains manual brain segmentations;
+* derived dataset `seg-lesions` contains manual segmentations of lesions.
+
+## Directory tree
+
+> [!NOTE]
+> Note that the `docs/` directories contains explanatory data (see [Provenance as a RDF graph](#provenance-as-a-rdf-graph)) that is not required to encode provenance.
+
+```
+.
+├── dataset_description.json
+├── derivatives
+│   ├── seg-brain
+│   │   ├── dataset_description.json
+│   │   ├── descriptions.tsv
+│   │   ├── docs
+│   │   │   ├── prov-seg.jsonld
+│   │   │   └── prov-seg.png
+│   │   ├── prov
+│   │   │   ├── provenance.tsv
+│   │   │   ├── prov-seg_act.json
+│   │   │   └── prov-seg_ent.json
+│   │   └── sub-001
+│   │       └── anat
+│   │           ├── sub-001_space-orig_desc-exp1_dseg.json
+│   │           ├── sub-001_space-orig_desc-exp1_dseg.nii.gz
+│   │           ├── sub-001_space-orig_desc-exp2_dseg.json
+│   │           └── sub-001_space-orig_desc-exp2_dseg.nii.gz
+│   └── seg-lesions
+│       ├── dataset_description.json
+│       ├── descriptions.tsv
+│       ├── docs
+│       │   ├── prov-seg.jsonld
+│       │   └── prov-seg.png
+│       ├── prov
+│       │   ├── provenance.tsv
+│       │   ├── prov-seg_act.json
+│       │   └── prov-seg_ent.json
+│       └── sub-001
+│           └── anat
+│               ├── sub-001_space-orig_desc-exp1_dseg.json
+│               ├── sub-001_space-orig_desc-exp1_dseg.nii.gz
+│               ├── sub-001_space-orig_desc-exp2_dseg.json
+│               └── sub-001_space-orig_desc-exp2_dseg.nii.gz
+├── docs
+│   ├── prov-study.jsonld
+│   └── prov-study.png
+├── README.md
+└── sourcedata
+    └── raw
+        ├── dataset_description.json
+        └── sub-001
+            └── anat
+                ├── sub-001_T1w.json
+                └── sub-001_T1w.nii.gz
+```
+
+## Provenance as a RDF graph
+
+For each derived dataset, provenance metadata can be aggregated as JSON-LD RDF graphs, which are available inside the `docs` directory:
+* [`derivatives/seg-brain/docs/prov-seg.jsonld`](derivatives/seg-brain/docs/prov-seg.jsonld)
+* [`derivatives/seg-lesions/docs/prov-lesions.jsonld`](derivatives/seg-lesions/docs/prov-seg.jsonld)
+
+These two graphs can be be merged into a study-level provenance graph available here:
+* [`docs/prov-study.jsonld`](docs/prov-study.jsonld)
+
+Here are the rendered versions of the three graphs, also available in each `docs` directory.
+* [`derivatives/seg-brain/docs/prov-seg.png`](derivatives/seg-brain/docs/prov-seg.png).
+![Rendered version of the RDF graph for derivative dataset seg-brain](derivatives/seg-brain/docs/prov-seg.png)
+* [`derivatives/seg-lesions/docs/prov-lesions.png`](derivatives/seg-lesions/docs/prov-seg.png).
+![Rendered version of the RDF graph for derivative dataset seg-lesions](derivatives/seg-lesions/docs/prov-seg.png)
+* [`docs/prov-seg.png`](docs/prov-study.png).
+![Rendered version of the RDF graph for the study dataset](docs/prov-study.png)

provenance_manual/dataset_description.json

@@ -0,0 +1,13 @@
+{
+  "Name": "Brain segmentations from 2 experts",
+  "BIDSVersion": "1.10.0",
+  "DatasetType": "study",
+  "License": "CC0",
+  "Authors": [
+    "Boris Clénet"
+  ],
+  "DatasetLinks": {
+    "seg-brain": "derivatives/seg-brain",
+    "seg-lesions": "derivatives/seg-lesions"
+  }
+}

provenance_manual/derivatives/seg-brain/dataset_description.json

@@ -0,0 +1,12 @@
+{
+  "Name": "Manual brain segmentations",
+  "BIDSVersion": "1.10.0",
+  "DatasetType": "derivative",
+  "License": "CC0",
+  "Authors": [
+    "Boris Clénet"
+  ],
+  "DatasetLinks": {
+    "raw": "../../sourcedata/raw"
+  }
+}

provenance_manual/derivatives/seg-brain/descriptions.tsv

@@ -0,0 +1,3 @@
+desc_id	description
+desc-seg1	Files generated by expert #1
+desc-seg2	Files generated by expert #2

provenance_manual/derivatives/seg-brain/docs/prov-seg.jsonld

@@ -0,0 +1,43 @@
+{
+  "@context": "https://bids-specification--2099.org.readthedocs.build/en/2099/provenance-context.json",
+  "Records": {
+    "Software": [],
+    "Activities": [
+      {
+        "Id": "bids::prov#segmentation-nO5RGsrb",
+        "Label": "Manual brain segmentation",
+        "Command": null,
+        "Used": [
+          "bids:raw:sub-001/anat/sub-001_T1w.nii.gz"
+        ]
+      },
+      {
+        "Id": "bids::prov#segmentation-mOOypIYB",
+        "Label": "Manual brain segmentation",
+        "Command": null,
+        "Used": [
+          "bids:raw:sub-001/anat/sub-001_T1w.nii.gz"
+        ]
+      }
+    ],
+    "Files": [
+      {
+        "Id": "bids::sub-001/anat/sub-001_space-orig_desc-exp1_dseg.nii.gz",
+        "Label": "sub-001_space-orig_desc-exp1_dseg.nii.gz",
+        "AtLocation": "sub-001/anat/sub-001_space-orig_desc-exp1_dseg.nii.gz",
+        "GeneratedBy": "bids::prov#segmentation-nO5RGsrb",
+        "Type": "http://uri.interlex.org/base/ilx_0744009"
+      },
+      {
+        "Id": "bids::sub-001/anat/sub-001_space-orig_desc-exp2_dseg.nii.gz",
+        "Label": "sub-001_space-orig_desc-exp2_dseg.nii.gz",
+        "AtLocation": "sub-001/anat/sub-001_space-orig_desc-exp2_dseg.nii.gz",
+        "GeneratedBy": "bids::prov#segmentation-mOOypIYB",
+        "Type": "http://uri.interlex.org/base/ilx_0744009"
+      }
+    ],
+    "Datasets": [],
+    "prov:Entity": [],
+    "Environments": []
+  }
+}

provenance_manual/derivatives/seg-brain/prov/prov-seg_act.json

@@ -0,0 +1,20 @@
+{
+  "Activities": [
+    {
+      "Id": "bids::prov#segmentation-nO5RGsrb",
+      "Label": "Manual brain segmentation",
+      "Command": null,
+      "Used": [
+        "bids:raw:sub-001/anat/sub-001_T1w.nii.gz"
+      ]
+    },
+    {
+      "Id": "bids::prov#segmentation-mOOypIYB",
+      "Label": "Manual brain segmentation",
+      "Command": null,
+      "Used": [
+        "bids:raw:sub-001/anat/sub-001_T1w.nii.gz"
+      ]
+    }
+  ]
+}

provenance_manual/derivatives/seg-brain/prov/provenance.tsv

@@ -0,0 +1,2 @@
+provenance_label	description
+prov-seg	Manual brain segmentation performed by two experts

provenance_manual/derivatives/seg-brain/sub-001/anat/sub-001_space-orig_desc-exp1_dseg.json

@@ -0,0 +1,4 @@
+{
+  "GeneratedBy": "bids::prov#segmentation-nO5RGsrb",
+  "Type": "http://uri.interlex.org/base/ilx_0744009"
+}

provenance_manual/derivatives/seg-brain/sub-001/anat/sub-001_space-orig_desc-exp2_dseg.json

@@ -0,0 +1,4 @@
+{
+  "GeneratedBy": "bids::prov#segmentation-mOOypIYB",
+  "Type": "http://uri.interlex.org/base/ilx_0744009"
+}

provenance_manual/derivatives/seg-lesions/dataset_description.json

@@ -0,0 +1,12 @@
+{
+  "Name": "Manual lesions segmentations",
+  "BIDSVersion": "1.10.0",
+  "DatasetType": "derivative",
+  "License": "CC0",
+  "Authors": [
+    "Boris Clénet"
+  ],
+  "DatasetLinks": {
+    "raw": "../../sourcedata/raw"
+  }
+}

provenance_manual/derivatives/seg-lesions/descriptions.tsv

@@ -0,0 +1,3 @@
+desc_id	description
+desc-seg1	Files generated by expert #1
+desc-seg2	Files generated by expert #2

provenance_manual/derivatives/seg-lesions/docs/prov-seg.jsonld

@@ -0,0 +1,41 @@
+{
+  "@context": "https://bids-specification--2099.org.readthedocs.build/en/2099/provenance-context.json",
+  "Records": {
+    "Software": [],
+    "Activities": [
+      {
+        "Id": "bids::prov#segmentation-q9FNOUjZ",
+        "Label": "Manual lesion segmentation",
+        "Command": null,
+        "Used": [
+          "bids:raw:sub-001/anat/sub-001_T1w.nii.gz"
+        ]
+      },
+      {
+        "Id": "bids::prov#segmentation-i4MyEr34",
+        "Label": "Manual lesion segmentation",
+        "Command": null,
+        "Used": [
+          "bids:raw:sub-001/anat/sub-001_T1w.nii.gz"
+        ]
+      }
+    ],
+    "Files": [
+      {
+        "Id": "bids::sub-001/anat/sub-001_space-orig_desc-exp1_dseg.nii.gz",
+        "Label": "sub-001_space-orig_desc-exp1_dseg.nii.gz",
+        "AtLocation": "sub-001/anat/sub-001_space-orig_desc-exp1_dseg.nii.gz",
+        "GeneratedBy": "bids::prov#segmentation-q9FNOUjZ"
+      },
+      {
+        "Id": "bids::sub-001/anat/sub-001_space-orig_desc-exp2_dseg.nii.gz",
+        "Label": "sub-001_space-orig_desc-exp2_dseg.nii.gz",
+        "AtLocation": "sub-001/anat/sub-001_space-orig_desc-exp2_dseg.nii.gz",
+        "GeneratedBy": "bids::prov#segmentation-i4MyEr34"
+      }
+    ],
+    "Datasets": [],
+    "prov:Entity": [],
+    "Environments": []
+  }
+}

provenance_manual/derivatives/seg-lesions/prov/prov-seg_act.json

@@ -0,0 +1,20 @@
+{
+  "Activities": [
+    {
+      "Id": "bids::prov#segmentation-q9FNOUjZ",
+      "Label": "Manual lesion segmentation",
+      "Command": null,
+      "Used": [
+        "bids:raw:sub-001/anat/sub-001_T1w.nii.gz"
+      ]
+    },
+    {
+      "Id": "bids::prov#segmentation-i4MyEr34",
+      "Label": "Manual lesion segmentation",
+      "Command": null,
+      "Used": [
+        "bids:raw:sub-001/anat/sub-001_T1w.nii.gz"
+      ]
+    }
+  ]
+}

provenance_manual/derivatives/seg-lesions/prov/provenance.tsv

@@ -0,0 +1,2 @@
+provenance_label	description
+prov-seg	Manual lesion segmentation performed by two experts

provenance_manual/derivatives/seg-lesions/sub-001/anat/sub-001_space-orig_desc-exp1_dseg.json

@@ -0,0 +1,3 @@
+{
+  "GeneratedBy": "bids::prov#segmentation-q9FNOUjZ"
+}

provenance_manual/derivatives/seg-lesions/sub-001/anat/sub-001_space-orig_desc-exp2_dseg.json

@@ -0,0 +1,3 @@
+{
+  "GeneratedBy": "bids::prov#segmentation-i4MyEr34"
+}

provenance_manual/docs/prov-study.jsonld

@@ -0,0 +1,79 @@
+{
+  "@context": "https://bids-specification--2099.org.readthedocs.build/en/2099/provenance-context.json",
+  "Records": {
+    "Software": [],
+    "Activities": [
+      {
+        "Id": "bids:seg-brain:prov#segmentation-nO5RGsrb",
+        "Label": "Manual brain segmentation",
+        "Command": null,
+        "Used": [
+          "bids:raw:sub-001/anat/sub-001_T1w.nii.gz"
+        ]
+      },
+      {
+        "Id": "bids:seg-brain:prov#segmentation-mOOypIYB",
+        "Label": "Manual brain segmentation",
+        "Command": null,
+        "Used": [
+          "bids:raw:sub-001/anat/sub-001_T1w.nii.gz"
+        ]
+      },
+      {
+        "Id": "bids:seg-lesions:prov#segmentation-q9FNOUjZ",
+        "Label": "Manual lesion segmentation",
+        "Command": null,
+        "Used": [
+          "bids:raw:sub-001/anat/sub-001_T1w.nii.gz"
+        ]
+      },
+      {
+        "Id": "bids:seg-lesions:prov#segmentation-i4MyEr34",
+        "Label": "Manual lesion segmentation",
+        "Command": null,
+        "Used": [
+          "bids:raw:sub-001/anat/sub-001_T1w.nii.gz"
+        ]
+      }
+    ],
+    "Files": [
+      {
+        "Id": "bids:seg-brain:sub-001/anat/sub-001_space-orig_desc-exp1_dseg.nii.gz",
+        "Label": "sub-001_space-orig_desc-exp1_dseg.nii.gz",
+        "AtLocation": "seg-brain/sub-001/anat/sub-001_space-orig_desc-exp1_dseg.nii.gz",
+        "GeneratedBy": "bids:seg-brain:prov#segmentation-nO5RGsrb",
+        "Type": "http://uri.interlex.org/base/ilx_0744009"
+      },
+      {
+        "Id": "bids:seg-brain:sub-001/anat/sub-001_space-orig_desc-exp2_dseg.nii.gz",
+        "Label": "sub-001_space-orig_desc-exp2_dseg.nii.gz",
+        "AtLocation": "seg-brain/sub-001/anat/sub-001_space-orig_desc-exp2_dseg.nii.gz",
+        "GeneratedBy": "bids:seg-brain:prov#segmentation-mOOypIYB",
+        "Type": "http://uri.interlex.org/base/ilx_0744009"
+      },
+      {
+        "Id": "bids:raw:sub-001/anat/sub-001_T1w.nii.gz",
+        "Label": "sub-001/anat/sub-001_T1w.nii.gz",
+        "AtLocation": "sourcedata/raw/sub-001/anat/sub-001_T1w.nii.gz",
+        "Digest": {
+          "SHA-256": "545d4c7e7c9e73275f4b3cbffbecc93b522afa97da2325b73737f3777c5395bf"
+        }
+      },
+      {
+        "Id": "bids:seg-lesions:sub-001/anat/sub-001_space-orig_desc-exp1_dseg.nii.gz",
+        "Label": "sub-001_space-orig_desc-exp1_dseg.nii.gz",
+        "AtLocation": "seg-lesions/sub-001/anat/sub-001_space-orig_desc-exp1_dseg.nii.gz",
+        "GeneratedBy": "bids:seg-lesions:prov#segmentation-q9FNOUjZ"
+      },
+      {
+        "Id": "bids:seg-lesions:sub-001/anat/sub-001_space-orig_desc-exp2_dseg.nii.gz",
+        "Label": "sub-001_space-orig_desc-exp2_dseg.nii.gz",
+        "AtLocation": "seg-lesions/sub-001/anat/sub-001_space-orig_desc-exp2_dseg.nii.gz",
+        "GeneratedBy": "bids:seg-lesions:prov#segmentation-i4MyEr34"
+      }
+    ],
+    "Datasets": [],
+    "prov:Entity": [],
+    "Environments": []
+  }
+}