[ENH][BEP028] Specification update for BEP028 BIDS-Prov

[bclenet]

BEP 028: Under review

This BEP is currently under review: #2405

Please post significant discussion to that thread. Minor issues, such as typos or clarifications, may be posted as suggestions to this PR. These suggestions may be accepted or rejected without discussion, so do not make substantial proposals in this way.

Original post

This is a work in progress PR proposing a specification update for BEP028 BIDS-Prov.

Todo lists

• Road to community review - Top priorities BEP028_BIDSprov#167
• Road to community review - Secondary priorities BEP028_BIDSprov#168

Blocking points

• [schema] allow for an optional subdirectory prov/[prov-<label>] BEP028_BIDSprov#169
• [validator] provenance files are considered as sidecars BEP028_BIDSprov#172

Associated BIDS examples

• [BEP028] provenance - custom script using Nilearn #524
• [BEP028] provenance - study dataset + manual annotation #519
• [BEP028] provenance - fMRIPrep preprocessing #502
• [BEP028] provenance - SPM preprocessing #497
• [BEP028] provenance - heudiconv conversion #496
• [BEP028] provenance - dcm2niix conversion #494

Shortcuts

• RTD Preview: https://bids-specification--2099.org.readthedocs.build/en/2099/modality-agnostic-files/provenance.html

[bclenet]

Hi @yarikoptic,

As part of this PR (integration of BEP028 in the BIDS specification), I'm adding the BIDS-Prov context into the repo. For now it's under : src/modality-agnostic-files/provenance-context.json but do you see a better place where to add it ? Thanks !

[yarikoptic]

some initial notes

[satra]

looks very nice @bclenet - left some comments which should be very lightweight to address.

[satra]

any reason not to use the same uri as Id here?

[bclenet]

For the Label, I guess we want to use the name of the file (without directory path) so that is more human readable.
About AtLocation, we describe this metadata as:

For input files, this is the relative path to the file on disk.

Should we be more specific for this definition ?

[satra]

i meant to use the bids uri here as well.

[cmaumet]

Discussed w/ Boris today. We could use uri and change the spec to say "this is the path to the file on disk." (instead of "relative path").

But... Now that the id is in fact the path to the file in bids -- maybe we should consider removing AtLocation from BIDS-Prov as this only creates duplication of information. What do you think?

[satra]

the unfortunate bit here is that the bids uri is not persistent. we do need to add something to the id for uniqueness. otherwise two datasets loaded into a graph could have the same id for a same named file (e.g. sub-01/fmri/sub-01_task-rest_bold.nii.gz could be in many datasets). i don't think their id's should be the same. so i would suggest keeping the bids uri for atlocation and then consider how to get uniqueness to the ids here and elsewhere.

for a different project we have been constructing the id using a function of the checksum of the metadata associated with the node. this allows us to generate a graph without creating new nodes each time we run the process. and if the metadata changes (whether in keys and/or their values) a new node id is generated.

[cmaumet]

@satra - could we get uniqueness using the dataset name in the BIDS URI? e.g. bids:ds109:sub-01/fmri/sub-01_task-rest_bold.nii.gz is unique). I think this may be a BIDS issue more than BIDS-Prov.

Note: we might need another online meeting to discuss those remaining bits? Let me know how you'd like to proceed.

[satra]

unfortunately, the dataset name is technically a key from the dataset_description (see: https://bids-specification.readthedocs.io/en/stable/common-principles.html#examples_1), so not guaranteed to be unique across datasets. bids intentionally decoupled uniqueness or persistence from the key name and in that section on bids uri notes that while certain aspects of a scheme uri are not being used, it could be used in the future (see: https://bids-specification.readthedocs.io/en/stable/common-principles.html#future-statement).

my suggestion would be to perhaps do what is being done for uniqueness of other ids.

[cmaumet]

@satra I am sorry I don't we understand well what is the proposed update. Would it be feasible to share a small example, maybe describing how you would change lines 293-296 above? Thank you!

[satra]

a potential example:

"Id": "bids::prov#entity-acea8093",
"Label": "sub-01_task-tonecounting_bold.nii",
"AtLocation": "bids::sub-01/func/sub-01_task-tonecounting_bold.nii",

the hash acea8093 would be computed based on the other keys of this object.

[bclenet]

Hi Satra, Camille,

There is indeed an issue with the lines 293-296 above. It is the provenance record for an entity inside a provenance file (prov/[<subdir>/]prov-<label>_env.json) and therefore it can not describe a BIDS file that is inside the dataset (provenance metadata for such files are in sidecar JSONs).

So in this specific case, we can use the Id you propose (or an Id of the form urn:<uid>) as we do not require the Id to be a BIDS URI.

But this is a bit more complicated for other ways of describing entities in BIDS Prov. So I suggest we discuss this issue during a future meeting.

Concerning AtLocation, using a BIDS URI is fine for me.

[yarikoptic]

Next batch of comments and recommendations

[yarikoptic]

I love present alignments to rdfs: ... is there any property within LD PROV ecosystem which could relate here?

[bclenet]

I could not find a relevant term for that, but any suggestion is welcome.

[yarikoptic]

this and other definitions (Id etc) are used multiple times across different metadata "array" properties. Looking through here is an example of how to define each once on top level and then reuse in specific constructs -- @effigies tell me if I am misguiding here:

DeidentificationMethodCodeSequence:
  name: DeidentificationMethodCodeSequence
  display_name: Deidentification Method Code Sequence
  description: |
    A sequence of code objects describing the mechanism or method use to remove the Patient's identity.
    Corresponds to [DICOM Tag 0012, 0064](https://dicomlookup.com/dicomtags/(0012,0064))
    `De-identification Method Code Sequence`.
  type: array
  items:
    type: object
    recommended:
      - CodeValue
      - CodeMeaning
      - CodingSchemeDesignator
      - CodingSchemeVersion
    properties:
      CodeValue:
        $ref: objects.metadata.CodeValue
      CodeMeaning:
        $ref: objects.metadata.CodeMeaning
      CodingSchemeDesignator:
        $ref: objects.metadata.CodingSchemeDesignator
      CodingSchemeVersion:
...

it should be quite doable by claude-code to achieve quickly, let me know if I should push such RF

[bclenet]

We could do that, but for it to work we need unified description fields, e.g.:

Id:
    name: Id
    description: |
        [Identifier](SPEC_ROOT/modality-agnostic-files/provenance.md#provenance-identifiers) for the object.

        Corresponds to [JSON-LD `@id`](https://www.w3.org/TR/json-ld11/#syntax-tokens-and-keywords).
    type: string
    format: uri
Label:
    name: Label
    description: |
        Name for the object.

        Corresponds to [RDF Schema `rdfs:label`](https://www.w3.org/TR/rdf-schema/#ch_label).

which feels more confusing to me.

[effigies]

This BEP is currently under review: #2405

Please post significant discussion to that thread. Minor issues, such as typos or clarifications, may be posted as suggestions to this PR. These suggestions may be accepted or rejected without discussion, so do not make substantial proposals in this way.