[ENH] Define PET surface map derivatives

[bendhouseart]

No description provided.

[bendhouseart]

oh that's failing a lot of the bst tests....

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.07%. Comparing base (75ceda1) to head (1a73eaf).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2416   +/-   ##
=======================================
  Coverage   83.07%   83.07%           
=======================================
  Files          22       22           
  Lines        1696     1696           
=======================================
  Hits         1409     1409           
  Misses        287      287           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bendhouseart]

Removed tpl entity filter for raw rules following Ross' PR to the validator bids-standard/bids-validator#399.

[bendhouseart]

@bendhouseart need to test on Cyrus' atlases examples next.

[effigies]

This is introducing the hemisphere entity into volumetric data. Seems worth a discussion, not a [SCHEMA] PR that describes itself as a fix.

From the data I saw, you have values on a surface, not volumetric data, so the appropriate extension would be .func.gii:

surface_map:
  $ref:
    - meta.templates.deriv.surface
  extensions:
    - .func.gii
    - .json

(I understand that FreeSurfer supports putting surface data in NIfTIs, and NiBabel is capable of reading those files, but it is not a documented NIfTI extension, and I would want to see support from the other major software projects before boosting it in BIDS.)

You would then do:

pet_pet_surface:
  $ref:
    - meta.templates.deriv.surface_map
    - rules.files.raw.pet.pet
  entities:
    $ref:
      - meta.templates.deriv.surface_map.entities
      - rules.files.raw.pet.pet.entities

pet_pet_surface_atlas:
  $ref: rules.files.deriv.imaging.pet_pet_surface
  entities:
    $ref:
      - meta.templates.atlas.base.entities
      - rules.files.deriv.imaging.pet_pet_surface.entities

If it's common to work with CIFTI, you could add .dscalar.nii to the extensions.

[melanieganz]

Dear @bendhouseart ,
I had discussed the above things with @effigies when he was in Cph. I did the changes that Chris requested regarding (hemi and the gifti format instead of nifti). When I test this with the dev validator (https://bids-standard.github.io/bids-validator/dev/) and link it to the ReadTheDocs of this PR (https://bids-specification--2416.org.readthedocs.build/en/2416/schema.json), I still get an error:

Multiple filename rules were found as potential matches. All of them had at least one issue during filename validation.
/tpl-fsaverage/pet/tpl-fsaverage_hemi-L_atlas-NRUSV2A2024_desc-BmaxCOV_pet.func.gii (Rules that matched with issues: rules.files.deriv.imaging.pet_pet_volumetric, rules.files.deriv.imaging.pet_pet_volumetric_atlas, rules.files.raw.pet.pet)

So I am not sure if all issues are resolved? 

[bendhouseart]

Small issues, having removed "anti" selectors following Ross' update with this PR, and yes the dataset I'm validating is a derivative dataset:

{
  "Name": "NRU SV2A atlas converted to BIDS derivatives",
  "BIDSVersion": "1.11.1",
  "DatasetType": "derivative",
  "License": "CC-BY-NC-SA-4.0",
  "GeneratedBy": [
    {
      "Name": "convertNRUatlasses.py",
      "Version": "1.0.0",
      "Description": "Downloads NRU SV2A atlas and renames files to BIDS atlas-derivatives style"
    }
  ]
}

Expand details below to see the error.

Details

anthony@computer:~/Projects/bids-validator$ ./local-run ../bids-examples/atlas-sv2a/ --ignoreNiftiHeaders --ignoreWarnings --schema file:///home/anthony/Projects/bids-validator/atlas_fixe.json --verbose
        [ERROR] MISSING_REQUIRED_ENTITY Missing required entity for files with this suffix.
                /tpl-MNI152/pet/tpl-MNI152_atlas-NRUSV2A2024_desc-BmaxMean_pet.json - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-MNI152/pet/tpl-MNI152_atlas-NRUSV2A2024_desc-BmaxCOV_pet.gii.gz - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-MNI152/pet/tpl-MNI152_atlas-NRUSV2A2024_desc-BmaxSD_pet.json - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-MNI152/pet/tpl-MNI152_atlas-NRUSV2A2024_desc-BmaxSD_pet.gii.gz - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-MNI152/pet/tpl-MNI152_atlas-NRUSV2A2024_desc-BmaxCOV_pet.json - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-MNI152/pet/tpl-MNI152_atlas-NRUSV2A2024_desc-BmaxMean_pet.gii.gz - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-R_atlas-NRUSV2A2024_desc-BmaxCOV_pet.json - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-R_atlas-NRUSV2A2024_desc-BmaxMean_pet.json - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-R_atlas-NRUSV2A2024_desc-BmaxSD_pet.json - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-L_atlas-NRUSV2A2024_desc-BmaxSD_pet.nii.gz - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-L_atlas-NRUSV2A2024_desc-BmaxMean_pet.nii.gz - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-R_atlas-NRUSV2A2024_desc-BmaxMean_pet.nii.gz - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-L_atlas-NRUSV2A2024_desc-BmaxCOV_pet.json - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-R_atlas-NRUSV2A2024_desc-BmaxSD_pet.nii.gz - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-L_atlas-NRUSV2A2024_desc-BmaxCOV_pet.nii.gz - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-R_atlas-NRUSV2A2024_desc-BmaxCOV_pet.nii.gz - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-L_atlas-NRUSV2A2024_desc-BmaxSD_pet.json - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-L_atlas-NRUSV2A2024_desc-BmaxMean_pet.json - sub missing from rule rules.files.raw.pet.pet - rules.files.raw.pet.pet

        Please visit https://neurostars.org/search?q=MISSING_REQUIRED_ENTITY for existing conversations about this issue.

        [ERROR] ENTITY_NOT_IN_RULE Entity not listed as required or optional for files with this suffix
                /tpl-MNI152/pet/tpl-MNI152_atlas-NRUSV2A2024_desc-BmaxMean_pet.json - tpl, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-MNI152/pet/tpl-MNI152_atlas-NRUSV2A2024_desc-BmaxCOV_pet.gii.gz - tpl, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-MNI152/pet/tpl-MNI152_atlas-NRUSV2A2024_desc-BmaxSD_pet.json - tpl, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-MNI152/pet/tpl-MNI152_atlas-NRUSV2A2024_desc-BmaxSD_pet.gii.gz - tpl, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-MNI152/pet/tpl-MNI152_atlas-NRUSV2A2024_desc-BmaxCOV_pet.json - tpl, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-MNI152/pet/tpl-MNI152_atlas-NRUSV2A2024_desc-BmaxMean_pet.gii.gz - tpl, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-R_atlas-NRUSV2A2024_desc-BmaxCOV_pet.json - tpl, hemi, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-R_atlas-NRUSV2A2024_desc-BmaxMean_pet.json - tpl, hemi, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-R_atlas-NRUSV2A2024_desc-BmaxSD_pet.json - tpl, hemi, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-L_atlas-NRUSV2A2024_desc-BmaxSD_pet.nii.gz - tpl, hemi, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-L_atlas-NRUSV2A2024_desc-BmaxMean_pet.nii.gz - tpl, hemi, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-R_atlas-NRUSV2A2024_desc-BmaxMean_pet.nii.gz - tpl, hemi, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-L_atlas-NRUSV2A2024_desc-BmaxCOV_pet.json - tpl, hemi, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-R_atlas-NRUSV2A2024_desc-BmaxSD_pet.nii.gz - tpl, hemi, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-L_atlas-NRUSV2A2024_desc-BmaxCOV_pet.nii.gz - tpl, hemi, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-R_atlas-NRUSV2A2024_desc-BmaxCOV_pet.nii.gz - tpl, hemi, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-L_atlas-NRUSV2A2024_desc-BmaxSD_pet.json - tpl, hemi, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet
                /tpl-fsaverage/pet/tpl-fsaverage_hemi-L_atlas-NRUSV2A2024_desc-BmaxMean_pet.json - tpl, hemi, atlas, desc not in rule rules.files.raw.pet.pet - rules.files.raw.pet.pet

        Please visit https://neurostars.org/search?q=ENTITY_NOT_IN_RULE for existing conversations about this issue.

        [ERROR] EXTENSION_MISMATCH Extension used by file does not match allowed extensions for its suffix
                /tpl-MNI152/pet/tpl-MNI152_atlas-NRUSV2A2024_desc-BmaxCOV_pet.gii.gz - rules.files.raw.pet.pet
                /tpl-MNI152/pet/tpl-MNI152_atlas-NRUSV2A2024_desc-BmaxSD_pet.gii.gz - rules.files.raw.pet.pet
                /tpl-MNI152/pet/tpl-MNI152_atlas-NRUSV2A2024_desc-BmaxMean_pet.gii.gz - rules.files.raw.pet.pet

        Please visit https://neurostars.org/search?q=EXTENSION_MISMATCH for existing conversations about this issue.


          Summary:                         Available Tasks:        Available Modalities:
          20 Files, 5.53 kB                                        PET                  
          0 - Subjects 1 - Sessions                                                     


anthony@computer:~/Projects/bids-validator$ 

Again, not sure what's going on since it lookls like rules.files.raw.pet.pet still is applying to template/derivative folders. Am I misunderstanding what PR 399 to the validator did?

[melanieganz]

I am trying to test the current state with the conversions I made, but I can't get the link to the schema file right when trying to give it as an advanced option to the online validator. How should I link to the schema that you changed above, @bendhouseart? @effigies showed me, but I can't seem to reproduce it.
https://github.com/bids-standard/bids-specification/pull/2416/src/schema.json, but somehow this is not correct.

[bendhouseart]

I am trying to test the current state with the conversions I made, but I can't get the link to the schema file right when trying to give it as an advanced option to the online validator. How should I link to the schema that you changed above, @bendhouseart? @effigies showed me, but I can't seem to reproduce it. https://github.com/bids-standard/bids-specification/pull/2416/src/schema.json, but somehow this is not correct.

This link to the preview should do it -> https://bids-specification--2416.org.readthedocs.build/en/2416/schema.json

I had bundled a schema.json in this PR initially for testing, but it's not necessary given the above.

[melanieganz]

Thanks, I was able to try it now, and I still get an error using this version regarding the filenames:
Multiple filename rules were found as potential matches. All of them had at least one issue during filename validation.

/tpl-fsaverage/pet/tpl-fsaverage_hemi-L_atlas-NRUSV2A2024_desc-BmaxCOV_pet.func.gii (Rules that matched with issues: rules.files.deriv.imaging.pet_pet_volumetric, rules.files.deriv.imaging.pet_pet_volumetric_atlas, rules.files.raw.pet.pet)
 /tpl-fsaverage/pet/tpl-fsaverage_hemi-L_atlas-NRUSV2A2024_desc-BmaxMean_pet.func.gii (Rules that matched with issues: rules.files.deriv.imaging.pet_pet_volumetric, rules.files.deriv.imaging.pet_pet_volumetric_atlas, rules.files.raw.pet.pet)
 /tpl-fsaverage/pet/tpl-fsaverage_hemi-L_atlas-NRUSV2A2024_desc-BmaxSD_pet.func.gii (Rules that matched with issues: rules.files.deriv.imaging.pet_pet_volumetric, rules.files.deriv.imaging.pet_pet_volumetric_atlas, rules.files.raw.pet.pet)
 /tpl-fsaverage/pet/tpl-fsaverage_hemi-R_atlas-NRUSV2A2024_desc-BmaxCOV_pet.func.gii (Rules that matched with issues: rules.files.deriv.imaging.pet_pet_volumetric, rules.files.deriv.imaging.pet_pet_volumetric_atlas, rules.files.raw.pet.pet)
 /tpl-fsaverage/pet/tpl-fsaverage_hemi-R_atlas-NRUSV2A2024_desc-BmaxMean_pet.func.gii (Rules that matched with issues: rules.files.deriv.imaging.pet_pet_volumetric, rules.files.deriv.imaging.pet_pet_volumetric_atlas, rules.files.raw.pet.pet)
 /tpl-fsaverage/pet/tpl-fsaverage_hemi-R_atlas-NRUSV2A2024_desc-BmaxSD_pet.func.gii (Rules that matched with issues: rules.files.deriv.imaging.pet_pet_volumetric, rules.files.deriv.imaging.pet_pet_volumetric_atlas, rules.files.raw.pet.pet)

[bendhouseart]

Oh boo, it's passing validation for the examples here -> bids-standard/bids-examples#552 and bids-standard/bids-examples#557 but my gifti nonsense seems to have broken CI.

[melanieganz]

@bendhouseart , thank you, it works now! And I found my issue when comparing to your examples - I had not zipped the .gii files and the func.gii that @effigies had mentioned above had been added to pet as pet.func.gii.gz and was now fixed to pet.gii.gz.

[melanieganz]

Finally, there is a warning left right now regarding the subject folders:
There are no subject directories (labeled "sub-*") in the root of this BIDS dataset. /dataset_description.json
Do we need this? It doesn't hurt, but it might confuse people.

[effigies]

A couple broad comments. Do you want me to take a shot at this?

[bendhouseart]

A couple broad comments. Do you want me to take a shot at this?

@melanieganz would like to be able to push/convert some atlases with confidence by the end of the week. If you can make one or two tweaks that will merge this have at it. I'll be addressing your comments from yesterday shortly and will let you know how I land, but by no means should you wait for me if you're able to close this out.

[effigies]

Looks like you disabled maintainer pushes to this branch, so I pushed to master...effigies:bids-specification:pet-derivs-basic

[bendhouseart]

Looks like you disabled maintainer pushes to this branch, so I pushed to master...effigies:bids-specification:pet-derivs-basic

Not on purpose, changes merged into this branch.

I think my subject warning "fix" is causing the CI to fail, but otherwise this works with the current examples at:

https://github.com/bids-standard/bids-examples/pull/552/changes

Thanks Chris

[effigies]

My main concerns at this point are:

1. This introduces .func.gii without any text explaining to users how/when to use it, except in the glossary.
2. We should get some agreement from the GIFTI world that we are not producing an incorrect description of .func.gii. I don't currently have a list of people, but HCP/Civet, FreeSurfer, AFNI reps would be good to hear from.

I think that can be done outside this PR, but it should be done before we make a release with this text. I think it's descriptive of current use, but if it's subtly wrong, we risk making things confusing for users and annoying for developers.

[bendhouseart]

I think that can be done outside this PR, but it should be done before we make a release with this text. I think it's descriptive of current use, but if it's subtly wrong, we risk making things confusing for users and annoying for developers.

Agreed, see issue #2449.

Gifti does provide us with a path out of some of this confusion by not requiring an extra extension:

9.0 File Name Extensions
Initially, the extension “.gii” was suggested for all GIFTI files. However, it
is desirable to have a unique extension for each intent type.

But as you said, I think that's a better discussion outside of this PR. We have many tools at our disposal to make gifti use precise for users and developers alike.