Add `unnormalized_reflectance` to VIIRS SDR (`viirs_sdr`) reader

[djhoese]

There are currently two level 1 readers for the VIIRS instrument. viirs_sdr for the NOAA VIIRS SDR and viirs_l1b for the NASA VIIRS L1b format. The L1Bs follow many other file formats for level 1 data and do not include any normalization of the reflectance data by / cos(SZA). However, the SDR format does include this correction already and DataIDs coming from this reader include modifiers=("sunz_corrected",) to reflect this. This complicates some of the changes we want to make for Satpy 1.0 like #3397. In that PR we'd like to make the default normalization for RGBs be the effective_solar_pathlength_corrected modifier (Li and Shibata) for better looking RGBs and to remove the default correction limits of the sunz_corrected modifier so users could get the scientifically accurate "reflectance" value without the correction limits used for RGB composites.

The primary issue is that if VIIRS composite definitions were changed to request inputs with the pathlength correction then data from the VIIRS SDR reader would never be found by the dependency tree in Satpy. For example, if an RGB composite requests an input with ("effective_solar_pathlength_corrected", "rayleigh_corrected") modifiers, then the dependency tree will first look for a dataset matching those exact modifiers from already loaded datasets and then the reader. The dependency tree would then remove the last modifier in the sequence and search for a dataset matching that set of modifiers, so ("effective_solar_pathlength_corrected",). Failing to find a match in the already loaded data and the reader, the dependency tree then looks for the "base" dataset with no modifiers modifiers=(). In the VIIRS SDR reader, as currently implemented, this search would always fail as there is only the modifiers=("sunz_corrected",) version of this data.

The initial proposal was to overwrite the sunz_corrected modifier for VIIRS and make it equal to the effective pathlength correction. This has the benefit of RGBs still looking nice, but the downside that users requesting DataQuery(name="I01", modifiers=("sunz_corrected",)) for the real reflectance, would not get a / cos(SZA) operation as they would with every other instrument in Satpy after #3397 is merged.

During today's monthly community meeting a new idea was considered where the viirs_sdr would be modified to provide the base no-modifier version of the reflectance data by loading the Radiance data from the file and doing the reflectance calibration itself but not do the / cos(SZA). With this, the dependency tree will find this base dataset when trying to fulfill dependencies for RGB recipes requesting modifiers=("effective_solar_pathlength_corrected","). This has the benefit of SDR data getting the preferred effective solar pathlength correction for RGBs and the sunz_corrected modifier behaving the same way it does for every instrument. In this case, requesting modifiers=("sunz_corrected",) will still load the expected and desired file-based version of the reflectance data for scientific use.

So the remaining questions are:

1. Do VIIRS SDR files have the necessary information to properly calibrate radiances to unnormalized reflectances?
2. If the SDR reader is updated to do this, do the composite YAML files need to be updated beyond the existing expected changes of using the pathlength correction for RGBs?

I'm hoping @adybbroe and/or @simonrp84 or @mraspaud or someone can help me answer question 1. I will have to think about question 2 and I'll update this description or post a comment when I have a better idea. I've asked some questions on slack for how loading unnormalized reflectance and reflectance will behave after #3292 is merged. That behavior will determine the answer for question 2 above as we may need to know what happens when we request something just by name (ex. "I01"). Do we get the least-modified DataID (the unnormalized reflectance) or the highest calibration (sunz corrected reflectance).

[djhoese]

To provide more details on question 2, I think there are 3 cases to consider and I think they behave the same whether you think of it as a user passing the query to Scene.load or as a composite dependency. How do each of these requests behave:

a. "I01" purely by name, no modifiers mentioned.
b. "I01" with modifiers=("sunz_corrected",)
c. "I01" with modifiers=("effective_solar_pathlength_corrected",)

when #3292 is merged and we have these 2 DataIDs in the SDR reader:

• DataID(name="I01", ..., calibration="unnormalized_reflectance", modifiers=())
• DataID(name="I01", ..., calibration="reflectance", modifiers=("sunz_corrected",))

With my basic test below:

Details

In [3]: from satpy.dataset import DataID, DataQuery, DatasetDict

In [4]: from satpy.dataset.dataid import default_id_keys_config

In [5]: did_a = DataID(default_id_keys_config, name="I01", wavelength=(0.6, 0.64, 0.68), resolution=371, calibration="radiance", modifiers=())

In [6]: did_b = DataID(default_id_keys_config, name="I01", wavelength=(0.6, 0.64, 0.68), resolution=371, calibration="reflectance", modifiers=("sunz_corrected",))

In [7]: DataQuery(name="I01").sort_dataids([did_a, did_b])
Out[7]: 
((DataID(name='I01', wavelength=WavelengthRange(min=0.6, central=0.64, max=0.68, unit='µm'), resolution=371, calibration=<1>, modifiers=('sunz_corrected',)),
  DataID(name='I01', wavelength=WavelengthRange(min=0.6, central=0.64, max=0.68, unit='µm'), resolution=371, calibration=<3>, modifiers=())),
 (377, 378))

In [8]: DataQuery(name="I01", modifiers=()).sort_dataids([did_a, did_b])
Out[8]: 
((DataID(name='I01', wavelength=WavelengthRange(min=0.6, central=0.64, max=0.68, unit='µm'), resolution=371, calibration=<3>, modifiers=()),
  DataID(name='I01', wavelength=WavelengthRange(min=0.6, central=0.64, max=0.68, unit='µm'), resolution=371, calibration=<1>, modifiers=('sunz_corrected',))),
 (378, inf))

requesting just by name returns the higher calibration (the reflectance with sunz_corrected modifier). This means that to answer question 2 case a (2a), we'd get the same behavior as it is currently, but this may be undesired since L1b and SDR return different levels of correction for this request.

For 2b, this should behave as expected and you get the from-file data for VIIRS SDRs and the satpy-corrected version from VIIRS L1b (unnormalized loaded from the reader and then satpy's sunz_corrected modifier is used).

For 2c, this should work as the dependency tree will request the base version of DataQuery(name="I01", modifiers=()) which will return the unnormalized reflectance from the reader.

So we either leave the configs as desired in #3397 (pathlength default for RGBs, purely by name otherwise). Or we update all purely by name composite recipes to explicitly request modifiers=() or modifiers=("sunz_corrected",) wherever it makes sense.

[strandgren]

So we either leave the configs as desired in #3397 (pathlength default for RGBs, purely by name otherwise). Or we update all purely by name composite recipes to explicitly request modifiers=() or modifiers=("sunz_corrected",) wherever it makes sense.

As discussed yesterday, I have now changed etc/composites/viirs.yaml in #3397 to use the plain EffectiveSolarPathLengthCorrector/effective_solar_pathlength_corrected modifier for built-in composites, in order to be aligned with the other instruments. Hence, it won't work for VIIRS SDR for now.

But even with the unnormalized_reflectance support for the SDR data I guess it wouldn't work out of the box because of the dependency tree behavior that you describe above? So at least for the VIIRS recipes we'd probably need to define the calibration level for all datasets where we expect unnormalized_reflectance?

Or what if we specify the calibration level in the sunz_corrected and effective_solar_pathlength_corrected modifiers? Is that even supported? And would satpy then be clever enough to understand that this then applies to all recipes using that modifier?

[djhoese]

See the current definition in main for sunz_corrected:

satpy/satpy/etc/composites/visir.yaml

Lines 9 to 12 in 70e48cd

	sunz_corrected:
	modifier: !!python/name:satpy.modifiers.SunZenithCorrector
	optional_prerequisites:
	- solar_zenith_angle

  sunz_corrected:
    modifier: !!python/name:satpy.modifiers.SunZenithCorrector
    optional_prerequisites:
    - solar_zenith_angle

It only lists additional dependencies, but not the actual dataset that will be modified. That comes from the request for the dataset with the modifiers specified (ex. composite recipe prerequisites or DataQuery(name="X", modifiers=(...))).

I don't think we could update the support for modifiers to include a calibration: X in their definition as it may conflict with other modifiers that could be chained along in a single request. The confusing part is our most common modifier usage is where we'd want the calibration to make sense (sunz corrected wanting unnormalized reflectances as input and rayleigh corrected wanting real reflectances as input).

But even with the unnormalized_reflectance support for the SDR data I guess it wouldn't work out of the box because of the dependency tree behavior that you describe above? So at least for the VIIRS recipes we'd probably need to define the calibration level for all datasets where we expect unnormalized_reflectance?

It depends. We may need to double check that the dependency tree doesn't add an implicit modifiers=() to compositor prerequisite definitions. If it doesn't, maybe it should. Otherwise to answer your question, I think it depends if we want users to have the experience they had before (sunz corrected VIIRS SDRs are used) or if we want them to have the "correct" experience where unnormalized reflectances are used as would be the case for other instruments. You could argue for either specifying the calibration or modifiers: [] in those cases, but I think updating the dependency tree logic to implicitly add empty modifiers to composite prerequisite requests would be good.

For example, a composite lists name: I01 as an input. The dependency tree should convert that into DataQuery(name="I01", modifiers=()).