Adding calibration dataformat constants

[juanpala1997]

This PR closes #3052

It adds three new H5DF data format constants to ctapipe.io.hdf5dataformat

[Copilot]

Pull request overview

Adds missing HDF5 data format constants needed by DPPS CalibPipe (per #3052) to ctapipe.io.hdf5dataformat, extending the set of standardized group paths used across ctapipe’s HDF5 I/O.

Changes:

• Added DL2_TEL_IMPACT_GROUP for "/dl2/event/telescope/impact".
• Added DL1_MONITORING_ATMOSPHERE and DL1_MONITORING_CLEAR_NIGHT for atmospheric monitoring paths.
• Exported the new constants via __all__.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ctao-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[mexanick]

What kind of data would be stored there? Is it format unique for clear nights?

[Voutsi]

No, not unique. I agree that the naming (clear night) should be changed. The data to be stored there will be the assessment of the night. In the table structure, a column will be the night assessment, i.e. clear, night, Calima, single Cirrus layer etc.

[mexanick]

could you provide a provisional schema for the table?

[Voutsi]

Table schema:
name, type, units
night_assessment String None
VAOD float64 None
fractional_rms_variability_amplitude float64 None

[mexanick]

What's the plan of interfacing of this with existing atmospheric module?

[Voutsi]

I don't understand your comment. Do you mean the connection with ATMOSPHERE_DENSITY_PROFILE_TABLE?
Those are very different things. The point of adding the DL1_MONITORING_ATMOSPHERE group is to write down information monitoring the quality of the atmosphere. I think this is information is useful in order to monitor long term trends.

[mexanick]

I got an impression you were planning also to store the contemporary VAOD profiles, and I would naively assume that have to be eventually combined with the density profiles in order to compute the total attenuation of the Cherenkov light.

[Voutsi]

I see. Let me clarify. I do plan to store the VAOD. However the VAOD is not a profile, it is a single value expressing the optical depth at zenith from the ground to the top of the atmosphere.

I would naively assume that have to be eventually combined with the density profiles in order to compute the total attenuation of the Cherenkov light.

Indeed one has to combine the Rayleigh scattering profile (not density) with molecular absorption and aerosol profiles. But this is outside of the scope of this PR.

[kosack]

the atmosphere density profile is Service data, and this is Monitoring, so it think it's clear that they are different.

[Voutsi]

Hi @kosack , I am wondering if the atmosphere density profile should be service data. CalibPipe will produce density profiles to be used for tailored simulations. That should be then monitoring data, right?

[kosack]

Monitoring data is something that changes in time during an observation, i.e. it is a table with a TIMESTAMP column, and values that are interpolated to the event time.

An atmosphere profile is at least static for a given day. or obs_id, I guess, right? You would not expect to have to interpolate it to an event time. If so, it should still be Service data. But if you plan to write out a time-series of these for a given night or something like that, then it could be Monitoring. What is the plan?

[Voutsi]

Hi @kosack , an atmospheric density model will be static for a day or OB (I am not sure if that's somewhere documented but I don't think it makes sense to produce density models in time intervals shorter than an OB, the density changes slowly).

On the other hand, we might produce more than one extinction profile within one OB, when estimating good time intervals.

You would not expect to have to interpolate it to an event time. If so, it should still be Service data.

That makes me think that the inter - cross calibration coefficients should go to service data as well...

[maxnoe]

I don't understand why a group would be named clear_night.

I would expect an assessment of the atmosphere to be stored in some metrics inside of a dataset, not in a group name.

[Voutsi]

I don't understand why a group would be named clear_night.

I would expect an assessment of the atmosphere to be stored in some metrics inside of a dataset, not in a group name.

I see your point. As I wrote above, the naming I chose for the group is unfortunate. I would suggest to remove then the DL1_MONITORING_CLEAR_NIGHT group and write the assessment of the night inside a table under DL1_MONITORING_ATMOSPHERE

[mexanick]

In case a group (like DL1_MONITORING_ATMOSPHERE) will have multiple tables that aren't separated by the telescopes (or other array element ID), the table paths should be also prescribed in separate constants, like e.g.

ATMOSPHERE_DENSITY_PROFILE_TABLE = "/simulation/service/atmosphere_density_profile"