Skip to content

Adding calibration dataformat constants #3053

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
juanpala1997 wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Adding calibration dataformat constants #3053

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c8a9e93
Adding calibration dataformat constants
Jul 2, 2026
09ecc05
Removing DL1_MONITORING_CLEAR_NIGHT
Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions src/ctapipe/io/hdf5dataformat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
"DL2_TEL_GEOMETRY_GROUP",
"DL2_TEL_ENERGY_GROUP",
"DL2_TEL_PARTICLETYPE_GROUP",
"DL2_TEL_IMPACT_GROUP",
Comment thread
mexanick marked this conversation as resolved.
"DL0_TEL_POINTING_GROUP",
"DL1_SUBARRAY_POINTING_GROUP",
"DL1_TEL_POINTING_GROUP",
Expand All @@ -61,6 +62,7 @@
"DL1_SKY_PEDESTAL_IMAGE_GROUP",
"DL1_FLATFIELD_IMAGE_GROUP",
"DL1_FLATFIELD_PEAK_TIME_GROUP",
"DL1_MONITORING_ATMOSPHERE",

@mexanick mexanick Jul 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the plan of interfacing of this with existing atmospheric module?

@Voutsi Voutsi Jul 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand your comment. Do you mean the connection with ATMOSPHERE_DENSITY_PROFILE_TABLE?
Those are very different things. The point of adding the DL1_MONITORING_ATMOSPHERE group is to write down information monitoring the quality of the atmosphere. I think this is information is useful in order to monitor long term trends.

@mexanick mexanick Jul 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got an impression you were planning also to store the contemporary VAOD profiles, and I would naively assume that have to be eventually combined with the density profiles in order to compute the total attenuation of the Cherenkov light.

@Voutsi Voutsi Jul 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Let me clarify. I do plan to store the VAOD. However the VAOD is not a profile, it is a single value expressing the optical depth at zenith from the ground to the top of the atmosphere.

I would naively assume that have to be eventually combined with the density profiles in order to compute the total attenuation of the Cherenkov light.

Indeed one has to combine the Rayleigh scattering profile (not density) with molecular absorption and aerosol profiles. But this is outside of the scope of this PR.

@kosack kosack Jul 22, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the atmosphere density profile is Service data, and this is Monitoring, so it think it's clear that they are different.

@Voutsi Voutsi Jul 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @kosack , I am wondering if the atmosphere density profile should be service data. CalibPipe will produce density profiles to be used for tailored simulations. That should be then monitoring data, right?

@kosack kosack Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Monitoring data is something that changes in time during an observation, i.e. it is a table with a TIMESTAMP column, and values that are interpolated to the event time.

An atmosphere profile is at least static for a given day. or obs_id, I guess, right? You would not expect to have to interpolate it to an event time. If so, it should still be Service data. But if you plan to write out a time-series of these for a given night or something like that, then it could be Monitoring. What is the plan?

@Voutsi Voutsi Jul 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @kosack , an atmospheric density model will be static for a day or OB (I am not sure if that's somewhere documented but I don't think it makes sense to produce density models in time intervals shorter than an OB, the density changes slowly).

On the other hand, we might produce more than one extinction profile within one OB, when estimating good time intervals.

You would not expect to have to interpolate it to an event time. If so, it should still be Service data.

That makes me think that the inter - cross calibration coefficients should go to service data as well...

"DL2_SUBARRAY_MONITORING_GROUP",
"DL2_SUBARRAY_INTER_CALIBRATION_GROUP",
"DL2_SUBARRAY_CROSS_CALIBRATION_GROUP",
Expand Down Expand Up @@ -102,6 +104,7 @@
DL2_TEL_GEOMETRY_GROUP = "/dl2/event/telescope/geometry"
DL2_TEL_ENERGY_GROUP = "/dl2/event/telescope/energy"
DL2_TEL_PARTICLETYPE_GROUP = "/dl2/event/telescope/particle_type"
DL2_TEL_IMPACT_GROUP = "/dl2/event/telescope/impact"

DL2_GROUP = "/dl2"
DL2_SUBARRAY_GROUP = "/dl2/event/subarray"
Expand Down Expand Up @@ -152,6 +155,7 @@
DL1_PIXEL_HISTOGRAMS_GROUP = (
"/dl1/monitoring/telescope/calibration/camera/pixel_histograms"
)
DL1_MONITORING_ATMOSPHERE = "/dl1/monitoring/atmosphere"
DL2_SUBARRAY_MONITORING_GROUP = "/dl2/monitoring/subarray"
DL2_SUBARRAY_INTER_CALIBRATION_GROUP = "/dl2/monitoring/subarray/inter_calibration"
DL2_SUBARRAY_CROSS_CALIBRATION_GROUP = "/dl2/monitoring/subarray/cross_calibration"
Expand Down
Loading