Basic averaging

.github/workflows/ci.yml

@@ -54,6 +54,20 @@ jobs:
       - name: Run end to end test with discrete colour axis
         run: uv run shadems -x TIME -y uv -c FLAG HLTau_B6cont.calavg.tav300s
 
+      - name: Run end to end test with averaging
+        run: uv run shadems --average CHAN:4 --average TIME:60 HLTau_B6cont.calavg.tav300s
+
+      # this MS has 500 MHz channels, so CHAN:1GHz bins two of them together
+      - name: Run end to end test with averaging by quantity
+        run: uv run shadems --average CHAN:1GHz --average TIME:60s HLTau_B6cont.calavg.tav300s
+
+      - name: Run end to end test with averaging and a discrete colour axis
+        run: uv run shadems -x TIME -y uv -c FLAG --average CHAN:2 --average TIME:60 HLTau_B6cont.calavg.tav300s
+
+      # averaging to a single channel leaves the FREQ axis with a zero-width range
+      - name: Run end to end test with averaging down to one channel
+        run: uv run shadems -x FREQ -y DATA:amp --average CHAN:all HLTau_B6cont.calavg.tav300s
+
 
   # deploy:
   #   needs: [test]

README.md

@@ -132,6 +132,31 @@ $ shadems --xaxis CORRECTED_DATA:real,uv --yaxis CORRECTED_DATA:imag,CORRECTED_D
 
 * You can also iterate over SPWs, scans, correlations and (coming soon) antennas.
 
+### Averaging
+
+* The data can be time- and/or channel-averaged before plotting, which cuts noise and the
+number of points to render. Use `--average AXIS:BIN`, where `BIN` is the bin size. A bare
+number counts timeslots (`TIME`) or channels (`CHAN`), while a quantity with units is taken
+as such (`60s`, `2min`, `8MHz`). Use `AXIS:all` to collapse the whole axis. The flag is
+repeatable. Averaging is weighted (using `WEIGHT_SPECTRUM`/`WEIGHT` if present) and
+flag-aware -- flagged samples do not contribute to a bin that has unflagged data in it, and a
+bin comes out flagged only if everything in it was flagged. With `--noflags`, flags are ignored
+here too, and every sample counts towards its bin:
+
+```
+$ shadems --xaxis CHAN --yaxis DATA:amp --average CHAN:4 <msname>          # 4 channels per bin
+$ shadems --xaxis CHAN --yaxis DATA:amp --average CHAN:8MHz <msname>       # 8 MHz-wide bins
+$ shadems --xaxis TIME --yaxis DATA:amp --average TIME:60 <msname>         # 60 timeslots per bin
+$ shadems --xaxis TIME --yaxis DATA:amp --average TIME:60s <msname>        # 60-second time bins
+$ shadems --xaxis FREQ --yaxis DATA:amp --average TIME:all <msname>        # collapse all time
+$ shadems --xaxis TIME --yaxis DATA:amp --average TIME:2min --average CHAN:4 <msname>
+```
+
+* Supported axes are `TIME` (timeslots or a time quantity) and `CHAN` (channels or a
+bandwidth). A bin size spanning all the available data falls back to `all` (with a warning).
+Channel selection (`--chan`) is applied *before* averaging. Time averaging is done per scan
+(bins never span scan boundaries).
+
 ### Plotting residuals
 
 * If you want to see how well your model fits your data then you can subtract the `MODEL_DATA` column from the `CORRECTED_DATA` column prior to plotting. For example, to show this residual product on a uv-distance plot:
@@ -211,8 +236,10 @@ Plot types and data sources:
                         explicitly include a column. For multiple plots, this
                         can be given multiple times, or as a comma-separated
                         list. Two-column arithmetic is recognized.
-  --noflags             Enable to ignore flags. Default is to omit flagged
-                        data.
+  --noflags             Enable to ignore flags entirely: flagged data is not
+                        omitted, and takes part in any --average as if it were
+                        unflagged. Default is to omit flagged data.
+                        Incompatible with plotting a flag column.
   --noconj              Do not show conjugate points in u,v plots (default =
                         plot conjugates).
 
@@ -263,6 +290,16 @@ Data subset selection:
   --chan CHAN           Channel slice, as [start]:[stop][:step], default is to
                         plot all channels
 
+Data averaging:
+  --average AXIS:BIN    Average the data along an axis before plotting, as
+                        'AXIS:BIN' where BIN is the bin size. A bare number
+                        counts timeslots (TIME) or channels (CHAN), while a
+                        quantity with units is taken as such, e.g. '--average
+                        TIME:60s --average CHAN:8MHz'. 'all' collapses the
+                        whole axis, as does a bin size spanning all the data.
+                        Repeatable, e.g. '--average TIME:60 --average CHAN:4'.
+                        Supported axes: TIME, CHAN.
+
 Rendering settings:
   -X XCANVAS, --xcanvas XCANVAS
                         Canvas x-size in pixels (default = 1280)

pyproject.toml

@@ -21,6 +21,7 @@ classifiers = [
 dependencies = [
     "bokeh>=3.8.2,<4",
     "cmasher>=1.6.3,<2",
+    "codex-africanus[dask]>=0.4.4,<0.5",
     "dask>=2024.1.0,<2025",
     "dask-ms[xarray]>=0.2.20,<0.3; python_version < '3.12'",
     "dask-ms[xarray]>=0.2.23,<0.3; python_version >= '3.12'",

shade_ms/__main__.py

@@ -1,5 +1,3 @@
-# -*- coding: future_fstrings -*-
-
 import argparse
 import itertools
 import numpy
@@ -69,7 +67,9 @@ def cli():
              Two-column arithmetic is recognized.""")
 
     group_opts.add_argument("--noflags", action="store_true",
-        help="Enable to ignore flags. Default is to omit flagged data.")
+        help="""Enable to ignore flags entirely: flagged data is not omitted, and takes part in
+             any --average as if it were unflagged. Default is to omit flagged data.
+             Incompatible with plotting a flag column.""")
     group_opts.add_argument("--noconj", action="store_true",
         help="Do not show conjugate points in u,v plots (default = plot conjugates).")
 
@@ -151,6 +151,14 @@ def cli():
     group_opts.add_argument("--chan",
         help="""Channel slice, as [start]:[stop][:step], default is to plot all channels""")
 
+    group_opts = parser.add_argument_group("Data averaging")
+    group_opts.add_argument("--average", action="append", metavar="AXIS:BIN",
+        help="""Average the data along an axis before plotting, as 'AXIS:BIN' where BIN is the
+             bin size. A bare number counts timeslots (TIME) or channels (CHAN), while a
+             quantity with units is taken as such, e.g. '--average TIME:60s --average CHAN:8MHz'.
+             'all' collapses the whole axis, as does a bin size spanning all the data.
+             Repeatable, e.g. '--average TIME:60 --average CHAN:4'. Supported axes: TIME, CHAN.""")
+
     group_opts = parser.add_argument_group("Rendering settings")
     group_opts.add_argument("-X", "--xcanvas", default=1280, type=int,
         help="Canvas x-size in pixels (default = %(default)s)")