PSS-2: do StaA:SecureM and SBS:BeamBlockingM carry EPICS timestamps, or does the gateway serve them unstamped?

[xmap]

Question

Reading the post-migration PSS PVs from arcturus on 2026-08-09, three of them disagree on timestamps despite all arriving through the same gateway (s2pvgate.xray.aps.anl.gov:5064) and all returning sensible values:

PV	Value	caget -a timestamp
S02BM-PSS:FES:BeamBlockingM	ON	2026-08-06 14:53:48.074021
S02BM-PSS:StaA:SecureM	ON	<undefined>
S02BM-PSS:SBS:BeamBlockingM	ON	<undefined>

<undefined> is EPICS timestamp zero, i.e. no timestamp was ever posted on the record.

S02BM-PSS:SBS:BeamBlockingM additionally reported a real timestamp on one read and <undefined> on two later reads within six minutes, so its behaviour appears intermittent rather than consistently absent.

Is unstamped the expected behaviour for these particular records, or does it indicate they have not processed since IOC start? And is the intermittency on SBS:BeamBlockingM known?

Why it matters

CORA records the time of every observation as part of the audit trail. Where the source supplies no time, CORA must record that fact rather than quietly substitute its own clock, which would fabricate provenance. Knowing whether "no timestamp" is normal for these PVs decides whether it is worth surfacing to an operator or simply expected.

Note that both affected PVs are safety-relevant: hutch secure state and station shutter state.

Context

Same reads that confirmed PSS-1 (#144). SR-ACIS:2BM:FesPermitM is also unstamped, and separately reports STATE / MAJOR alarm with value OFF, which is presumably the designed alarm for "permit not granted" during a shutdown rather than a fault.

[decarlof]

Follow-up from the beamline with fresh reads on 2026-08-12. This is a PSS / controls-side matter more than a beamline-operator one, but here is what the operator side can see.

Context first. The APS is in a maintenance shutdown that started 2026-08-05 08:00. All user-facing shutter operations are disabled for the duration — no beamline operator can (or has) toggled the FES, SBS, or hutch state since 08-05 08:00.

Fresh reads on 2026-08-12 ~14:47 CDT and a critical follow-up. All three PVs together with the local host time, then SBS repeated back-to-back in the same shell session:

$ date
Wed Aug 12 02:47:38 PM CDT 2026
$ caget -a S02BM-PSS:FES:BeamBlockingM
S02BM-PSS:FES:BeamBlockingM     2026-08-11 12:48:11.512717 ON
$ caget -a S02BM-PSS:StaA:SecureM
S02BM-PSS:StaA:SecureM          <undefined> ON
$ caget -a S02BM-PSS:SBS:BeamBlockingM
S02BM-PSS:SBS:BeamBlockingM     2026-08-12 14:54:24.512762 ON
$ caget -a S02BM-PSS:SBS:BeamBlockingM
S02BM-PSS:SBS:BeamBlockingM     <undefined> ON
$ caget -a S02BM-PSS:SBS:BeamBlockingM
S02BM-PSS:SBS:BeamBlockingM     <undefined> ON
$ caget -a S02BM-PSS:FES:BeamBlockingM
S02BM-PSS:FES:BeamBlockingM     2026-08-11 12:48:11.512717 ON
$ caget -a S02BM-PSS:StaA:SecureM
S02BM-PSS:StaA:SecureM          <undefined> ON
$ caget -a S02BM-PSS:SBS:BeamBlockingM
S02BM-PSS:SBS:BeamBlockingM     <undefined> ON

Three separate behaviours, all on the same gateway:

• FES — stamp 2026-08-11 12:48:11 (26 hours old). Consistent across repeated reads. In the ~3 days between your 2026-08-09 sweep and mine, the FES stamp advanced from 2026-08-06 14:53:48 to 2026-08-11 12:48:11, so one admin event landed in that window.
• StaA:SecureM — <undefined> on every read. Still <undefined> after 7 full days of maintenance activity. Consistent across repeated reads. Nothing has processed this record; the admin activity that touches the FES and SBS chains does not touch this one.
• SBS — flip-flops between a real timestamp and <undefined> on back-to-back reads seconds apart. First caget returned 2026-08-12 14:54:24; the next two immediate caget calls in the same shell returned <undefined>; a later read still returned <undefined>. That is your intermittency, now reproducible on demand.

Diagnosis of the SBS intermittency. Back-to-back caget calls with no external event in between cannot see two different processed-states of a single underlying record. The only way this makes sense is that S02BM-PSS:SBS:BeamBlockingM is being served by two upstream PSS IOCs simultaneously, with different processed states — one has processed the record since its last restart (real TS 2026-08-12 14:54:24); the
other has not yet processed it (<undefined> at INIT). CA client search-responses win by whichever source replies first, so each caget invocation independently races and lands on one or the other.

This is a common pattern on APS safety-network PVs (redundant PSS IOCs with hot-standby chains). The behaviour is not a gateway cache anomaly and not a beamline-side issue — it is a real "two sources, inconsistent state" condition on the PSS safety network that only surfaces once you look at TIME, because VAL happens to agree on both sides (ON).

The FES and StaA:SecureM records don't show this flip-flop, which is consistent with either "only one IOC serves those" or "both IOCs happen to be in the same processed-state for those records" (both processed for FES, both unprocessed for StaA). The SBS case sits in the middle — split-state — and that is what makes it flip-flop.

Interpretation for the maintenance-timestamp question. The FES timestamps of 2026-08-06 14:53:48 and 2026-08-11 12:48:11 are both mid-shutdown (30 h and 6 days after the 2026-08-05 08:00 maintenance start) and cannot correspond to beamline-side shutter operations — nothing has operated the shutter since 08-05 08:00. Most likely cause: PSS-admin activity during maintenance exercising the FES and SBS chains (inte
rlock tests, PSS IOC restarts, chain re-arms). Each such event causes the bi/mbbi record to process and stamps a new time, even though VAL stays ON.

Implications for the diagnostic questions:

• Is unstamped expected for these records? Yes, given the pattern above. SCAN=Passive bi/mbbi records that have not had an INIT-time processing event and haven't been touched by admin activity since IOC start will show TIME=<undefined> indefinitely. That is normal EPICS behaviour, not a fault. There is nothing on the beamline side that would fix it.
• Is the SBS intermittency known? Not on the beamline side, but the back-to-back-caget sequence above pins down the mechanism: two upstream sources are serving the same PV with inconsistent processed-states, and each CA search-response race lands on whichever one replies first. This is a controls / PSS-network condition (redundant IOCs, split-state), not a gateway cache anomaly. Worth an internal APS PSS-admin ticket
  to establish (a) whether the two chains are supposed to be in agreement, (b) which is the "canonical" one, and (c) whether the same split-state condition applies silently to other records where VAL also happens to agree.

Clock skew ruled out. If the PSS IOC clock were skewed relative to arcturus, every stamped record would be shifted by the same offset. The fact that FES stamps track real (admin-side) events at plausible mid-shutdown times, while other records on the same gateway stay unstamped, means the underlying source-time is honest. arcturus clock (date) and PSS timestamps are consistent modulo the "when did the record last pro
cess" semantic.

Recommendation for CORA:

Keep your existing rule — where the source supplies no time, record that fact rather than substitute CORA's own clock. Do NOT interpret a "recent" PSS timestamp as evidence of a recent operator action; during maintenance in particular, TIME fields on PSS records track PSS-admin activity, not shutter operations. If CORA needs to distinguish "operator-caused" from "admin-caused" state changes, that distinction has to come from
a different source (the operator's own logbook, tomoscan run records, etc.) — the raw EPICS record timestamp cannot resolve it.

Recommendation for the PSS / controls side (not this ticket's scope): the SBS <undefined> intermittency is worth an internal APS ticket to whoever maintains the s2pvgate.xray.aps.anl.gov:5064 gateway or the underlying PSS IOC, since it is the only actively-anomalous behaviour in the set.