PRD run: onboarding, unified Event, maintenance drift, mapper engine + logbook profiles - #51
Open
jeandavidt wants to merge 46 commits into
Open
PRD run: onboarding, unified Event, maintenance drift, mapper engine + logbook profiles#51jeandavidt wants to merge 46 commits into
jeandavidt wants to merge 46 commits into
Conversation
The Data Explorer force-refetched and reset its time range on actions that didn't require it, causing slow plots and the date inputs snapping back to defaults: - Adding a stream called _invalidate_data_cache(), wiping the range-keyed cache and forcing a full reload of every already-plotted stream. Removed — the cache is keyed by (stream, start, end), so a new stream loads lazily while existing streams stay cached (incremental, not full reload). - The date-input on_change callback called st.rerun() inside the callback, which races the widget commit and snapped the date back to its default. The callback now only clears the id-keyed annotation/event caches; Streamlit reruns automatically after the change and the range-keyed timeseries cache refreshes on its own. Adds a red→green AppTest regression: adding a second stream must not re-fetch an already-plotted stream.
…native zoom + brush)
Swaps the interactive scalar view on the Data Explorer from Plotly to ECharts
(streamlit-echarts), the slowest part of the old UX. Plotly's datetime x-axis
rendering and full-rerun-per-selection are gone; ECharts canvas gives native
dataZoom (slider + scroll/pinch) and a toolbox brush (rect / polygon / lineX).
The annotation / quality-flag / equipment-event workflow is preserved: the brush
round-trip keeps the old identity contract. A JS handler returns the brushed
(seriesIndex, dataIndex) list and resolve_brush_selection maps it back to the
same {x, y, obs_id, id} the buttons consumed from Plotly customdata.
- New pure module app/components/explore_echarts.py:
build_scalar_echarts_option (option + series_index_map + overlay_rows) and
resolve_brush_selection — all decision-bearing logic, fully unit-tested.
- _render_scalar_view wires st_echarts + the resolver; quality-code colours,
annotation/event markArea+markLine overlays, and the summary table are kept.
- Campaign Story keeps the Plotly _build_scalar_figure (read-only, unchanged).
- Pin streamlit-echarts==0.4.0: 0.7.x moved to st.components.v2 and fails to
import on Streamlit 1.55.
- Tests: new unit suite for the builder/resolver; the Plotly-coupled Explore
AppTests updated to the brush-payload shape (component value is injectable via
session_state["scalar_chart"]).
Browser verification of the live brush event still pending (AppTest can't run
component JS).
Adds the ability to spread scalar streams across several plots instead of overlaying everything on one chart. Streams stay in the canonical explore_active_channels/series lists (so the global export is unaffected); a thin assignment layer (explore_plot_of) groups them per plot. - "➕ Add plot" creates a plot and makes it the target for newly added streams; each plot block has an "Add here" button to retarget. - Each active-stream chip gets a "Plot" selectbox to move it between plots. - _render_scalar_view takes a key `suffix` so it renders once per plot without colliding Streamlit widget keys; one ECharts chart + brush per plot. - Vector / matrix / image stay single-stream pickers (rendered once over all active streams — multi-plot adds nothing there). Tests: new test_explore_multiplot.py (add-plot retargets + new stream lands in target; chip move control reassigns). Existing lab AppTests updated for the per-plot "_p1" key suffix.
…nnotation Refines the Explore scalar UX per follow-up feedback: - Plot-management zone: a badge per plot (click to make it the target for new streams, ✕ to delete). Deleting a plot reassigns its streams to the first remaining plot (streams keep their own ✕); the last plot can't be deleted. Replaces the per-plot inline "Add here" buttons. - Selected points now render in a table (stream, kind, timestamp, value, observation) below the chart. - Annotation is selection-driven: the sensor "Annotate selected points" button targets only the streams that have brushed points (not every channel in the plot), scoped to the selected points' span / pinned observation. Removed the two "view range" annotation controls — annotations no longer apply to the plotted range. Tests: new test_explore_selection.py (selection table + scoped annotation), plot delete in test_explore_multiplot.py; homogeneous-lab-dialog test updated to the selection-driven button.
F4: get_equipment_installations queried the dropped EquipmentInstallation
table, 500ing GET /equipment/{id}/lifecycle. Repoint to EquipmentLocationHistory
(maps 1:1 to InstallationOut; date filters use ValidFrom/ValidTo).
F8: relocate_equipment/rewire_equipment/deploy_das now raise before any write
when the destination equals the current active row, instead of churning an
identical history row.
F12: drop the 5 dead SignalPort* NotImplementedError stubs (keep the live
find_signal_port_type_by_name shim); fix stale SignalPort docstrings in ingest.
The port a Channel is gated through was stored both in ChannelPortHistory
(temporal source of truth) and a denormalised Channel.SignalInterfacePort_ID
column that nothing kept in sync — classic drift risk (consistency audit F3,
decision D1=DROP).
Drop the column. The current port is now resolved from the active
ChannelPortHistory row through a new view, vw_ChannelResolved; every read that
needs the port selects FROM that view instead of the Channel base table, so the
equipment-resolution join predicates are byte-for-byte unchanged. Writes route
through channel_repository.set_channel_active_port — the single writer of the
CPH active-row invariant — which closes the prior active row before opening the
new one. That also fixes the POST /channels/{id}/port-history 500 on a second
post (it collided with UQ_ChannelPortHistory_ActiveRow).
Schema bump 2.0.0 -> 2.1.0 (pre-release, fresh install only; init.sql repointed,
DDL + docs regenerated). The channel CRUD form re-adds signal_interface_port_id
explicitly (form_specs) since it is no longer YAML-derived.
Reads repointed in: channel_repository, sensor_status_repository,
temporal_history_repository, vw_ChannelEquipmentAtTime, vw_ChannelStatus,
vw_DeviceStatus.
NOTE: channel_repository.py also carries the other agent's uncommitted
stream-pedigree WIP (it was dirty in the same file); reconcile separately.
Adds the detection layer for the original question (does the system notice when a DAS moves and its equipment is left behind?): - vw_DeploymentCoherence (F1): lists equipment whose active location Site differs from the Site its connected DAS is currently deployed to. A standing reconciliation report of DAS-move drift. - get_das_move_equipment_conflicts(das_id, new_site_id) (F1): the equipment a pending DAS move would strand, so the move flow can offer to relocate them. - get_active_campaign_deployment(equipment_id) (F13): the still-running campaign whose deployment placed an equipment, so reconfiguring it can warn that the campaign's deployment will be closed. Schema 2.1.0 -> 2.2.0 (pre-release; init.sql repointed, DDL+docs regenerated). Tests: tests/unit/test_deployment_coherence.py (6). Full unit suite 461 passed, 3 skipped (3 failing test_stream_pedigree.py are the other agent's untracked WIP). F2/F9 from the plan were found moot: create_campaign_deployment establishes campaign↔SP membership idempotently, so there is no non-member-SP path to reject. Deferred to morning (needs browser verification): the F1c inline 'relocate these too' UI in the DAS-move/campaign wizard, plus thin GET endpoints exposing these helpers.
GET /lineage/streams/{id}/pedigree returns a stream's organizational/spatial
pedigree (distinct from its processing provenance): time-invariant identity plus
a time-bound deployment timeline — sampling location, process unit, site,
campaign, and responsible person per deployment segment.
- A sensor channel's location/campaign are historical (equipment is rewired and
moved over time), so the timeline carries one segment per EquipmentLocationHistory
it spanned (EWH×ELH temporal join, via vw_ChannelResolved). A lab series returns
one fixed open segment. Optional from/to restrict to segments overlapping the
exported window.
- Also fixes the missing Query/datetime imports in lineage.py that blocked
api.main from importing.
- Regenerate OpenAPI spec for the new route.
(channel_repository.get_stream_pedigree shipped earlier in 86bd74f.)
Data download now produces one zip for all active streams: per stream a CSV (UTC timestamps, value, plus annotation/equipment-event overlay columns and the sampling-location/campaign active at each row's timestamp) paired with a pedigree YAML; image streams embed their files. Two-step Generate → Download UX so the heavy build (per-stream pedigree + image fetches) runs once, not every rerun. Adds a page-level campaign filter: selecting a campaign snaps the time window to its span and scopes the pickable streams (_scope_to_campaign), so both the plot and the export stay within the campaign. - New pure builder app/components/explore_export.py (UTC compare, overlay/segment matching, CSV shaping, image embedding) — unit tested. - api_client.get_stream_pedigree wrapper (forwards the export window). - Remove the old per-type CSV download buttons and now-dead flat-CSV helpers (+ their obsolete test); drop unused io/csv imports. - Queries reconciled to the F3 vw_ChannelResolved refactor.
Introduce CONTEXT.md with canonical definitions: Stream, Provenance (the processing-step lineage DAG) and Pedigree (the time-bound organizational/spatial context — a deployment timeline). Disambiguates the two axes the export and the existing Provenance panel each cover.
…acing Batch 2 part 2 (F1c). Surfaces the detection layer landed in 6648ff3 so a DAS move no longer silently strands its equipment. - GET /das/{id}/move-conflicts?site_id= — equipment a pending DAS move would strand (active wiring to this DAS, active location at a different Site). - GET /equipment/{id}/active-campaign — the still-running campaign whose deployment placed this equipment (reconfigure warning, F13). - Campaign wizard: when a chosen DAS is active at another site, the existing conflict warning now also names the equipment the move would strand and points to Equipment Move to relocate them. - api_client: get_das_move_conflicts / get_active_campaign_deployment. - Tests: contract (both endpoints) + AppTest (wizard names stranded equipment). - openapi.json regenerated. No schema change (endpoints/app only).
The generated CREATE script listed views alphabetically. Once F3 made vw_ChannelEquipmentAtTime select from vw_ChannelResolved, the alphabetical order created the dependent view first — SQL Server resolves view references at CREATE time, so db-init failed with 'Invalid object name dbo.vw_ChannelResolved' (exit 16, whole compose stack blocked). - render.py (the live docs/init generator) and generate_sql.py (legacy, used by test_sql_generator) both now topologically order views: a view that references another is emitted after it. Dependencies are read from each view's SQL text (whole-word match), Kahn's algorithm, alphabetical tiebreak for a stable script. Raises on a cycle rather than emitting broken SQL. - Regenerated v2.2.0_create_mssql.sql (view reorder only; SchemaVersion stamp preserved). db-init now exits 0 against the real MSSQL container. - Regression test: every view is created after the views it references.
Make broken equipment/wiring links observable instead of silently NULL.
- F6: vw_ChannelLocationAtTime now passes through Resolution (equipment leg)
and adds LocationResolution (location leg: resolved / no-location /
no-equipment), so a NULL SamplingPoint from a broken chain is
distinguishable from a genuinely absent location.
- F5: vw_UnlinkedChannels — raw channels (SignalInterface_ID NOT NULL) with
observations but no active EquipmentWiringHistory ('N channels need wiring').
Derived channels are excluded (deliberately unwired, not forgotten).
- F11: vw_InactiveParentReferences — active wiring rows still pointing at a
soft-deleted (IsActive=0) SignalInterface/SignalInterfacePort.
- Endpoints GET /data-health/unlinked-channels and
/data-health/inactive-parent-references (the latter filterable by
interface/port) + api_client wrappers + a 'Data Health' report page.
- version 2.2.0->2.3.0; DDL+seed+docs+openapi regenerated; init.sql repointed.
db-init verified exit 0 against the real MSSQL container.
- Tests: contract (2 endpoints), AppTest (Data Health page warn/clean),
view-contract guards (F6 discriminators), and a version-description-length
guard so an over-long stamp fails in unit tests instead of in docker.
The *History tables guard only one *open* row (filtered unique index); nothing stopped a backdated ValidFrom from inverting the row it closes or landing inside a closed [ValidFrom, ValidTo) interval, after which get_*_at_time can match two rows or none and vw_ChannelEquipmentAtTime silently picks one. - Shared _assert_valid_from_ok guard: before the close-and-open swap, reject a valid_from that is not strictly after the active row's start, or that falls inside any closed interval, for the entity. Wired into rewire_equipment, relocate_equipment, open_location_for_campaign, deploy_das, and backdated register_equipment_at_interface (a None start_time = 'now' skips it). - All writes already funnel through these repos, so this is the right altitude (no trigger needed). - Test: backdated overlap rejected before any write/commit; guard SQL shape + params pinned. Updated two existing repo tests for the extra guard SELECT.
…lete (F10, Batch 5) Per decision D2, CampaignEquipment/CampaignSamplingLocation are the source of truth for campaign membership; ELH.Campaign_ID is provenance only. delete_campaign_deployment dropped the CampaignSamplingLocation link only when an *open* ELH row still placed equipment there, so once the last placement was closed the SP link vanished while closed ELH rows kept the campaign tag — junction-derived scope and ELH-derived scope could then disagree. - Remove the CampaignEquipment membership row first, then the active ELH placement (closed rows keep their provenance untouched), then drop the CampaignSamplingLocation link only if no *other* equipment still in the campaign has any placement provenance at that SP — junction-driven and agnostic to ELH open/closed state. - list_campaign_deployments already drives from the junctions; no change. - Test: delete removes both junction rows, the SP-link SQL is junction-based and carries no ValidTo coupling, CE is deleted before the SP-link check, and the no-SP path skips the link.
…, Batch 6) A campaign's sites are now derived from its sampling-location membership (CampaignSamplingLocation -> SamplingPoint.Site) rather than stored as a single FK, so a campaign may span several sites. version 2.3.0 -> 2.4.0.
Regenerated create/seed scripts + reference docs from the dictionary; init.sql repointed to v2.4.0; demo seed Campaign inserts drop Site_ID (membership comes from CampaignSamplingLocation). Verified db-init exits 0 on a fresh install.
- campaign_repository: _campaign_sites() derives sites via CSL->SamplingPoint->Site; list/get expose site_ids/site_names lists; list filter uses junction EXISTS; insert/update/patch + overview watershed no longer touch Campaign.Site_ID. - channel_repository: pedigree site_fallback derived from membership, used only when the campaign is unambiguously single-site. - schemas: CampaignOut gains site_ids/site_names; CampaignIn/Patch drop site_id.
Campaign create payload + form_specs drop site_id (the wizard's chosen site now flows only to sampling-location/DAS creation); campaigns list filter matches against derived site_ids; campaign story shows the joined site_names list.
- test_campaign_multisite (new): derivation returns >1 site, list filter is junction membership not Campaign.Site_ID, insert/update never write Site_ID. - repoint pedigree/contract/wizard tests to the derived site_ids/site_names and the campaign payload no longer carrying site_id.
…ep rerun Selecting an equipment on the Equipment Deployments step triggers a Streamlit rerun on which the step-1 site / sampling-location widgets aren't rendered, so their live session_state keys get dropped — the step then read None and showed 'No sampling locations selected', losing the assignments. Add wizard_helpers.snapshot_get (live state wins, else the step's persistent snapshot) and use it for the cross-step site / SL reads on the deployment and review steps. Unit test covers the dropped-key fallback.
The export CSV emitted the raw QualityCode id in the quality_code column. Map it to the QualityCode name (via list_quality_codes) so rows read 'Accepted' / 'Suspect' etc.; fall back to the raw value for unknown ids. Annotation and equipment-event kinds already export as their names.
Step 2 (Sites & Sampling Locations) now adds one block per site, each with its own sampling-location multiselect; the campaign's locations are the union. _selected_sls() resolves picks per block (SL ids are unique across sites; names may collide), and deployment/review/execute all read through it. Review re-injects earlier snapshots so its display survives the cross-step widget drop. Tests: _selected_sls union/collision (unit) + AppTest add-site, multi-site deployment list, and the equipment-selection-keeps-SLs bug repro.
…ection lists, Y-axis units Addresses the regression where annotation + equipment-event controls became invisible (they were gated behind a brush selection that's easy to miss / not discover) and the Y-axis lost its parameter+unit label after the ECharts swap. - Click-to-select: clicking a marker now selects that point (single-obs pin), returning the same (seriesIndex, dataIndex) shape as the brush so one resolver handles both. Reliable, discoverable path that doesn't depend on the toolbox brush. - Equipment-event button is always visible again (events are equipment + time based, not tied to a point selection); the selected span pre-fills the dialog. - Under-chart selection summary, next to the action buttons, updates live as you click/brush: left column lists the selected observations (stream, time, value, obs id) with the Annotate buttons; right column lists the selected equipment with the equipment-event button — so it's clear what gets annotated. - Y-axis is labelled "parameter (unit)" again: the unit comes from the loaded time-series payload (the picker meta has no unit), rendered as a centered, rotated axis title like the old Plotly one. Tests: Y-axis label + click-handler shape (unit suite); equipment-event always available + selected-equipment list (selection suite).
Record the unified Event refactor (generalize EquipmentEvent → Event with an eight-FK exclusive-arc target spanning both operational hierarchies + Campaign; EventKind; Annotation repoint; maintenance drift as a derived Channel). Add CONTEXT glossary entries: Event, EventKind, Annotation, and the Measurement Range / Variable Range / Control Limit distinction. Grounded in IWA Metadata Collection Ch3 (cause/effect annotation split, calibration-curve history) and Ch5 (sensor QA vocabulary, controlled- reference requirement for trueness). Pre-release: no migration script.
…arc + seed EventKind vocab (#35) - New Event table: 8 nullable FK exclusive arc (Channel, Equipment, SignalInterface, DataAcquisitionSystem, SamplingPoint, ProcessUnit, Site, Campaign) with CHECK CK_Event_ExclusiveArc (exactly-one non-NULL) - New EventKind table: 15-entry seed vocab (Calibration, Cleaning, Repair, PartReplacement, Replacement, SoftwareUpdate, Validation, Verification, VisualInspection, Commissioning, Decommissioning, OutOfService, PowerOutage, ControllerCrash, OperationalChange) - EquipmentEvent + EquipmentEventKind moved to schema_dictionary/deprecated/ - Annotation.EquipmentEvent_ID FK repointed → Event.Event_ID (col rename in S3) - app/components/form_specs.py: load EventKind YAML for equipment_event_kind slug - DDL regenerated as v2.5.0; sql/init.sql updated; version.yaml bumped to 2.5.0 - 7 new red→green unit tests in tests/unit/test_event_schema_s1.py - Pre-release: no migration script (fresh install only) Closes-related: #35
… with exclusive-arc validator (#36) Add Pydantic schemas (EventIn/Out, EventKindIn/Out) with @model_validator enforcing exactly one of eight arc-target FKs; event_repository with full CRUD for both Event and EventKind; FastAPI routers /events + /event-kinds wired into the v1 protected router. 16 unit tests, all green.
…copied to deprecated/ in S1, not deleted); regen DDL
…ID in YAML, schemas, repo, app (#37)
…; form-coverage green (#38) - api_client: add list/create/update/delete for /event-kinds and /events + list_event_kinds_lookup - form_specs: add 'event_kind' slug pointing to EventKind YAML - app/pages/event_kinds.py: new CRUD page using render_crud_page + get_form_fields('event_kind') - app/pages/events.py: new Event list/create/delete page with arc-FK target picker - Home.py: new 'Events' nav section wiring both pages; equipment_event_kinds.py preserved - test_form_coverage.py: add EventKindIn contract for 'event_kind' slug - test_event_app_s4.py: red→green smoke tests for new form spec + page imports
…ng → dumb preview) (#41)
… shared resolver) (#42)
… follow-on (#34) - Add dismiss button that sets onboarding_dismissed in session_state and reruns - Panel skips rendering when onboarding_dismissed is True (reappears next session) - Add icons to all step bullets (🏭 Site, 👤 Person, 📍 SamplingPoint, 📡 DAS, 🔌 SignalInterface, 📊 Channel, 🧪 Laboratory, 🔬 Lab) - Add campaign optional caption at bottom of panel - Add test_panel_dismissed_hides_panel covering the new dismiss behaviour
…ingStep (#39) - Add MaintenanceDriftIn / MaintenanceDriftOut Pydantic schemas - Add maintenance_drift_repository.create_drift_channel: inserts ProcessingStep (method='maintenance_drift', OperationKind=DriftCorrection), ProcessingLineage edge, and derived Channel; no dbo.Value reads or writes - Add POST /channels/derived/maintenance-drift endpoint (201) - Wire router into api/v1/router.py under /channels prefix - Add 9 unit tests (red→green) covering schema validation; 542 passed 0 failed
…t to /ingest/lab (#43) - resolver.py: exact short_name match before fuzzy name match for parameters - mapper.py: _resolve_all() resolves col-level (param+unit) and row-level (sp, datetime, replicate, values) - Preview ingest table with per-row resolved/skipped status - Submit button → _do_submit() builds sample + measurements payloads, reports per-row ok/error - Unresolved rows are never silently submitted - tests/unit/test_lab_profile_s3.py: 7 red→green tests covering resolution + submit flow
- New page app/pages/maintenance_control_chart.py: drift channel picker, configurable UCL/LCL number inputs, ECharts line+scatter chart with out-of-limit (red triangle) and quality-flagged (orange diamond) markers, source channel overlay series, stats summary row, OOL table - Wired into Home.py navigation under Operations section - 8 unit tests in tests/unit/test_maintenance_chart_s2.py covering dataframe build, chart option structure, limit/flag separation, source series inclusion, and nav wiring (all red→green)
Thin profile reusing the shared mapper engine: tag timestamp/tag/parameter/ unit/value columns, resolve parameter+unit to DB entities, group resolved rows by (tag, parameter, unit) channel, and submit tagged /ingest/sensor payloads. Unresolved rows are surfaced, never silently submitted. A required DAS-name input names the source system (no empty-named DAS).
…e→submit pipeline (#46)
…st-unit target → create) (#47)
…er-confirmed overrides + bulk-apply (#48)
…ing maintenance Events (#49)
…GET /events/{id}/maintenance-drift + before/after UI) (#50)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bundles the full PRD milestone run (branch
prd-get-started, cut offstage). One focused commit per issue for bisectability; unit suite green throughout (592 passed / 4 skipped).Milestones (all issues closed)
Event(ADR 0006),Annotation.Event_IDrename, Event/EventKind CRUD.GET /events/{id}/maintenance-drift+ before/after UI).This session's slices (PRD-3 S6 → PRD-4)
resolve_targetpicks smallest logical unitderive_before_after+ "drift since last cleaning" panelGuardrails
dbo.Valuewrites (drift read-back reads viavalue_repositoryonly).Test plan
uv run pytest tests/unit/→ 592 passed / 4 skipped.