Skip to content

feat(space): maintain sessions.visible_message_count counter [#2336] - #2358

Open
lsm wants to merge 6 commits into
devfrom
space/2336-maintain-sessions-visible-message-count-counter
Open

feat(space): maintain sessions.visible_message_count counter [#2336]#2358
lsm wants to merge 6 commits into
devfrom
space/2336-maintain-sessions-visible-message-count-counter

Conversation

@lsm

@lsm lsm commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Replaces the correlated COUNT(*) over sdk_messages that spaceSessions.bySpace ran for every session on every 150ms-debounced re-evaluation (~92ms warm for dev-neokai) with a maintained sessions.visible_message_count column that the query reads directly.

What changed

  • schema: visible_message_count INTEGER NOT NULL DEFAULT 0 on sessions.
  • migration 171: ALTER TABLE + one-time backfill using the badge visibility predicate. (Renumbered twice — dev shipped 169 for message_subtype_norm (perf(daemon): make sdk_messages subtype filters sargable (#2330) #2346) and 170 for preset-agent backfill (fix(space): backfill missing preset agents into existing Spaces [#846] #2370), both since rebased onto.)
  • SDKMessageRepository: increments on visible inserts (saveSDKMessage / saveUserMessage / saveHyperNeoActionMessage); recomputes affected sessions on send_status transitions, rewind deletes (deleteMessagesAt/After), and pending-message removal. Each mutation + counter update is wrapped in one transaction (FTS search-index work stays outside, best-effort) so an FTS throw can't strand the counter. The visibility predicate (top-level rows, non-deferred user rows, non-hidden subtypes) is centralized in isVisibleBadgeRow.
  • reactivity: threaded reactiveDb into the repo; notifyChange('sessions') fires after the counter transaction commits and before the fallible FTS work, so the live badge re-evaluates on every counter-changing event. Wired through the facade and the space-runtime write path.
  • spaceSessions.bySpace: selects s.visible_message_count instead of the per-session subquery.

Maintenance no-ops gracefully when the column/table is absent, so partial-schema test harnesses are unaffected.

Verification

  • spaceSessions.bySpace no longer runs a per-session COUNT(*) (decoupling test); a visible save / status-flip-into-visibility triggers a live re-eval with the new count, while an invisible save does not (reactivity test).
  • Repo tests cover visible/invisible inserts, status flips, action messages, rewind recompute, and an anti-drift cross-check (counter == fresh COUNT(*)).
  • migration-171 test covers backfill correctness, idempotency, and missing-table guards.
  • bun run check (lint/typecheck/knip/db-schema-parity) green; storage/runtime/handlers tests pass under Vitest.

Notes

  • check:test-quality flags one pre-existing issue in provider-registry.test.ts (unmodified).
  • Rollback reconciliation (stale counters after a rollback to a pre-M171 binary) deferred as a follow-up per review — a blind every-boot backfill is too costly, and it's partially self-healing on the next counter-triggering mutation.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lsm has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

@lsm
lsm force-pushed the space/2336-maintain-sessions-visible-message-count-counter branch from 3afafb3 to 721c5cf Compare August 3, 2026 06:05

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lsm has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

Comment thread packages/daemon/src/storage/repositories/sdk-message-repository.ts Outdated

@lsm lsm left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Review by glm-5.1[1m] (GLM / Zhipu)

Model: glm-5.1[1m] | Client: NeoKai | Provider: GLM (Zhipu)

Recommendation: REQUEST_CHANGES (one P3 nit; everything else is solid).

What I verified

  • The perf fix lands correctly. spaceSessions.bySpace now reads s.visible_message_count directly — the correlated per-session COUNT(*) is gone. The new "messageCount is decoupled from sdk_messages" test proves adding message rows can no longer move the badge; only an explicit update to the maintained column does.
  • Predicate parity is exact across all three sites — INSERT (isVisibleBadgeRow), RECOMPUTE (recomputeVisibleMessageCount), and the migration-170 backfill. I checked every dimension: parent_tool_use_id IS NULL, user rows gated on send_status IN (consumed, failed) with NULL→consumed, subtype not in HIDDEN_SYSTEM_SUBTYPES + thinking_tokens, and non-user rows always visible. NULL/empty-string handling for message_subtype and send_status is identical in all three. messages.bySession matches on every badge-relevant dimension (its two extra filters are transcript-rendering only, intentionally excluded from the badge).
  • Every runtime sdk_messages mutation is covered. All writes funnel through SDKMessageRepository; the 6 structural methods all maintain the counter (3 inserts bump, 2 rewinds + pending-delete + status-flip recompute). Session archive preserves messages (counter stays correct); hard-delete FK-cascades and the session row — and the counter — go together, so no drift. No clear/reset/import/fork bulk-insert path exists.
  • Transactions on the hot paths. saveSDKMessage/saveUserMessage wrap insert + bump in this.db.transaction(...) — atomic. ✓
  • Migration is robust: idempotent (recompute-on-rerun), missing-table guards, correct backfill. Tested.
  • No-op guard (supportsVisibleMessageCount) makes maintenance a safe no-op for partial-schema test harnesses — good design.
  • Tests are comprehensive: visible/invisible inserts, status flips, action messages, rewind recompute, an anti-drift cross-check (counter == fresh COUNT(*)) across a mixed sequence, plus migration backfill/idempotency/guards. Relocating the visibility-predicate coverage from the LQH test to the repo test is the right call.

Verification run locally

check:db-schema-parity ✅ · 4-space-storage shard 1893/1893 ✅ · live-query-handlers test 108/108 ✅ · new repo + migration-170 tests 128/128 ✅.

Findings

  • P3 — saveHyperNeoActionMessage counter bump is outside a transaction (see anchored comment on sdk-message-repository.ts:1786). The other two save paths wrap insert+bump in a transaction; this one doesn't. Trivial one-line fix for consistency. The only thing keeping this from an approve.

Optional observation (not a finding, no change requested)

BADGE_HIDDEN_SUBTYPES (Set, insert path) and EXCLUDED_FROM_PAGINATION_SQL_LIST (SQL, recompute path) both encode HIDDEN_SYSTEM_SUBTYPES + 'thinking_tokens' ~20 lines apart. They're consistent today and the anti-drift test guards against divergence; this also matches the file's existing multi-list style, so I'm not asking for a change — just noting the two representations must stay in sync.

Summary

Clean, well-scoped implementation of exactly what the issue asked for, with thorough tests and correct predicate parity. Only the one P3 transaction-consistency nit on the action-message path — fix that and this is good to go.

lsm added a commit that referenced this pull request Aug 3, 2026
…ction

Match saveSDKMessage / saveUserMessage: wrap the INSERT and the
visible_message_count bump in a single transaction so a failure between the
two autocommit statements can't leave the counter under-counted. Addresses
review P3 on #2358.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lsm has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

@lsm lsm left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Review by glm-5.1[1m] (GLM / Zhipu)

Model: glm-5.1[1m] | Client: NeoKai | Provider: GLM (Zhipu)

Recommendation: APPROVE — zero findings remain.

The P3 from the previous round is addressed correctly in dfa38a0: saveHyperNeoActionMessage now wraps the INSERT + bumpVisibleMessageCount in this.db.transaction(...), with the prepared statement hoisted and upsertMessageSearchRow left outside — exactly matching saveSDKMessage / saveUserMessage. Surgical change, no scope creep.

Re-verified this round:

  • Fix is well-formed and consistent with the other two save paths.
  • Repo suite 121/121 (incl. the action-message increment test) ✅.
  • 4-space-storage shard 1893/1893 ✅.
  • PR is OPEN / MERGEABLE; the single review thread is resolved.

Everything from the prior review stands: the perf fix lands (spaceSessions.bySpace reads the maintained column, correlated per-session COUNT(*) removed), predicate parity is exact across insert/recompute/backfill (incl. NULL handling), every runtime sdk_messages mutation maintains the counter, and tests are comprehensive. Approving.

@lsm

lsm commented Aug 5, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dfa38a031f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/daemon/src/lib/rpc-handlers/live-query-handlers.ts
Comment thread packages/daemon/src/lib/rpc-handlers/live-query-handlers.ts

@lsm lsm left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Review by glm-5.1[1m] (GLM / Zhipu)

Model: glm-5.1[1m] | Client: NeoKai | Provider: GLM (Zhipu)

Recommendation: REQUEST_CHANGESsupersedes my earlier APPROVE. A reactivity regression surfaced (raised by @chatgpt-codex-connector and confirmed on re-investigation) that breaks the live unread badge — the PR's central purpose. The earlier P3 (transaction wrap) is fixed and fine; this is a separate, more serious issue I missed in both prior rounds.

P1 — badge query lost its sdk_messages dependency, and the counter write emits no sessions event

How the badge stayed live before. LiveQueryEngine derives each query's table dependencies from extractTables(sql) and re-evaluates only on a change to one of those tables (live-query.ts:306, 402). The old spaceSessions.bySpace SQL had a correlated FROM sdk_messages subquery, so its deps were ["sdk_messages","sessions","spaces"]. Every saveSDKMessage emits a sdk_messages change (reactive-database.ts METHOD_TABLE_MAP), which re-evaluated the badge — that was the live-update path.

What changed. I ran extractTables on both SQL strings:

  • OLD deps: ["sdk_messages","sessions","spaces"]
  • NEW deps: ["sessions","spaces"]sdk_messages is gone.

So a sdk_messages change no longer re-evaluates spaceSessions.bySpace. And the maintained counter is updated by raw UPDATE sessions SET visible_message_count=… inside SDKMessageRepository.bumpVisibleMessageCount / recomputeVisibleMessageCount. The reactive proxy emits table events from the method name (METHOD_TABLE_MAP), not from the SQL — prepare().run() binds to the raw target and emits nothing. So that sessions write emits no sessions change either.

Net effect. A counter-changing event no longer triggers badge re-evaluation. The badge now refreshes only incidentally when an unrelated reactive updateSession (e.g. a processing_state transition around a turn) happens to fire. That is unreliable and demonstrably misses cases the old code handled:

  • saveHyperNeoActionMessage — action-message inserts with no coincident session-row write.
  • updateMessageStatus flips (deferred→consumed) after a turn ends.
  • the tail of a turn once processing_state has gone idle.

This violates the codebase's own reactivity pattern. Every other repository that writes raw SQL calls reactiveDb.notifyChange('<table>') afterwards (task, skill, provider, mcp-enablement, app-mcp-server, space-task, goal, …). state-projection-service.ts:285/308 even calls notifyChange('sessions') for its raw UPDATE sessions SET last_error. The counter maintenance is the one raw sessions write that skips it.

Suggested fix (preserves the perf win)

Thread reactiveDb into SDKMessageRepository (as task/skill/provider repos already do) and call this.reactiveDb.notifyChange('sessions') after bumpVisibleMessageCount / recomputeVisibleMessageCount. This restores the live re-eval without re-introducing the correlated COUNT(*): the query stays a cheap column read, and re-evaluation frequency returns to what it was (still 150ms-debounced via DEBOUNCE_SPACE_SESSIONS_MS).

A test for this should assert that a saveSDKMessage on a space session causes spaceSessions.bySpace to re-evaluate (e.g. via the LiveQueryEngine change event / a subscriber delta), not just that the column is updated — the repo-level counter tests pass today but don't cover the reactivity gap.

Standing

Everything else from prior rounds holds (predicate parity across insert/recompute/backfill; all mutation paths covered; migration idempotent; the P3 transaction wrap is now in). Only this P1 blocks.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create an environment for this repo.

@lsm

lsm commented Aug 6, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dfa38a031f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/daemon/src/storage/schema/migrations.ts Outdated
Comment thread packages/daemon/src/storage/repositories/sdk-message-repository.ts Outdated

@lsm lsm left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Review by glm-5.1[1m] (GLM / Zhipu)

Model: glm-5.1[1m] | Client: NeoKai | Provider: GLM (Zhipu)

Both @chatgpt-codex-connector P2s are legitimate — I verified the mechanisms. Consolidating my take and how they relate to the in-flight P1 fix.

P2 (endorse, recommend fixing now) — wrap status/delete mutations + counter in one transaction

upsertMessageSearchRow (sdk-message-repository.ts:245) only guards JSON.parse; its FTS DELETE/INSERT writes (313–341) are unguarded. So in the four structural paths the ordering is: mutation commits → FTS loop → recomputeVisibleMessageCount:

  • updateMessageStatus (1324 run → 1325 FTS loop → 1326 recompute)
  • deleteMessagesAfter / deleteMessagesAtAndAfter (DELETE → 1426/1450 FTS loop → 1427/1451 recompute)
  • deletePendingUserMessage (DELETE → 1386 FTS → 1389 recompute)

If an FTS write throws, the counter recompute is skipped and the status/delete is already committed → permanent drift (until an incidental recompute). A crash in that window drifts too, but self-heals; the FTS-throw case does not, on its own. This is the same class of issue as the insert-path P3 already fixed in dfa38a031, just unfixed on these four methods.

Recommend doing this together with the P1 notifyChange work — the coder is already in these methods. Wrap each as this.db.transaction(() => { mutation; recompute; notifyChange('sessions'); }) and keep the FTS loop outside (best-effort, as today). Cheap, consistent, and closes the gap.

P2 (real, but cost tradeoff — conscious decision) — rollback reconciliation

The mechanism checks out: M170 is marked (runs once), the column persists across a rollback to a pre-M170 binary (which writes sdk_messages without maintaining it), and re-upgrade skips M170 → stale counters until an incidental recompute. There's clear precedent for an unmarked startup reconciliation here — reconcileSdkMessageReplacementProjection (migrations.ts:752-756, "intentionally runs outside the one-shot marker") and the artifact_type reconciliation (776-777) exist for exactly this rollback-drift reason.

The caveat is cost: the backfill is UPDATE sessions SET visible_message_count = (SELECT COUNT(*) …) across all sessions — O(sessions × messages) on every boot, even when nothing rolled back. Worth doing, but I'd shape it as a cheap incremental pass or gate it rather than a blind full backfill each startup. Reasonable to land as a follow-up with eyes open; not a blocker alongside the P1.

Summary

  • P1 (reactivity, prior review) — blocks.
  • P2 transaction-wrap — endorse; fold into the in-flight P1 fix since the methods overlap.
  • P2 rollback reconciliation — real, has precedent, but carries a per-boot cost; fine as a deliberate follow-up.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create an environment for this repo.

lsm added a commit that referenced this pull request Aug 6, 2026
…updates

P1 (reactivity): removing the correlated COUNT(*) dropped sdk_messages from
spaceSessions.bySpace's table-deps, and the raw counter UPDATE emitted no
reactive 'sessions' event — so the live badge never re-evaluated on message
saves. Thread reactiveDb into SDKMessageRepository and call
notifyChange('sessions') after counter changes (wired in the facade and
space-runtime write paths; read-only repo instances unaffected). notifyChange
fires after the mutation transaction commits so re-eval sees committed state.

P2 (transaction-wrap): wrap mutation + counter recompute in one transaction for
updateMessageStatus / deleteMessagesAfter / deleteMessagesAtAndAfter /
deletePendingUserMessage (FTS loop stays outside, best-effort), matching the
insert-path fix from dfa38a0 — an FTS throw can no longer leave the counter
stale.

Adds sdk-message-repository-live-query.test.ts asserting a visible save (and a
status flip into visibility) triggers a spaceSessions.bySpace re-eval with the
new count, while an invisible save does not.

Addresses review rounds 2-3 on #2358.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lsm has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1cac717408

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/daemon/src/storage/repositories/sdk-message-repository.ts Outdated
lsm added 4 commits August 5, 2026 22:34
Replace the correlated COUNT(*) over sdk_messages that spaceSessions.bySpace
ran for every session on every 150ms-debounced re-evaluation (~92ms warm for
dev-neokai) with a maintained sessions.visible_message_count column read
directly.

- schema: add visible_message_count INTEGER NOT NULL DEFAULT 0 to sessions
- migration 170: ALTER + one-time backfill using the badge predicate
  (renumbered from 169 because dev shipped 169 for message_subtype_norm in #2346)
- SDKMessageRepository: increment on visible inserts (saveSDKMessage,
  saveUserMessage, saveHyperNeoActionMessage); recompute affected sessions on
  send_status transitions, rewind deletes, and pending-message removal
- spaceSessions.bySpace: select s.visible_message_count instead of the
  per-session subquery

Counter maintenance no-ops gracefully when the column/table is absent (unit
test harnesses), keeping the existing suite green. The visibility predicate
(top-level rows, non-deferred user rows, non-hidden subtypes) is centralized in
isVisibleBadgeRow so the maintained counter cannot drift from the predicate it
replaces; a cross-check test asserts it equals a fresh COUNT(*) across a mixed
sequence of inserts, status flips, and rewinds.
…ction

Match saveSDKMessage / saveUserMessage: wrap the INSERT and the
visible_message_count bump in a single transaction so a failure between the
two autocommit statements can't leave the counter under-counted. Addresses
review P3 on #2358.
…updates

P1 (reactivity): removing the correlated COUNT(*) dropped sdk_messages from
spaceSessions.bySpace's table-deps, and the raw counter UPDATE emitted no
reactive 'sessions' event — so the live badge never re-evaluated on message
saves. Thread reactiveDb into SDKMessageRepository and call
notifyChange('sessions') after counter changes (wired in the facade and
space-runtime write paths; read-only repo instances unaffected). notifyChange
fires after the mutation transaction commits so re-eval sees committed state.

P2 (transaction-wrap): wrap mutation + counter recompute in one transaction for
updateMessageStatus / deleteMessagesAfter / deleteMessagesAtAndAfter /
deletePendingUserMessage (FTS loop stays outside, best-effort), matching the
insert-path fix from dfa38a0 — an FTS throw can no longer leave the counter
stale.

Adds sdk-message-repository-live-query.test.ts asserting a visible save (and a
status flip into visibility) triggers a spaceSessions.bySpace re-eval with the
new count, while an invisible save does not.

Addresses review rounds 2-3 on #2358.
…import

Move notifySessionsChanged() to immediately after the counter transaction
commits and BEFORE the best-effort search-index work
(upsert/deleteMessageSearchRow) across all six message-mutation paths, so an
FTS throw can no longer strand the live badge with a committed-but-unnotified
counter. Addresses the codex P2 on #2358.

Also adopt dev's sqlite-compat DB import in the new reactivity test (bun:sqlite
is being phased out under the Vitest + node:sqlite migration).
@lsm
lsm force-pushed the space/2336-maintain-sessions-visible-message-count-counter branch from 1cac717 to 8fb1403 Compare August 6, 2026 02:45

@lsm lsm left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Review by GLM-5.1 (HyperNeo)

Model: GLM-5.1 | Client: HyperNeo | Provider: HyperNeo

Recommendation: REQUEST_CHANGES — CI is red on head 8fb140339, and the reactivity fix in 441b2676d/8fb140339 introduces a real reactive-transaction scope-poisoning bug.

🔴 P1 (blocking): unscoped notifyChange('sessions') poisons reactive-transaction flush scope → CI red

CI on this head:

  • Daemon Unit Tests (4-space-storage) — ❌ failure
  • All Tests Pass — ❌ failure

This contradicts the last handoff ("239 affected + 1510 space-runtime tests pass under Vitest") — the storage shard wasn't covered by that verification.

The single failing test (scoped-invalidation.test.ts, not changed by this branch — last touched by dev's Vitest migration):

ReactiveDatabase — scope extraction > transaction flush preserves compatible scope
  expect(txEvent.scope).toEqual({ sessionId: 'sess-5' })
  Expected: { sessionId: 'sess-5' }
  Received: undefined

Local repro: cd packages/daemon && bun test tests/unit/4-space-storage/storage/scoped-invalidation.test.ts → 15 pass, 1 fail.

Root cause. SDKMessageRepository.notifySessionsChanged() calls reactiveDb.notifyChange('sessions') with no scope (sdk-message-repository.ts:1176-1177). notifyChange is the manual escape hatch and takes no scope param, so incrementAndEmit('sessions') runs with scope = undefined. Inside a reactive transaction:

  1. addPendingScope('sessions', undefined) — the guard !scope?.sessionId && !scope?.taskId is true → marks the sessions pending scope null (reactive-database.ts:289-293).
  2. On commitTransaction, flushPendingTables sees hasUnscopedTable (any pending table whose scope is null) and emits the entire batch with eventScope = undefined (reactive-database.ts:333-337).

In the failing test, saveSDKMessage('sess-5') runs inside beginTransaction()/commitTransaction(). Its unscoped notifyChange('sessions') pollutes the batch, so the properly-scoped sdk_messages write for sess-5 is emitted with scope: undefined instead of { sessionId: 'sess-5' }.

Introduced by this PR, not pre-existing. origin/dev's sdk-message-repository.ts contains zero notifyChange / notifySessionsChanged / reactiveDb references — they were added in 441b2676d ("restore live badge reactivity"). I verified with git show origin/dev:...sdk-message-repository.ts | grep notifyChangeNONE on origin/dev.

Production impact (beyond the test). During batched/transactional message writes, the unscoped sessions notify defeats LiveQuery scope filtering: every sessions-dependent query re-evaluates on each flush regardless of scope (the scopeFilter is skipped when scope is absent — live-query.ts:417). That's over-broad re-evaluation — the opposite of this PR's perf goal — and can drive cross-space spaceSessions.bySpace re-evaluation during streaming turns.

Fix (backward-compatible)

1. reactive-database.ts — let the escape hatch carry scope:

// ReactiveDatabase interface + impl
notifyChange(table: string, scope?: TableChangeScope): void {
  incrementAndEmit(table, scope);
}

Adding an optional param is non-breaking — the ~40 existing notifyChange(table) callers across other repositories (task/skill/space-task/mcp-enablement/provider/…) are unaffected. (reactive-database.ts isn't in this PR's diff, so it'll need to be added in the same PR.)

2. sdk-message-repository.ts — thread the session id:

private notifySessionsChanged(sessionId: string): void {
  this.reactiveDb?.notifyChange('sessions', { sessionId });
}

{ sessionId } alone is sufficient for spaceSessions.bySpace: its scope filter does a live membership lookup (members.has(scope.sessionId), live-query-handlers.ts:3543) and doesn't require spaceId on the scope. The single-session call sites (saveSDKMessage / saveUserMessage / saveHyperNeoActionMessage / deleteMessagesAfter / deleteMessagesAtAndAfter) just pass their sessionId.

⚠️ updateMessageStatus can touch multiple sessionsaffectedSessions is DISTINCT session_id over messageIds (line 1341). Collect the changed set and notify per session so scopes don't collapse to undefined:

const changed = new Set<string>();
this.db.transaction(() => {
  stmt.run(newStatus, ...messageIds);
  for (const { sid } of affectedSessions) {
    if (this.recomputeVisibleMessageCount(sid)) changed.add(sid);
  }
})();
for (const sid of changed) this.notifySessionsChanged(sid);

After the fix, please run the full 4-space-storage shard (./scripts/test-daemon.sh 4-space-storage), not just the affected files, and confirm CI is green before re-requesting review. The prior rounds' predicate-parity and transaction-wrap work is solid; this is purely the reactivity-notification scope.

Comment thread packages/daemon/src/storage/repositories/sdk-message-repository.ts Outdated
… scope

notifySessionsChanged() called reactiveDb.notifyChange('sessions') with no
scope. Inside a reactive transaction that batches a scoped sdk_messages write,
addPendingScope('sessions', undefined) marks the sessions scope null, and
flushPendingTables then emits the whole batch with undefined scope — poisoning
the properly-scoped sdk_messages write and defeating scope filtering. This made
the "transaction flush preserves compatible scope" test fail (4-space-storage
shard red) and caused over-broad spaceSessions.bySpace re-eval in production.

Fix:
- reactive-database.ts: notifyChange(table, scope?) forwards the scope to
  incrementAndEmit (optional param, non-breaking for the other callers).
- sdk-message-repository.ts: notifySessionsChanged(sessionId) passes
  { sessionId }, compatible with the sdk_messages scope in the same batch.
- updateMessageStatus (multi-session): collect the changed sessions and notify
  per session instead of a single unscoped notify.

Verified the full 4-space-storage shard (1896/1896) under Vitest. Addresses the
P1 on #2358.

@lsm lsm left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Review by GLM-5.1 (HyperNeo)

Model: GLM-5.1 | Client: HyperNeo | Provider: HyperNeo

Recommendation: APPROVE

The P1 reactive-transaction scope-poisoning bug is fixed correctly in 3a119076b, and CI is green. All prior-round findings are addressed.

Verified this round (fresh):

  • Fix correctness. notifyChange(table, scope?) now forwards scope to incrementAndEmit (reactive-database.ts) — backward-compatible: the ~40 other single-arg callers receive scope = undefined, identical to prior behavior. notifySessionsChanged(sessionId) passes { sessionId }. All six call sites updated (saveSDKMessage, saveUserMessage, saveHyperNeoActionMessage, deleteMessagesAfter, deleteMessagesAtAndAfter, updateMessageStatus). updateMessageStatus correctly collects the changed-session set and notifies per session (it can span multiple sessions via the DISTINCT session_id lookup), so multi-session batches don't collapse to undefined.
  • { sessionId } is sufficient. spaceSessions.bySpace's scope filter does a live membership lookup (members.has(scope.sessionId)), so it doesn't need spaceId on the scope — the session's own space's badge feed re-evaluates, other spaces skip. Scope filtering is preserved end-to-end.
  • Interface safety. The one zero-arg mock ({ notifyChange: () => {} } as unknown as ReactiveDatabase) is cast-bypassed and ignores extra args; single-arg notifyChange('tasks') callers are unchanged. No implementer breaks; typecheck is additive.
  • Empirical. Full 4-space-storage shard: 1896/1896 pass under Vitest (incl. the previously-failing "transaction flush preserves compatible scope"), reproduced locally.
  • CI. All Tests Pass ✅ green; zero failing checks on 3a119076b (this aggregator was red last round due to 4-space-storage — now green).
  • Conversations. All 7 review threads resolved (0 unresolved).

Thanks for running the full shard this time rather than just the affected files — that's what caught the regression path. The predicate-parity, transaction-wrap, and now scoped-notification work are all solid.

P0: 0 · P1: 0 · P2: 0 · P3: 0

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3a119076bd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/daemon/src/lib/rpc-handlers/live-query-handlers.ts

@lsm lsm left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Review by GLM-5.1 (HyperNeo)

Model: GLM-5.1 | Client: HyperNeo | Provider: HyperNeo

Recommendation: REQUEST_CHANGES (supersedes my earlier approve/submit — a valid P2 surfaced after I submitted).

To be clear: the core change in 3a119076b is sound — the P1 scope-poisoning fix is correct and backward-compatible, CI All Tests Pass is green, the full 4-space-storage shard is 1896/1896, and all prior threads are resolved. I'm re-opening solely on a new finding (raised by @chatgpt-codex-connector) that I verified.

🟡 P2 (should-fix): scripts/recover-messages.ts bypasses the counter → badge drift after message recovery

I confirmed the bypass by reading the script:

  • Step 4 (recover-messages.ts:156-182): recovers messages into existing sessions via direct INSERT OR IGNORE INTO sdk_messages (...), never going through SDKMessageRepository and never touching sessions.visible_message_count. The existing session's counter is now stale (too low) for any recovered visible message.
  • Step 5 (recover-messages.ts:199-206): creates placeholder sessions with INSERT INTO sessions (...) that omits visible_message_count → it defaults to 0, then inserts messages into them. Those sessions show 0 visible messages despite containing recovered ones.

Why it doesn't self-heal at startup: migration 171 is already marked on such a database, so a restart won't rerun the backfill. The drift persists until a later send_status transition or rewind delete happens to call recomputeVisibleMessageCount for those sessions.

This is a regression introduced by this PR. Under the old correlated COUNT(*), recovery was correct (the badge recomputed live). Switching to a maintained column creates an obligation this script violates — a classic integration gap ("omissions and integration risks"). Severity is P2 (not P1): it's a manual operational tool, the drift is display-only, bounded, and partially self-healing — but the fix is trivial and deterministic, so worth doing now rather than as a silent known-bad.

Fix: after the inserts, recompute visible_message_count for every touched session using the same badge predicate as isVisibleBadgeRow / migration 171 backfill (top-level rows; non-deferred user rows with NULL→consumed; exclude HIDDEN_SYSTEM_SUBTYPES + 'thinking_tokens'). Sketch:

// After Step 4 + Step 5 inserts, for each touched session id:
db.prepare(`
  UPDATE sessions SET visible_message_count = (
    SELECT COUNT(*) FROM sdk_messages
    WHERE session_id = ?
      AND parent_tool_use_id IS NULL
      AND (message_type != 'user' OR COALESCE(send_status,'consumed') IN ('consumed','failed'))
      AND COALESCE(message_subtype,'') NOT IN (${EXCLUDED_FROM_PAGINATION_SQL_LIST})
  ) WHERE id = ?
`).run(sid, sid);

To avoid predicate drift, prefer reusing the shared predicate rather than re-literalizing it here (e.g., expose the backfill SQL or a recomputeVisibleMessageCount(sid) entry point the script can call). If you'd rather track this as a follow-up instead of fixing inline, that's defensible (consistent with the deferred rollback-reconciliation) — but please state it explicitly in the PR description rather than leaving it silent.

Secondary check: grep confirms recover-messages.ts is the only non-repository direct INSERT into sdk_messages; all runtime mutations go through SDKMessageRepository. So this is the one bypass to close.

P0: 0 · P1: 0 · P2: 1 · P3: 0

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create an environment for this repo.

scripts/recover-messages.ts inserted sdk_messages rows directly, bypassing
SDKMessageRepository, so the maintained counter drifted after message recovery
(a regression vs. the old live COUNT(*)). Exposed recomputeVisibleMessageCount
as a public entry point (same shared badge predicate — no re-literalizing) and
call it for every session touched by the script's direct inserts (existing
sessions that received messages + newly created orphan sessions). No-op on a
pre-M171 schema that doesn't carry the column yet (the migration backfill
covers that). Added a repo test that recomputes after a simulated bypass insert.

Addresses the P2 on #2358.

@lsm lsm left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Review by GLM-5.1 (HyperNeo)

Model: GLM-5.1 | Client: HyperNeo | Provider: HyperNeo

Recommendation: APPROVE

The recover-messages P2 is fixed correctly and inline in 179bb2fd9. All findings across rounds are now resolved; the counter-maintenance contract is airtight.

Verified this round (fresh):

  • Fix correctness. recomputeVisibleMessageCount is now public and reuses the exact shared badge predicate — same SQL as isVisibleBadgeRow / the M171 backfill, so no re-literalizing and no predicate drift (exactly what was asked). The script tracks every touched session: step 4 adds on each successful insert into an existing session, step 5 adds each newly-created orphan session, and step 6 recomputes each after all inserts complete (so it sees the recovered rows). new SDKMessageRepository(db) is a valid construction (reactiveDb optional), and recompute needs only this.db + the schema guard — no live-query notify, which is correct for an offline maintenance tool.
  • No-op pre-M171. supportsVisibleMessageCount() guards the recompute, so the migration backfill still owns first-time population. ✓
  • Test. The new repo test simulates a raw bypass INSERT (assistant + user, both visible), asserts the counter stays stale at 0, repairs via the public recompute → counter == 2 == fresh COUNT(*), and confirms a second call is a no-op (returns false). Good coverage of the bypass→repair path.
  • No regression. Making the method public is safe (idempotent; consumed by script + test, so knip-clean); existing private call sites are unchanged; mergeable reconfirmed.
  • Empirical. Full 4-space-storage shard: 1897/1897 pass under Vitest (incl. the new test). CI Daemon Unit Tests (4-space-storage) ✅ success with zero failing completed checks; all review threads resolved (0 unresolved); PR open + mergeable.

Good call fixing it inline rather than deferring, and reusing the shared predicate was exactly the right move. Approving.

P0: 0 · P1: 0 · P2: 0 · P3: 0

@lsm lsm left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Review by GLM-5.1 (HyperNeo)

Model: GLM-5.1 | Client: HyperNeo | Provider: HyperNeo

Follow-up: CI All Tests Pass red on 179bb2fd9 is a pre-existing flake, not a PR regression. Code approval stands.

The aggregator went red only because Daemon Unit Tests (4-space-migrations-a) failed — specifically migration-45_test.ts > "fresh DB has node column names (not step)" with a test timeout (5000ms), not an assertion failure.

Evidence it's a flake unrelated to this PR:

  • Reproduces green locally. The full 4-space-migrations-a shard passes locally: 248/248, 0 fail, 45.3s (CI took 155.6s under coverage — ~3.4× slower). The file alone passes 6/6 in 2.48s.
  • Unrelated migration. It's Migration 45 (rename stepnode in workflow tables). This PR adds Migration 171 (visible_message_count). The PR doesn't touch migration-45_test.ts, migration 45, or the migration runner (git log origin/dev..HEAD -- migration-45_test.ts → empty).
  • Full chain runs fine. migration-45_test.ts:68 calls runMigrations(db) (all migrations, incl. the new 171) then asserts — it passes, so migration 171 neither breaks nor hangs the chain.
  • Flake signature + origin. Timeout (not assertion); the stack originates in tests/bun-test-shim.ts:32, introduced by dev's Vitest/node:sqlite migration fd867582f (#2364) — the same commit that last touched this test file. Not in the flaky-quarantine list.

Recommendation: rerun the failed 4-space-migrations-a job (it'll pass on retry). The durable fix — raising the per-test timeout for full-chain migration tests, or splitting the chain — is a dev-side test-infra matter out of scope for this PR. My code approval (P1 + P2 fixed and verified) stands; I'll submit for approval once CI is green.

@lsm lsm left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Review by glm-5.1 (GLM)

Model: glm-5.1 | Client: HyperNeo | Provider: GLM

Recommendation: APPROVE — CI is now green on 179bb2fd9; all findings resolved.

Update on the CI flake: the earlier All Tests Pass red was a pre-existing flake, now confirmed empirically. I reran the single failed job Daemon Unit Tests (4-space-migrations-a) (the migration-45 timeout, unrelated to this PR) and it passed. The full PR check matrix is now green: All Tests Pass: pass, Daemon Unit Tests (4-space-storage): pass (this PR's own shard — includes the new recomputeVisibleMessageCount repair test and the Migration 171 test), Coverage Quality Gate: pass, and all daemon-online/lint/web jobs pass.

Review cycle summary (both findings fixed and verified):

  • P1 — scope poisoning (3a119076b): notifyChange(table, scope?) now accepts an optional scope; notifySessionsChanged(sessionId) passes { sessionId }; updateMessageStatus collects changed sessions and notifies per-session. Verified: full 4-space-storage shard green, plus the scoped-invalidation tests.
  • P2 — recover-messages bypass (179bb2fd9): recomputeVisibleMessageCount exposed public; the recovery script tracks every session touched by its direct inserts and recomputes via the shared badge predicate (no drift; no-op pre-M171). Verified by a new repo test simulating a bypass insert → repair.

Merge readiness: OPEN, MERGEABLE (CLEAN), 8/8 review conversations resolved (0 unresolved), CI green. The PR's own tests pass in CI; the only prior red was the unrelated migration-45 timeout flake, now cleared on rerun. Ready to merge into dev (squash, no --delete-branch).

Submitting for human approval (space autonomy requires sign-off).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant