perf: coalesce streamed message_update events to cut remote bandwidth (#375) - #393
perf: coalesce streamed message_update events to cut remote bandwidth (#375)#393shani-singh1 wants to merge 1 commit into
Conversation
Every message_update carries the full accumulated message, and the agent emits one per streamed chunk. The events route forwarded each one, so a remote client received the whole message again on every chunk — transfer grew O(n^2) with message size (~150x amplification over the actual content on VPN/metered connections, more with thinking). agegr#375 Buffer message_update events and flush only the latest on an ~80ms timer, so a burst of chunks collapses into one send while streaming stays smooth (~12/s). Any other event (tool/turn/agent_end) flushes the pending update first, so a message_update is never reordered past a later event and the message's final content is always delivered before the next boundary. Server-side only; the client still renders each message_update as before, just fewer of them (which also cuts client re-render cost). Frontend delta rendering would drive this to 1x and is a natural follow-up. Fixes agegr#375
|
Thanks for the contribution and for keeping the proposed change server-side and contained. Coalescing full However, a fixed 80 ms window only reduces the constant factor. It still sends a growing full-message snapshot repeatedly, so total transfer remains O(n²). When update intervals are 80 ms or slower, little or no coalescing occurs. Because of that, this cannot be the final fix for #375. pi-web has now upgraded to Pi 0.84, which defines an official delta-only JSON/RPC contract for
This provides linear wire usage without maintaining repeated full snapshots. I’m therefore closing this PR in favor of the replacement implementation. We will keep #375 open until that implementation lands. Thank you for the clear implementation and tests. |
2 participants
Summary
Fixes #375.
On remote access (VPN / metered / public network), pi-web's SSE stream is hugely bandwidth-heavy: every
message_updateevent carries the full accumulated message, and the agent emits one per streamed chunk. The events route forwarded each one, so the browser received the entire message again on every chunk — transfer grows O(n²) with message size. The reporter measured ~150× amplification over the actual content (~15 KB content → ~2.2 MB transferred; >6 MB withthinking: high).Fix (server-side only)
Coalesce streamed
message_updateevents in the events route:message_update(each already carries the full message, so older ones are redundant) and flush it on a short ~80 ms timer — a burst of chunks collapses into one send, while streaming stays smooth (~12/s).agent_end) flushes the pending update first, so amessage_updateis never reordered past a later event, and the message's final content is always delivered before the next boundary event.No protocol or frontend change: the client still renders each
message_updateit receives exactly as before — it just receives far fewer of them, which also reduces client re-render cost. Full delta rendering (frontend appendsassistantMessageEventdeltas) would take this to 1× and is a natural follow-up; this change is the contained, low-risk server-side win.The coalescing logic is extracted into
lib/event-coalescer.tsso it can be unit-tested independently of the SSE plumbing.Testing
tsc --noEmitpasseseslintpassesnode --test lib/event-coalescer.test.mjs— 5/5 pass, covering: buffering until flush, coalescing consecutive updates to the latest, flushing pending before a following non-update event (ordering), non-update passthrough, and no-op flush.