Skip to content

fix: stop orphaned zms processes accumulating, without killing streams on tab hide - #5120

Open
connortechnology wants to merge 3 commits into
masterfrom
4706-zms-orphan-cleanup
Open

fix: stop orphaned zms processes accumulating, without killing streams on tab hide#5120
connortechnology wants to merge 3 commits into
masterfrom
4706-zms-orphan-cleanup

Conversation

@connortechnology

@connortechnology connortechnology commented Sep 7, 2026

Copy link
Copy Markdown
Member

Fixes the orphaned zms accumulation reported in #4706, without killing streams on tab hide — which @connortechnology ruled out there, since killing takes the status polling and alarm sounds down with it.

Reproduced and verified on a live instance. Two monitors on a montage page, one simulated tab hide/show cycle each time:

before after 1 cycle after 2 cycles processes
stock zms + stock MonitorStream.js 2 4 6 replaced each cycle, old ones orphaned
with this branch 2 2 2 the same two, resumed

The pre-fix run is exactly the accumulation #4706 describes. With the branch the process ids do not change at all across the cycles, and both pictures are live afterwards.

The two defects

A stopped zms could never exit (src/zm_monitorstream.cpp:670). The stopped branch slept and continued without writing anything, so the socket was never touched and a client that had gone away was never discovered: no write, no EPIPE, no SIGPIPE. The continue also skipped the ttl check at the bottom of the loop, so even a caller's deadline did not apply. The only way out was an explicit CMD_QUIT, and a stream that missed one stayed until the machine was restarted. This is why they accumulate rather than merely appear.

The last captured frame is now held on entering the stopped state and re-sent every five seconds, as the paused state already does, and ttl applies here too. Confirmed in the logs of a running stream:

DB1 [Got STOP command]
DB1 [Saving stopped image from index 0]
DB2 [Sending keepalive frame while stopped]      <- every 5s thereafter

setLastViewed is deliberately still not called: capture and decoding should not be held active for a stream that is not playing.

select_zms() replaced the connkey without quitting the process it addressed (web/js/MonitorStream.js). Once the key is gone nothing can reach that zms again, so no CMD_QUIT can ever be delivered to it. getStreamCmdResponse() already learned this — its reload path calls quitConnKey() first, with a comment saying exactly why — but the path every ordinary start() takes did not. quitConnKey had precisely two references in the file: its own definition and that one use.

Resuming instead of rebuilding

The third commit makes the hide/show path resume the stopped zms with CMD_PLAY rather than build a new one — what CMD_STOP is designed for, and what the discussion on #4706 assumed should work. It did not, for three separate reasons, none of which were visible from reading:

  1. stop() clears activePlayer, which is the only thing the resume branch tests, so after a stop it was unreachable.
  2. srcAuthCurrent required a non-empty zmAuth.hash, which is '' whenever auth is off — so those installs always rebuilt, for no reason, since there is no hash that can go stale.
  3. streamCommand() drops anything sent while !started, and started is not set until the end of select_zms(). The resume issued its CMD_PLAY into nothing. Resuming after a pause worked only because pause() leaves started set.

With those fixed the zms status reports stopped: 0 and ~15 fps — and the picture still looked frozen, because the rebuild path clears the "Loading..." info block from img_onload, which cannot fire when src never changes. The resumed stream was playing underneath its own overlay. Clearing the block on the resume path is the last piece.

@IgorA100 reported on #4706 that a CMD_PLAY after CMD_STOP does not restart the video. That was correct, and points 3 and 4 above are why.

Testing

  • zms builds clean, no new warnings; full C++ suite 138/138 pass; ESLint clean on MonitorStream.js.
  • Live before/after process counts as tabled above, on a 2-monitor montage page.
  • Keepalive behaviour confirmed in zms debug logs.
  • Resume confirmed end to end: same pids across two cycles, stopped: 0 and ~15 fps from the stream status query, and the burnt-in camera timestamp advancing in the browser afterwards.

Relationship to #4706

Supersedes it. Same diagnosis of the symptom, different mechanism: nothing is killed on tab hide, and the process that does get replaced is now told to quit rather than left behind.

🤖 Generated with Claude Code

connortechnology and others added 2 commits September 7, 2026 15:38
A zms in the stopped state could never exit. The branch handling it slept and
continued without writing anything, so the socket was never touched and a
client that had gone away was never discovered - no write, no EPIPE, no
SIGPIPE. The continue also skipped the ttl check at the bottom of the loop, so
even a deadline set by the caller did not apply. The only way out was an
explicit CMD_QUIT, and any stream that missed one stayed until the machine was
restarted.

That is what accumulates on the montage page: the tab is hidden, the browser
side sends CMD_STOP, and if the stream is later replaced rather than resumed,
nothing can address the old process again and it sleeps forever.

Hold the last captured frame on entering the stopped state and re-send it
every five seconds, the way the paused state already does, and apply ttl here
too. The frame is what makes a departed client detectable; it also keeps the
connection open for the resume that stopped is meant to allow, which is what
the state was documented to do. Where nothing has been captured yet there is
no frame to hold, so a text frame is sent instead - the write matters more
than what is in it.

setLastViewed is deliberately still not called: capture and decoding should not
be held active for a stream that is not playing.

refs #4706

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
select_zms() mints a fresh connkey whenever it has to rebuild the stream src,
without telling the process the old connkey belonged to that it is finished.
Once the key is replaced nothing can reach that process again: no CMD_QUIT can
be delivered, and a stopped one will not notice on its own.

getStreamCmdResponse() already learned this - its reload path calls
quitConnKey() first, with a comment saying why - but the path every ordinary
start() takes did not. quitConnKey had exactly two references in the file: its
own definition and that one use.

refs #4706

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@connortechnology

Copy link
Copy Markdown
Member Author

Tested on a live instance and dropped one of the three commits as a result.

Before/after, two monitors on a montage page, one hide/show cycle each time:

start 1 cycle 2 cycles
stock zms + stock MonitorStream.js 2 4 6
this branch 2 2 2

The keepalive works as intended — Got STOP command, Saving stopped image from index 0, then Sending keepalive frame while stopped every 5s, process alive and reachable throughout.

What I got wrong: the third commit made hide/show resume the stopped zms with CMD_PLAY rather than rebuilding. On paper that is what CMD_STOP is for. In practice the connkey and process are preserved, started returns to true — and the picture stays frozen on the keepalive frame. The streaming process logs viewing fps: 0.00 and goes on sending keepalives; no CMD_PLAY ever reaches it.

@IgorA100 said exactly this on #4706 and I dismissed it as the client re-creating the stream. It wasn't. Resuming on an existing connkey needs the command delivery path investigated first, which is a separate piece of work and not a MonitorStream.js change. That commit is gone; the branch now rebuilds as before, but quits the old process instead of abandoning it.

select_zms() has a branch that resumes an existing zms with CMD_PLAY rather
than rebuilding the stream, but three things stopped it ever working. Every one
of them showed up only when the page was driven for real.

stop() ends by clearing activePlayer, which is the only thing that branch
tests, so after any stop it was unreachable and start() fell through to "new
src, new connkey" - a second zms, the first left running and no longer
addressable. That is the montage page's accumulating processes: hide the tab,
the handler stops each stream, and returning replaces rather than resumes. The
same happens when a monitor is scrolled out of view and back. stop() now
remembers what it shut down, and select_zms() resumes on that as well as on
activePlayer, provided we still hold the connkey to address it.

srcAuthCurrent required a non-empty zmAuth.hash, which is '' whenever auth is
off or the relay carries no hash. There is nothing that can go stale in that
case, so the src is as current as it will ever be; requiring a hash sent every
such install down the rebuild path for no reason.

streamCommand() drops anything sent while !started, and started is not set
until the end of select_zms(), so the resume issued its CMD_PLAY into nothing
and the stream stayed stopped. Resuming after a pause worked only because
pause() leaves started set. It is now set before the command goes out.

Finally, the rebuild path clears the "Loading..." info block from img_onload,
which cannot fire on a resume because src never changes. Without clearing it,
a stream that had in fact resumed sat behind that block and its still image and
looked frozen - the fault that made this look unfixable at first.

restart() is excluded from all of it: it is the error path, whatever failed may
be that very zms, and a broken img is not repaired by CMD_PLAY.

Verified on a live montage page with two monitors: across two hide/show cycles
the same two zms processes are kept - no orphans, no respawn - and both
pictures are live afterwards, with the zms status reporting stopped=0 and
~15 fps.

refs #4706

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@connortechnology

Copy link
Copy Markdown
Member Author

Update: the resume commit is back, and now actually works. My earlier comment said resuming a stopped zms was not achievable from MonitorStream.js — that was wrong, I stopped one layer short.

Three things were preventing the CMD_PLAY from ever arriving:

  1. stop() clears activePlayer, the only thing the resume branch tests.
  2. srcAuthCurrent demands a non-empty zmAuth.hash, which is '' when auth is off — so those installs always rebuilt, though there is no hash that can go stale.
  3. streamCommand() drops anything sent while !started, and started is not set until the end of select_zms(). The resume was issuing its CMD_PLAY into nothing. Resuming after a pause works only because pause() leaves started set.

With those fixed the stream status came back stopped: 0, ~15 fps — and the picture still looked frozen. That last part was a red herring of the same kind: the rebuild path clears the "Loading..." info block from img_onload, which cannot fire when src never changes, so the resumed stream was playing underneath its own overlay.

Live result on a 2-monitor montage page, two hide/show cycles:

  • the same two zms pids throughout — not replaced, not orphaned
  • both pictures live afterwards, camera timestamp advancing

So hide/show now costs nothing to return from: no new process, no restart latency, and the status polling that drives alarm sounds is never interrupted. @IgorA100 — your observation on #4706 that CMD_PLAY after CMD_STOP does not restart the video was right; points 3 and 4 are why.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant