fix(rpc-gateway): tie the upstream pubsub socket to the client connection - #412
Open
KishiTheMechanic wants to merge 1 commit into
Open
fix(rpc-gateway): tie the upstream pubsub socket to the client connection#412KishiTheMechanic wants to merge 1 commit into
KishiTheMechanic wants to merge 1 commit into
Conversation
…tion `PubsubForward` opens one outbound WebSocket per client, but nothing made that socket's lifetime follow the client's. `connect_loop` ended only when `writer` or `reader` finished, and neither can be relied on: - `writer` waits on `up_rx.recv()`, and the senders are owned by `connect_loop` itself (the local `up_tx` binding plus the clone it keeps alive through its own `Arc<Inner>`). Dropping `PubsubForward` releases a third reference and changes nothing. - `reader` only wakes when the upstream sends a frame. A client that subscribes to something quiet and then leaves produces no frame, so `stream.next()` blocks forever. The task and its TCP socket then leak. Measured in production on 2026-09-04 (indexed-ty6-1): 343 upstream sockets against 3 live client connections and 13 real subscriptions, each socket showing `bytes_sent:356 bytes_acked:357` and no traffic afterwards, growing at ~0.85/min and holding 600 MB RSS. Give `PubsubForward` a `oneshot::Sender` whose receiver is selected on inside `connect_loop`. Dropping the struct — which is what happens when the client connection ends and `ConnectionState` goes out of scope — resolves the receiver and tears the socket down. The regression test spawns an upstream that answers the subscribe and then stays silent, drops the client, and requires the upstream to observe the close. Without the select arm it fails with `Elapsed`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
症状
slv-rpc-gatewayがクライアント 1 本につき 1 本開く upstream pubsub WebSocket が、クライアントが去っても閉じない。〔実測 2026-09-04
indexed-ty6-1〕クライアント接続 3 本・validator 側の実サブスクリプション 13 件に対して、127.0.0.1:7212への upstream が 343 本。プロセスの fd の 97% がこれで、RSS 600 MB。全接続がbytes_sent:356 bytes_acked:357で以後無通信〔ss -Htnim〕= WS upgrade と 1 メソッドだけ送って、あとは何もしないまま閉じない。増加率 0.85 本/分 で、LimitNOFILE=65536まで約 55 日。機序
connect_loopはwriterかreaderのどちらかが終わったときにしか終了しないが、どちらも終わらない条件が普通に起きる。writerはup_rx.recv()で待つ。この channel の sender はconnect_loop自身が持っている — ローカル束縛のup_txと、自分が握るArc<Inner>経由でState::Open { tx }に入っているクローン。PubsubForwardを drop しても 3 本目の参照が消えるだけで、channel は開いたまま。readerは upstream からフレームが来たときにしか起きない。静かな購読をしてから去ったクライアントではフレームが来ないのでstream.next()が永久にブロックする。⇒
tokio::select!が発火せず、task と TCP socket が漏れる。修正
PubsubForwardにoneshot::Sender<()>を持たせ、その receiver をconnect_loopのselect!に足す。クライアント接続が終わってConnectionStateがスコープを抜けるとPubsubForwardが drop され、sender が落ちて receiver が解決し、socket が畳まれる。検証
回帰テスト
upstream_pubsub_socket_closes_when_client_disconnectsを追加。upstream モックは subscribe に応答したあと沈黙し、read 側が close を観測したらoneshotで報告する。クライアントを unsubscribe せずに drop し、5 秒以内に upstream が閉じることを要求する。60 passed; 0 failedselect!から_ = closed_rx => {}の 1 行だけ外して同じテストを撃つ →panicked ... upstream socket must close when the client disconnects: Elapsed(())/test result: FAILED🤖 Generated with Claude Code