jmp shell Unix socket becomes unreachable when router tunnel drops; tunnel should auto-reconnect

[ambient-code]

Summary

When using jmp shell, the connection path is:

j (CLI) → Unix socket → [local proxy] ──tunnel──► jumpstarter-router ──► exporter

If the tunnel leg (between the local proxy and the jumpstarter-router) drops while the shell session is still active, the local Unix socket remains up but all j commands time out with:

Error: Service is temporarily unavailable or unreachable. Verify endpoint/network and retry.
Details: failed to connect to all addresses; last error: UNKNOWN: unix:/run/user/<uid>/jumpstarter-<id>/socket: connection attempt timed out before receiving SETTINGS frame

The proxy does not detect the tunnel failure and does not attempt to reconnect. The only workaround is to exit and re-enter the shell, which recreates the tunnel from scratch.

Steps to Reproduce

1. Acquire a lease and enter a jmp shell
2. Perform operations (e.g. flash a device)
3. While still in the shell, allow the router tunnel to drop (observed intermittently; in the reported case it happened ~14:15 eastern after sustained use)
4. Run any j command (e.g. j power cycle)
5. Observe the SETTINGS frame timeout error

Expected Behavior

The local proxy inside jmp shell should detect when the router tunnel drops and transparently re-establish it, so j commands continue to work without requiring the user to exit and re-enter the shell.

Actual Behavior

The tunnel drop is not detected. The Unix socket stays up but all commands hang and eventually time out waiting for the HTTP/2 SETTINGS frame. Re-entering the shell resolves the issue.

Notes

• The device/exporter itself is not involved in this failure — the breakage is in the tunnel between the client-side local proxy and the jumpstarter-router
• Reproducibility: observed intermittently during sustained shell sessions; not yet confirmed to be reproducible on demand
• Reported via Slack by sbertram; majopela opened this issue to track it

[ambient-code]

Investigation Findings

I investigated the codebase to trace the full connection path and identify the root cause. Here is what I found.

Architecture Overview

The connection path for jmp shell is:

j (CLI) --> Unix socket --> [TemporaryUnixListener + handle_async] --Dial--> controller --token/endpoint--> router <-- exporter

Key code locations:

• Shell entry: python/packages/jumpstarter-cli/jumpstarter_cli/shell.py - _run_shell_with_lease_async() sets up the Unix socket via lease.serve_unix_async()
• Unix socket + tunnel handler: python/packages/jumpstarter/jumpstarter/client/lease.py - Lease.handle_async() calls Dial on the controller, gets a (router_endpoint, router_token), then connects to the router via connect_router_stream()
• Router stream bridge: python/packages/jumpstarter/jumpstarter/common/streams.py - connect_router_stream() opens a gRPC channel to the router and uses forward_stream() to bidirectionally pipe data between the Unix socket client and the router
• Router service (Go): controller/internal/service/router_service.go - Stream() pairs two peers (client + exporter) by stream name using sync.Map, then pipes data with Forward()

Root Cause

The root cause is in Lease.handle_async() (lease.py, line 313). Each time a j command connects to the Unix socket, handle_async is invoked by the TemporaryUnixListener. It calls Dial to get a fresh router token/endpoint, then opens a new connect_router_stream session for that single connection. This per-connection Dial model is actually correct -- each j command gets its own tunnel.

The problem is: when the tunnel drops mid-stream (e.g., the gRPC connection to the router dies), handle_async exits silently (line 355-356 logs a warning and returns) but does NOT surface any error to the Unix socket client. The forward_stream context manager catches stream errors via copy_stream which only logs warnings (streams/common.py, line 34). The Unix socket side simply sees the stream end, but the gRPC client on the j side may still be waiting for a response.

For subsequent j commands, the Unix socket listener is still running and handle_async will be called again with a new Dial, which should work. The SETTINGS frame timeout error described in the issue suggests something different is happening: the gRPC client-side channel (created by client_from_path in _run_shell_with_lease_async) is caching a stale HTTP/2 connection to the Unix socket that has become unhealthy.

The real problem chain is:

1. The gRPC channel created by client_from_path is established once and reused for the lifetime of the shell session
2. This gRPC channel connects to the Unix socket
3. When a tunnel drops, the gRPC channel's HTTP/2 connection to the Unix socket may enter a bad state
4. Subsequent j commands reuse the same gRPC channel which is now broken at the HTTP/2 layer
5. The new connection attempt hangs waiting for the HTTP/2 SETTINGS frame that will never come

Why the Unix socket stays up but commands fail

The TemporaryUnixListener (tempfile.py) remains alive for the entire shell session. It accepts new TCP-level connections just fine. But the gRPC channel in the j CLI reuses its existing HTTP/2 connection, which is broken. The gRPC keepalive settings (20s keepalive_time, 180s keepalive_timeout in grpc.py line 161-162) are quite generous, meaning a dead tunnel may not be detected for up to 3 minutes.

Proposed Fix Plan

Fix 1: Improve gRPC keepalive and reconnect detection (Quick Win)

In python/packages/jumpstarter/jumpstarter/common/grpc.py, tighten the keepalive settings for the Unix socket channel:

• Reduce grpc.keepalive_timeout_ms from 180000 (3 min) to something like 10000-20000 (10-20s)
• This would cause the gRPC channel to detect the broken HTTP/2 connection faster and trigger reconnection via gRPC's built-in channel reconnection

Note: these options are used for both the router channel and the Unix socket channel. The fix may need to differentiate between them, or the tighter settings may be acceptable globally.

Fix 2: Add reconnect logic to handle_async (Core Fix)

Modify Lease.handle_async() to retry the Dial + connect_router_stream when the router tunnel drops, instead of silently returning. This would look like:

async def handle_async(self, stream):
    max_retries = 3
    backoff = 0.5
    for attempt in range(max_retries + 1):
        try:
            response = await self.controller.Dial(...)
            async with connect_router_stream(..., stream, ...):
                return  # Clean exit
        except (BrokenResourceError, EndOfStream, AioRpcError) as e:
            if attempt < max_retries and not self.lease_ended:
                logger.warning("Router tunnel dropped, reconnecting (attempt %d/%d): %s", 
                               attempt + 1, max_retries, e)
                await sleep(backoff * (2 ** attempt))
                continue
            logger.warning("Router tunnel failed permanently: %s", e)
            return

However, this only helps if the same Unix-socket connection is still alive on the j side. For a new j command, the listener already creates a new handle_async invocation with a fresh Dial, so the retry logic here mainly helps long-running streaming operations (e.g., flashing).

Fix 3: Dedicated health-check / liveness on the Unix socket channel (Most Robust)

Add a background health monitoring task that runs alongside the shell session, similar to how log_stream_async already has reconnect logic (core.py line 462-540). This monitor would:

1. Periodically probe the Unix socket gRPC channel health (e.g., via gRPC health check or a lightweight RPC)
2. If the probe fails, close and recreate the gRPC channel
3. Surface a user-visible message (like the token expiry warnings already do)

This is the most complex option but provides the best UX.

Fix 4: Propagate errors to the Unix socket client (Complementary)

In streams/common.py, the copy_stream function logs stream errors as warnings but does not propagate them upward. If forward_stream detected a broken tunnel and explicitly closed the Unix socket side with an error, the gRPC channel would see the connection reset and establish a new one on the next j command. Currently, the silent handling causes the gRPC client to think the connection is still viable.

Recommended Approach

I would recommend combining Fix 1 (quick win to reduce detection time) and Fix 4 (propagate errors so gRPC reconnects naturally), as they are the lowest-risk changes that address the core issue. Fix 2 adds value for long-running operations. Fix 3 is a more significant undertaking that could be done as a follow-up.

Edge Cases and Risks

• In-flight data during reconnection: For Fix 2, data in the forward_stream pipe will be lost when the tunnel drops. This is unavoidable -- the j command will see an error and the user must retry. The improvement is that the next command works.
• Lease expiry during reconnect: The handle_async already checks for permission-denied (lease transferred). Reconnect logic should also check self.lease_ended to avoid retrying after lease expiry.
• Token validity: The router token from Dial has a 30-minute TTL (controller_service.go line 719). A reconnect attempt within the same handle_async call would need a fresh Dial to get a new token anyway, so this is handled naturally.
• Backward compatibility: All proposed changes are internal to the client. No protocol changes are needed. The router and exporter are unaffected.

Existing Reconnect Patterns in the Codebase

The codebase already has reconnect/retry patterns that can serve as models:

• log_stream_async in core.py (line 460-540): exponential backoff reconnect for log streaming
• _get_with_retry in lease.py (line 137-153): tenacity-based retry with exponential backoff for lease polling
• handle_async Dial retry loop (lease.py line 318-345): time-bounded retry for "exporter not ready"

[ambient-code]

Investigation Findings

I traced the full connection path for jmp shell:

j (CLI) -> Unix socket -> Lease.handle_async() -> controller.Dial() -> connect_router_stream() -> router -> exporter

How it works

1. jmp shell creates a TemporaryUnixListener that calls Lease.handle_async(stream) for each new Unix socket connection from a j command.
2. handle_async calls controller.Dial(lease_name) to get a router_endpoint and router_token.
3. connect_router_stream() opens a new gRPC channel to the router endpoint and bidirectionally forwards data via forward_stream.

Root cause

Two issues cause j commands to time out when the router tunnel drops:

1. No retry for transient network errors in Dial:
The existing handle_async only retried Dial on FAILED_PRECONDITION ("not ready"). When the tunnel drops, Dial fails with UNAVAILABLE, and the handler returns silently without retrying. The next j command triggers a new handle_async, which also fails immediately.

2. No channel readiness check in connect_router_stream:
Even when Dial succeeds and returns a router endpoint, connect_router_stream opens a gRPC channel to the router and immediately starts streaming. If the router is unreachable, the channel never completes the HTTP/2 handshake, causing the j command to hang indefinitely waiting for the SETTINGS frame — exactly matching the reported symptom.

Fix

• Added retry with exponential backoff for transient gRPC errors (UNAVAILABLE, RESOURCE_EXHAUSTED, ABORTED, INTERNAL) in the Dial call.
• Added retry with exponential backoff for the router connection establishment, including re-dialing for fresh tokens on retry.
• Added a channel_ready() timeout (10s) in connect_router_stream so connections to unreachable routers fail fast with an UNAVAILABLE error instead of hanging on the HTTP/2 SETTINGS frame exchange.

All retries are bounded by the existing dial_timeout (default 30s).

PR incoming.

[ambient-code]

PR: #679