Skip to content

fix(http): time out stalled transactions without killing switched protocols - #2008

Open
nickanderson wants to merge 1 commit into
mendersoftware:masterfrom
nickanderson:ME-731/master
Open

fix(http): time out stalled transactions without killing switched protocols#2008
nickanderson wants to merge 1 commit into
mendersoftware:masterfrom
nickanderson:ME-731/master

Conversation

@nickanderson

Copy link
Copy Markdown
Member

A server that accepts a request and then never answers left the response read
outstanding indefinitely, wedging the client silently — alive, no crash, no log
output, recoverable only by restart. Reproduced in a VM on both 5.0.5 and 5.1.0.

The gap was ReadHeader(): AsyncReadNextBodyPart already had a 5 minute timer,
but nothing bounded the wait for response headers.

This arms Beast's stream timeout before each operation of a transaction and
disarms it at the 101 Switching Protocols handover. That handover is what the
previous attempt (d8ddf6da / 4aa01ee8) got wrong: it left the timeout armed,
and Beast's timeout_handler calls close() on expiry regardless of who owns the
socket by then, which killed mender-connect's forwarded WebSocket after five
minutes and got the fix reverted under MEN-9433.

Connect is intentionally not covered — asio::async_connect runs on
lowest_layer(), the raw socket, which bypasses the stream timeout entirely.

Timeout is ClientConfig::stream_timeout_seconds, default 300s. No client
endpoint long-polls, and artifact download is a per-chunk body read (unchanged).

Tests

Both are red/green verified — each fails when its own half of the fix is removed:

test without the fix
HttpTest.TestStalledResponseTimesOut hangs to the event-loop watchdog (5006 ms)
HttpForwarderTests.ProtocolSwitchSurvivesIdlePastStreamTimeout fails at 1219 ms — MEN-9433 exactly

The forwarder already had five ProtocolSwitch* tests, but all exercise immediate
data flow; none left the connection idle, which is why MEN-9433 escaped.

Full suites pass: http_test 47/47, http_forwarder_test 13/13.

End-to-end on 5.0.5 with a stalled deployments/next and the reporting customer's
state scripts: unpatched logs nothing for 420 s, patched logs
Error while polling for deployment: The operation timed out at exactly t+300 s,
runs the SyncError/IdleEnter scripts, and resumes polling.

Ticket: ME-731

@mender-test-bot

Copy link
Copy Markdown

@nickanderson, start a full client pipeline with:

  • mentioning me and start client pipeline

my commands and options

You can prevent me from automatically starting CI pipelines:

  • if your pull request title starts with "[NoCI] ..."

You can trigger a client pipeline on multiple prs with:

  • mentioning me and start client pipeline --pr mender/127 --pr mender-connect/255

You can trigger a client pipeline for a specific Mender Client release with:

  • mentioning me and start client pipeline --release 6.0.x (can be given multiple times)
  • by default, a pipeline is triggered for each supported release the component is a part of

You can trigger GitHub->GitLab branch sync with:

  • mentioning me and sync

You can print PR statistics for a repository with:

  • mentioning me and print fast pr stats (Team stats only)
  • mentioning me and print full pr stats (Detailed report)
  • options: --repo <repo>, --team <name>, --all-repos, --exclude-drafts, --exclude-user <user>
  • mentioning me and print full pr stats --repo mender --all-repos --exclude-drafts

You can deploy a review app with:

  • mentioning me and start review app (OS environment)
  • mentioning me and start review app enterprise (Enterprise environment)

You can run e2e tests against a deployed review app with:

  • mentioning me and start review tests (defaults to os environment)
  • mentioning me and start review tests enterprise (for enterprise environment)

You can cherry pick to a given branch or branches with:

  • mentioning me and:
 cherry-pick to:
 * 1.0.x
 * 2.0.x

@nickanderson
nickanderson requested review from danielskinstad and removed request for danielskinstad August 4, 2026 19:02
@nickanderson
nickanderson marked this pull request as draft August 4, 2026 19:03
@nickanderson
nickanderson marked this pull request as ready for review August 5, 2026 13:12
…tocols

A server that accepts a request and never answers left the response read
outstanding forever, wedging the client until restart. Arm Beast's stream timeout
per operation and disarm it at the 101 Switching Protocols handover -- leaving it
armed there closed mender-connect's WebSocket after five minutes and got the
previous attempt reverted (MEN-9433).

Ticket: ME-731
Changelog: Fixed the client hanging indefinitely when a server accepts a request but
never sends a response, which left the device unmanageable until it was restarted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Nick Anderson <nick@cmdln.org>
@mender-test-bot

mender-test-bot commented Aug 5, 2026

Copy link
Copy Markdown

Merging these commits will result in the following changelog entries:

Changelogs

mender (ME-731/master)

New changes in mender since master:

Bug Fixes
  • Fixed the client hanging indefinitely when a server accepts a request but
    never sends a response, which left the device unmanageable until it was restarted.
    (ME-731, MEN-9433)

@codecov

codecov Bot commented Aug 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants