Skip to content

fix(http): time out stalled transactions without killing switched protocols - #2009

Open
nickanderson wants to merge 1 commit into
mendersoftware:5.0.xfrom
nickanderson:cherry-5.0.x-ME-731-http-stream-timeout
Open

fix(http): time out stalled transactions without killing switched protocols#2009
nickanderson wants to merge 1 commit into
mendersoftware:5.0.xfrom
nickanderson:cherry-5.0.x-ME-731-http-stream-timeout

Conversation

@nickanderson

@nickanderson nickanderson commented Aug 4, 2026

Copy link
Copy Markdown
Member

Backport of #2008 to 5.0.x. Both 5.0.5 and 5.1.0 were verified affected, so upgrading is
not a workaround.

A server that accepts a request and then never answers left the response read outstanding
indefinitely, wedging the client silently — alive, no crash, no log output, recoverable only
by restart. The gap was ReadHeader(): AsyncReadNextBodyPart already had a timer, but
nothing bounded the wait for response headers.

This arms Beast's stream timeout before each operation of a transaction and disarms it at
the 101 Switching Protocols handover
. That handover is what the previous attempt
(d8ddf6da / 4aa01ee8) got wrong: it left the timeout armed, and Beast's timeout_handler
calls close() on expiry regardless of who owns the socket by then, which killed
mender-connect's forwarded WebSocket after five minutes and got the fix reverted under
MEN-9433.

Connect is intentionally not covered — asio::async_connect runs on lowest_layer(), the
raw socket, which bypasses the stream timeout entirely.

Why this touches an extra file versus master

src/common/common.hpp is included here to bring common::def_value over from master; 5.0.x
only has def_bool. It is needed for the new ClientConfig::stream_timeout_seconds field.

A plain default member initializer would have been smaller but is wrong on this branch:
cmake/build_mode.cmake compiles Debug with -std=c++11 deliberately, and under C++11 a
default member initializer stops ClientConfig being an aggregate — which breaks
http::ClientConfig client_config {"server.localhost.crt"} in the HTTPS tests. That is the
same reason def_bool already exists.

Tests

Both are red/green verified — each fails when its own half of the fix is removed:

test without the fix
HttpTest.TestStalledResponseTimesOut hangs to the event-loop watchdog (5006 ms)
HttpForwarderTests.ProtocolSwitchSurvivesIdlePastStreamTimeout fails at 1219 ms — MEN-9433 exactly

The forwarder already had five ProtocolSwitch* tests, but all exercise immediate data flow;
none left the connection idle, which is why MEN-9433 escaped.

Built on 5.0.x in Debug (the C++11 path, so the aggregate trap above is exercised);
http_test and http_forwarder_test pass 58/58 via ctest.

End-to-end on 5.0.5 with a stalled deployments/next and the reporting customer's state
scripts: unpatched logs nothing for 420 s, patched logs
Error while polling for deployment: The operation timed out at exactly t+300 s, runs the
SyncError/IdleEnter scripts, and resumes polling.

Ticket: ME-731

@mender-test-bot

Copy link
Copy Markdown

@nickanderson, start a full client pipeline with:

  • mentioning me and start client pipeline

my commands and options

You can prevent me from automatically starting CI pipelines:

  • if your pull request title starts with "[NoCI] ..."

You can trigger a client pipeline on multiple prs with:

  • mentioning me and start client pipeline --pr mender/127 --pr mender-connect/255

You can trigger a client pipeline for a specific Mender Client release with:

  • mentioning me and start client pipeline --release 6.0.x (can be given multiple times)
  • by default, a pipeline is triggered for each supported release the component is a part of

You can trigger GitHub->GitLab branch sync with:

  • mentioning me and sync

You can print PR statistics for a repository with:

  • mentioning me and print fast pr stats (Team stats only)
  • mentioning me and print full pr stats (Detailed report)
  • options: --repo <repo>, --team <name>, --all-repos, --exclude-drafts, --exclude-user <user>
  • mentioning me and print full pr stats --repo mender --all-repos --exclude-drafts

You can deploy a review app with:

  • mentioning me and start review app (OS environment)
  • mentioning me and start review app enterprise (Enterprise environment)

You can run e2e tests against a deployed review app with:

  • mentioning me and start review tests (defaults to os environment)
  • mentioning me and start review tests enterprise (for enterprise environment)

You can cherry pick to a given branch or branches with:

  • mentioning me and:
 cherry-pick to:
 * 1.0.x
 * 2.0.x

@nickanderson
nickanderson marked this pull request as draft August 4, 2026 19:52
@nickanderson
nickanderson force-pushed the cherry-5.0.x-ME-731-http-stream-timeout branch from 2c7195f to 936be89 Compare August 5, 2026 13:19
…tocols

A server that accepts a request and never answers left the response read
outstanding forever, wedging the client until restart. Arm Beast's stream timeout
per operation and disarm it at the 101 Switching Protocols handover -- leaving it
armed there closed mender-connect's WebSocket after five minutes and got the
previous attempt reverted (MEN-9433).

Backport note: common::def_value does not exist on 5.0.x, so it comes over from
master too. A plain default member initializer would not do, since Debug builds
use C++11, where that stops ClientConfig being an aggregate.

Ticket: ME-731
Changelog: Fixed the client hanging indefinitely when a server accepts a request but
never sends a response, which left the device unmanageable until it was restarted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Nick Anderson <nick@cmdln.org>
(cherry picked from commit c7a9290)
@nickanderson
nickanderson force-pushed the cherry-5.0.x-ME-731-http-stream-timeout branch from 936be89 to 53ec1d2 Compare August 5, 2026 13:34
@nickanderson
nickanderson marked this pull request as ready for review August 5, 2026 16:36
@mender-test-bot

mender-test-bot commented Aug 5, 2026

Copy link
Copy Markdown

Merging these commits will result in the following changelog entries:

Changelogs

mender (cherry-5.0.x-ME-731-http-stream-timeout)

New changes in mender since 5.0.x:

Bug Fixes
  • Fixed the client hanging indefinitely when a server accepts a request but
    never sends a response, which left the device unmanageable until it was restarted.
    (ME-731, MEN-9433)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants