Linux: a child process stuck in uninterruptible sleep (D) stalls the agent's message loop — device stays online but stops answering (repro: lshw + VMware vmnet)

[jptamayo76]

Summary

On Linux, if any child process the agent spawns enters uninterruptible sleep (D) and never exits, the agent stops reading its server socket. The device keeps showing as online in MeshCentral but no longer answers anything, so Remote Desktop can never be established.

I hit this on a host with VMware Workstation installed, where the agent's hardware-inventory child (lshw -class disk) hangs forever. The lshw hang itself is not an agent bug — but the agent going deaf because of it is, and it would happen with any command that never returns.

The witness that pins it down is the Recv-Q of the agent→server socket: it sits stuck while the child is in D, and drains to 0 the instant the blocking helper is killed.

Environment

• Agent core: commit 62b206e0b485b296e8a73a6547cef02bbf5a2d62, ARCHID 6 (Linux x86-64), bundled with MeshCentral 1.2.1.
• OS: Ubuntu 24.04.4 LTS, kernel 7.0.0-28-generic, x86-64. Desktop is SDDM + LXQt on X11.
• Install: system service meshagent.service, running as root, --no-embedded=1 --installedByUser=0.
• Relevant third party: VMware Workstation, with vmnet1 and vmnet8 up and the vmnet kernel module loaded.
• lshw: 02.19.git.2021.06.19.996aaad9c7-2ubuntu0.24.04.1.

Symptom in the web UI

Two grades, depending on how long the agent has been deaf:

1. On the Desktop tab the Connect button is disabled.
2. If the agent is momentarily unblocked, Connect becomes enabled but clicking it stays at "Setup…" forever.

The device is shown online the whole time. Server side everything looks fine: agent.caps = 31 (desktop supported) and the users' node rights include REMOTECONTROL without NODESKTOP.

What actually happens

#	Step	Evidence
1	The agent spawns sh → lshw -class disk (hardware inventory)	ps: meshagent(1767) → sh(9687) → lshw(9688)
2	lshw issues an inet ioctl with interface name /dev/vmnet1	kernel stack below
3	The kernel tries to autoload a module for that name and runs /sbin/modprobe -q -- /dev/vmnet1	kernel stack below
4	modprobe opens that path as if it were a .ko and blocks forever inside VMware's vmnet driver	fd 0 -> /dev/vmnet1, stack below
5	lshw stays in D forever, and the agent stops reading its server socket	Recv-Q stuck (below)
6	The agent respawns the inventory later, so the stall is permanent	new sh/lshw appear right after the old ones are cleared

Kernel stack of the hung lshw

[<0>] call_usermodehelper_exec+0x15f/0x1b0
[<0>] __request_module+0x200/0x300
[<0>] dev_load+0x70/0xa0
[<0>] devinet_ioctl+0x68/0x740
[<0>] inet_ioctl+0x16f/0x1f0
[<0>] sock_do_ioctl+0x7b/0x140
[<0>] __x64_sys_ioctl+0xa5/0x100

Kernel stack of the modprobe helper it is waiting for

[<0>] VNetUserIfRead+0x29a/0x3f0 [vmnet]
[<0>] VNetFileOpRead+0x30/0x80 [vmnet]
[<0>] vfs_read+0xba/0x390
[<0>] ksys_read+0x71/0xf0
[<0>] __x64_sys_read+0x19/0x30

# ls -l /proc/<modprobe-pid>/fd
lr-x------ 1 root root 64 ... 0 -> /dev/vmnet1

The decisive witness — Recv-Q of the agent→server socket

# while the lshw child is stuck in D
ESTAB  Recv-Q 122  <host>:50065 -> <server>:443   users:(("meshagent",pid=31406,fd=13), ...)

# a few seconds after killing the blocking modprobe helper (lshw then completes)
ESTAB  Recv-Q 0    <host>:50065 -> <server>:443

122 bytes from the server had been sitting unread for minutes; they were consumed immediately once the child could exit. That is the whole bug in one measurement.

Note that the agent's main thread looks healthy the entire time — it sits in pselect:

[<0>] poll_schedule_timeout.constprop.0+0x41/0x90
[<0>] do_select+0x704/0x880
[<0>] core_sys_select+0x408/0x620
[<0>] do_pselect.constprop.0+0xe8/0x190

So the native chain is alive while the layer above it is not making progress. Looking at thread state alone is misleading here.

Workaround that fixes it

The agent never loads kernel modules, and the second autoload attempt in dev_load() — request_module("%s", name), the one that produces modprobe -q -- /dev/vmnet1 — only happens if the caller holds CAP_SYS_MODULE. Dropping that capability from the service makes the kernel not even try:

# /etc/systemd/system/meshagent.service.d/10-hardening.conf
[Service]
CapabilityBoundingSet=~CAP_SYS_MODULE

After a restart, verified: CapBnd with the CAP_SYS_MODULE bit cleared, no hung lshw, no stuck modprobe, no processes in D, Recv-Q = 0 — and Remote Desktop connects and renders normally. Positive control: lshw -class disk run from a root shell that does hold the capability still hangs, so it is the capability that makes the difference and not a side effect of the restart.

Side benefit: systemctl stop meshagent went from 90 s followed by SIGKILL (systemd waiting on the D child) to 0.04 s.

A second, separate observation

MeshCentral 1.2.1 already runs the inventory command with a flag that avoids the offending scan, in agents/modules_meshcore/computer-identifiers.js:193:

child.stdin.write("lshw -class disk -disable network | tr '\n' '`' | awk '…");

and that flag does prevent the hang on this host:

command	result
lshw -class disk	hangs (killed at 20 s)
lshw -class disk -disable network	exits 0 in 2 s

But the agent on this host runs the un-flagged form. Read from /proc/<pid>/cmdline (not ps, so no truncation), with the full ancestry:

lshw pid=36870
args: lshw -class disk
  <- pid=36869 sh: sh                                  (command fed through stdin)
  <- pid=36802 meshagent: /opt/.../meshagent --no-embedded=1 --installedByUser=0

The server has exactly one copy of computer-identifiers.js, it does contain -disable network, and there is no module override in meshcentral-data. So the running agent appears to be executing an older copy of that module than the one the server holds. I could not pin down where that copy comes from — the module cache in meshagent.db is not stored as plain text (a strings scan finds no module names at all, including modules the agent demonstrably uses), so I could not inspect it. If module updates can go stale like this, that may be worth a look independently of the stall.

Why I think this is an agent-side issue

The lshw hang belongs to VMware and to lshw. What does not is that one stuck child silences the whole agent. Any long-lived or wedged helper would do the same, and the failure is particularly unpleasant because the device still reports as online, so nothing surfaces the problem — no error, no log line, no offline event. The two UI states (Connect greyed out, or stuck at "Setup…") give no hint of the cause.

I am not proposing a patch because the right fix looks like a design decision: bounding the wait for inventory children with a timeout, or not letting child completion gate the message loop at all. I did not read enough of the agent's child-process handling to know which is appropriate, and I would rather report the measurement than guess at the implementation.

Happy to run further tests on this host — it reproduces on demand (re-add CAP_SYS_MODULE and the stall comes back within seconds of an inventory pass).

[si458]

This shouldn't be a problem anymore as ur AI explains
We already made the changes server side, so it doesn't call the original identifiers.js file anymore
If ur hitting this problem, we need to work out what's calling that command cuz in theory it shouldn't

[jptamayo76]

Thanks for looking at this.

You're right that the server-side copy is fine — I checked that before opening the issue.
The module on the server carries the -disable network form, and a scan of the running
server's memory found zero occurrences of the flagless form.

The call is not coming from the server. It comes from the agent's boot core, which
runs before any core the server sends. Here is what I measured on the affected machine:

• Read from /proc/<pid>/cmdline (not ps, so nothing is truncated), with the full
  ancestry: meshagent -> sh (command fed through stdin) -> lshw -class disk.
  No -disable network.
• After systemctl restart meshagent, repeated twice with the same result: the flagless
  form appears at 0.66 s, and only at ~3 s does the newer core take over and run
  lshw -class disk -disable network.
• The cached core in meshagent.db (CoreModule) already contains the new form and not
  the old one, so the cache is not the source either.
• The installed binary is byte-identical (md5) to the published 1.2.1 build.

So the flagless invocation appears to be baked into the boot core inside the released
binary, and it runs during that first window regardless of what the server later pushes.
Updating the agent or refreshing the core does not avoid that window.

Why it is fatal rather than slow: lshw passes an interface name of /dev/vmnet1 to a
network ioctl. The kernel's dev_load() tries to autoload a module twice, and the second
attempt — request_module("%s", name) — only happens if the caller holds
CAP_SYS_MODULE. That attempt runs modprobe -q -- /dev/vmnet1, which blocks forever
inside a third-party network driver (VMware Workstation's vmnet here). lshw is then
stuck in D state, and the agent waits on that child and stops reading messages from
the server.

Measured symptoms:

• lshw -class disk -> hangs, killed at 20 s; lshw -class disk -disable network ->
  rc=0 in 2 s.
• The receive queue on the agent's socket to the server grows and stays there
  (Recv-Q 122) while the machine still shows as online. No error, no log entry,
  no event. Remote control simply never starts.
• The agent's main thread looks healthy in pselect, so thread state is misleading here.

Workaround, with a positive control: denying CAP_SYS_MODULE to the agent's service unit
prevents the second autoload attempt. With it applied, Recv-Q drops to 0, no process is
left in D state, and remote control works again; restoring the capability reproduces the
hang.

The part I think is worth fixing upstream is not lshw itself — it's that a single stuck
child silences the whole message loop, with the machine still reported as online and no
diagnostic anywhere. A bounded wait on the child, or a message loop that does not depend
on the child finishing, would turn this into a slow start instead of a silent, invisible
outage.

Happy to run any further tests — the machine reproduces on demand by restoring the
capability.

[PTR-inc]

This is a fix in meshagent. I already have an update pending which fixes the single child_process.waitExit() issue, which produces the "waitexec() already in progress" error among others