Seed node is stale

[georgeee]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/MinaProtocol/mina/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report. Please post those things in GitHub Discussions: https://github.com/MinaProtocol/mina/discussions

Description

Running a load testing experiment (72+ hours) triggered a previously unobserved condition in the seed-1 and plain-2 nodes.

Both nodes became frozen at some point, with mina process running, but not responding. Because the process is running, automatic restart is not triggered. Logs are not written. Appears as if it was a deadlock.

Steps to Reproduce

Not sure what are the easier steps, but running a 5-day experiment series with the following configuration triggered the issue:

[
  {
    "infra_rev": "georgeee/mesa",
    "release_tag": "3.3.0-martyall-martyall-inc-rxs-per-zkapp-develop-9f82ebf-bullseye-devnet",
    "description": "Mesa: smoke testing latest pre-RC"
  },
  {
    "base_tps": 0.02,
    "stress_tps": 0.03,
    "zkapp_ratio": 0.5,
    "rounds": 1,
    "round_duration_min": 30,
    "pause_min": 0,
    "max_cost": false,
    "zkapp_soft_limit": 12
  },
  {
    "max_cost_mixed_tps_ratio": 0.5,
    "base_tps": 0.05,
    "stress_tps": 0.06,
    "zkapp_ratio": 0.5,
    "rounds": 2,
    "round_duration_min": 55,
    "pause_min": 5,
    "fees": {
      "min_zkapp": 4000000000,
      "max_zkapp": 5000000000
    },
    "zkapp_soft_limit": 12
  },
  {
    "base_tps": 0.65,
    "stress_tps": 0.65,
    "zkapp_ratio": 0.1,
    "max_cost": true,
    "rounds": 1,
    "round_duration_min": 55,
    "pause_min": 5,
    "min_stop_ratio": 0.1,
    "max_stop_ratio": 0.3,
    "stop_clean_ratio": 0.5,
    "stops_per_round": 4,
    "fees": {
      "min_zkapp": 4000000000,
      "max_zkapp": 5000000000
    },
    "zkapp_soft_limit": 12
  }
]

Expected Result

No deadlock behavior is expected.

Actual Result

Nodes are in a deadlock.

After seed-1 was killed with kill -9, it successfully restarted.

Daemon version

9f82ebf

Tag: 3.3.0-martyall-martyall-inc-rxs-per-zkapp-develop-9f82ebf-bullseye-devnet

Runtime config: https://github.com/o1-labs/gitops-infrastructure/blob/ee7aa533951657e609e215185fbb9781375ade03/platform/hetzner-rivendell-1/applications/mina-standard-itn/genesis-config.json

How frequently do you see this issue?

Frequently

What is the impact of this issue on your ability to run a node?

Blocker

Status

Can't be obtained due to deadlock

Additional information

• Seed-1 config directory dump: https://drive.google.com/file/d/1ny01yb5M8e7w7vuzkKbtDA-4rRMNsp-V/view?usp=sharing
• Plain-2 config directory dump: https://drive.google.com/file/d/1dYJf7AFV7qFyU1JdH9OwObmr1-qo0WX6/view?usp=sharing

[georgeee]

Stale node doesn't seem to respond to kill 16 signal. Only shuts down after kill -9.

[cjjdespres]

I've been examining the stuck node that's still running and had claude write up this one possibility for the bug. It doesn't explain why it happened twice in the cluster testing without being observed before. (Maybe we changed something that made the bug more likely to trigger?)

Root Cause: glibc pthread_cond_signal Lost Wakeup (Bug #25847)

Summary

A Mina daemon running on Debian bullseye (glibc 2.31) froze permanently due to glibc bug #25847, a known bug where pthread_cond_signal() silently fails to wake a waiting thread. This caused a deadlock in OCaml's master lock, permanently halting the Async scheduler. The daemon remained alive but completely unresponsive for 6+ days until we investigated.

This bug affects all Mina deployments running on glibc 2.27 through 2.40, which includes Debian bullseye, Debian bookworm, and Ubuntu 18.04 through 24.04. The fix (glibc 2.41) was released in January 2025 but has not been backported to any of these distributions' stable repositories.

The Deadlock

We attached gdb to the frozen daemon (which had been unresponsive for ~34 seconds before attachment) and found the following state:

All three OCaml threads that need the master lock are stuck in pthread_cond_wait:

Thread	Role	State
Thread 1 (scheduler)	Async event loop	Stuck in st_thread_yield → pthread_cond_wait
Thread 2 (worker)	Thread pool	Stuck in st_masterlock_acquire → pthread_cond_wait
Thread 4 (worker)	Thread pool	Stuck in st_masterlock_acquire → pthread_cond_wait
Thread 3 (tick)	Preemption ticker	In select() — does not interact with the master lock condvar

The master lock structure:

busy    = 0   (lock is FREE — nobody holds it)
waiters = 3   (all three threads are waiting)

This is a classic lost wakeup: the condition all threads are waiting for (busy == 0) is already satisfied, but they are stuck in pthread_cond_wait and need a signal to wake up and check it. No thread will send that signal because no thread is running.

We confirmed with breakpoints that the scheduler never reaches epoll_wait (the I/O poll), never processes events, and never runs Async jobs. The entire daemon is frozen at Thread.yield() in the scheduler loop.

The glibc Bug

glibc 2.27 (released 2018) introduced a new condvar implementation using a two-group waiter management scheme. This implementation has a bug (sourceware #25847) where pthread_cond_signal() can silently fail to wake any thread.

Root cause mechanism: When a waiter thread consumes a signal and is then preempted before reading __g1_start, multiple group rotations can occur. When the thread resumes, it sees a new __g1_start, incorrectly detects "signal stealing", and posts a replacement signal to the current group without properly updating __g_size. This creates a mismatch in the condvar's internal bookkeeping. Eventually, a signal is posted to a group with no actual futex waiters, so no thread wakes up.

Affected versions: glibc 2.27 through 2.40. Fixed in glibc 2.41 (January 2025).

Our container: Debian GLIBC 2.31-13+deb11u13 (the latest bullseye security update). No backport of the fix exists.

Why This Matches

The OCaml runtime's st_thread_yield (in otherlibs/systhreads/st_posix.h) uses pthread_cond_signal and pthread_cond_wait to implement cooperative thread scheduling:

// Simplified from st_thread_yield in OCaml 4.14.2
pthread_mutex_lock(&m->lock);
m->busy = 0;
pthread_cond_signal(&m->is_free);   // Wake a competitor
m->waiters++;
do {
    pthread_cond_wait(&m->is_free, &m->lock);  // Wait to be re-woken
} while (m->busy);
m->busy = 1;
m->waiters--;
pthread_mutex_unlock(&m->lock);

When glibc #25847 causes the pthread_cond_signal to silently fail, the competitor thread never wakes up, and the yielding thread enters pthread_cond_wait expecting to be woken later. If no thread is running to send the next signal, all threads become permanently stuck.

This has been independently confirmed by the OCaml community:

• OCaml Discuss thread: A user reported the exact same deadlock pattern in an OCaml Async application — multiple threads stuck on the master lock condvar. Xavier Leroy (OCaml maintainer) identified the cause as pthread_cond_signal being silently ignored. The reporter confirmed:
  • Deadlocks occurred on Ubuntu 18.04 (glibc 2.27) and 20.04 (glibc 2.31) but not on Ubuntu 16.04 (glibc 2.23, which predates the buggy condvar rewrite)
  • A one-line glibc patch eliminated the deadlock after 1+ days of stress testing
  • Unpatched systems deadlocked within hours under load
• Debian bug #986724 explicitly lists the OCaml runtime as an affected program
• Ubuntu bugs #1888857 and #1899800 document the same pattern

The OCaml code itself is correct per POSIX. The do-while pattern was deliberately introduced in PR #2112 to fix yield fairness and remains unchanged in OCaml 5.x trunk. The bug is entirely in glibc's violation of the POSIX guarantee that pthread_cond_signal will wake at least one waiting thread.

Impact on Mina

• Affected deployments: Any Mina daemon running on glibc 2.27-2.40. This includes all current Debian bullseye and bookworm container images.
• Trigger: Normal condvar contention during Thread.yield() in the Async scheduler loop. No special workload or error condition is required — the bug is probabilistic and timing-dependent.
• Symptoms: The daemon stops responding instantly — no gradual degradation, no error logs, no warning signs. GraphQL becomes unresponsive, peers can't connect, blocks stop being produced/validated.
• Non-recovery: There is no self-healing mechanism. The tick thread (the only other running thread) doesn't interact with the master lock condvar. The daemon stays frozen forever.
• k8s interaction: If the liveness probe checks the libp2p port (which libp2p_helper keeps open independently), the pod is never restarted. In our case, the pod ran frozen for 6+ days.

Affected Distributions

Distribution	glibc	Affected?
Debian 11 (bullseye)	2.31	Yes (our case)
Debian 12 (bookworm)	2.36	Yes
Ubuntu 18.04	2.27	Yes
Ubuntu 20.04	2.31	Yes (confirmed by OCaml community)
Ubuntu 22.04	2.35	Yes
Ubuntu 24.04	2.39	Yes
Any distro with glibc 2.41+	2.41+	No (fixed)
Ubuntu 16.04	2.23	No (predates buggy rewrite)

Recommended Mitigations

1. Upgrade the container base image to a distro shipping glibc 2.41+. This is the real fix.

2. Fix the k8s liveness probe to check the daemon's GraphQL port (3085) rather than the libp2p external port (10005). This won't prevent the deadlock but will ensure affected pods are restarted promptly.

3. Consider a watchdog thread that periodically calls pthread_cond_broadcast on the OCaml master lock condvar. This would limit the impact to at most one watchdog interval of downtime. (Note: there is a second glibc bug where even pthread_cond_broadcast can fail to repair corrupted condvar state, so this is not 100% reliable.)

References

• glibc bug #25847 — the root cause bug
• Debian bug #986724 — Debian tracker
• Ubuntu bug #1899800 — Ubuntu tracker
• OCaml Discuss thread — OCaml community confirmation
• OCaml PR #2112 — the commit introducing the do-while yield pattern
• "Finding the Second Bug in glibc's Condition Variable" — analysis of the secondary broadcast failure mode
• OCaml source: otherlibs/systhreads/st_posix.h (4.14 branch) — st_thread_yield, st_masterlock_acquire