Skip to content

fix(anomaly-detector): replace busy-wait nonce-db lock with async wait and stale-lock recovery - #357

Open
Yinklekay wants to merge 2 commits into
Vero-protocol:mainfrom
Yinklekay:fix/issue--async-nonce-db-lock
Open

fix(anomaly-detector): replace busy-wait nonce-db lock with async wait and stale-lock recovery#357
Yinklekay wants to merge 2 commits into
Vero-protocol:mainfrom
Yinklekay:fix/issue--async-nonce-db-lock

Conversation

@Yinklekay

Copy link
Copy Markdown

📌 Summary

Replaces the busy-wait nonce-db lock with an async wait and adds stale-lock recovery (Issue #346).

Problem: acquireNonceDbLock retried fs.mkdirSync in a synchronous spin loop that pegged a core and blocked the Node event loop for up to the full 5000ms timeout on every contended persist. Since saveNonces() runs on every poll cycle, contention stalled all anomaly detection and every outstanding HTTP request. There was also no stale-lock recovery: a process dying between mkdirSync and rmdirSync left a permanent .lock directory, so every subsequent persistNonceMap threw after 5s and nonce state silently stopped being persisted while the detector kept reporting normally.

Changes:

  • acquireNonceDbLock is now async and yields between retries via a setTimeout promise — no more busy-wait, the event loop stays responsive under contention.
  • Each lock writes an owner.json containing the owning PID and an acquisition timestamp. Locks older than NONCE_DB_LOCK_MAX_AGE_MS (default 30s) are reclaimed automatically; orphaned locks without owner metadata fall back to the directory mtime.
  • Persistence failures now surface as a NONCE_DB_PERSIST_FAILURE CRITICAL alert (console + Guardian dashboard dispatch + returned from runOnce) instead of being swallowed by the poll loop's generic catch.
  • New env knobs: NONCE_DB_LOCK_TIMEOUT_MS (default 5000), NONCE_DB_LOCK_RETRY_MS (default 50), NONCE_DB_LOCK_MAX_AGE_MS (default 30000). Documented in anomaly-detector/.env.example.

🔗 Related Issue

Closes #346

🧪 Testing Completed

  • Unit tests pass: npm test in anomaly-detector → 27 passed across 3 suites
  • Local build / verification successful: npx tsc --noEmit and npm run build are clean
  • New tests cover each acceptance criterion:
    • contended lock does not block the event loop (a concurrent 100ms timer fires on schedule while acquisition is retrying)
    • a pre-existing lock older than the max age is reclaimed and the write succeeds
    • an orphaned lock (no owner metadata) is reclaimed via mtime
    • a fresh lock held by another process is not reclaimed (times out as expected)
    • a persistence failure produces a visible NONCE_DB_PERSIST_FAILURE alert

📋 Contributor Checklist

  • My code follows the repository's style and contribution guidelines
  • I have linked the originating issue above using Closes #<number>
  • I have updated documentation if applicable

…t and stale-lock recovery

The synchronous mkdir spin in acquireNonceDbLock pegged a core and blocked
the event loop for up to the full 5s timeout on every contended persist,
stalling anomaly detection and HTTP handling. It also left orphaned .lock
directories behind if a process died mid-write, silently halting nonce
persistence forever.

- acquireNonceDbLock is now async and yields between retries via a
  setTimeout promise instead of busy-waiting
- locks record the owning PID and acquisition timestamp in owner.json;
  locks older than NONCE_DB_LOCK_MAX_AGE_MS (default 30s) are reclaimed,
  with an mtime fallback for orphaned locks without owner metadata
- persistence failures now surface as a NONCE_DB_PERSIST_FAILURE alert
  (console + dashboard dispatch + runOnce return) instead of being
  swallowed by the poll loop's generic catch
- new env knobs: NONCE_DB_LOCK_TIMEOUT_MS, NONCE_DB_LOCK_RETRY_MS,
  NONCE_DB_LOCK_MAX_AGE_MS

Closes Vero-protocol#346

Generated with Codebuff 🤖
Co-Authored-By: Codebuff <noreply@codebuff.com>
@Yinklekay
Yinklekay requested a review from N-thnI as a code owner August 24, 2026 11:59
…ero-protocol#346)

Generated with Codebuff 🤖
Co-Authored-By: Codebuff <noreply@codebuff.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace the busy-wait nonce-db lock with an async wait and stale-lock recovery

1 participant