fix(agent): add jitter to cluster status ticker to prevent thundering…

[himabindugit]

Problem

When agents start together — for example, after a fleet-controller restart or a batch of aircraft powering on in the same window — those agents begin their check-in tickers in sync. Without jitter, every agent in that cohort PATCHes its cluster status at the same instant every 15 minutes.

In steady state, clusters added at different times are naturally spread out. But for clusters that started together (e.g., 50–100 agents from the same restart or power-up window), the synchronised PATCHes arrive as a burst, producing a concentrated spike of watch events on the fleet-controller. This worsens the cache recompilation storm tracked in #5444, since the burst fires the hot path simultaneously across all affected clusters.

Note: The FleetController logs confirm the burst pattern but cannot distinguish cold-start events from synchronized agent PATCHes — Kubernetes audit logs are needed to isolate the agent PATCH timing specifically — investigation ongoing.

Fix

Add a random sleep of [0, checkinInterval) before the periodic ticker loop begins (ticker.go). The sleep uses a select so the goroutine exits cleanly if the context is cancelled during the jitter window rather than blocking on time.After.

Because the offset is random per-agent, the spread is permanent — not just on the first tick:

• Agent A: jitter=3min → fires at 3, 18, 33 ... min
• Agent B: jitter=7min → fires at 7, 22, 37 ... min

Tests

Two new test cases added to ticker_test.go:

• Context cancellation during jitter — verifies the goroutine exits cleanly when the context is cancelled before the jitter window expires, so no goroutine leak occurs.
• Thundering herd spread — starts 5 concurrent agents simultaneously and asserts their first periodic patches are spread across time rather than bunched at t=0.

Existing test (should patch the cluster status after checkinInterval) continues to pass — jitter does not prevent eventual check-in.

Refers to #5444

Additional Information

This fix was developed in collaboration with Claude Code (Anthropic) as part of a broader
investigation into fleet-controller CPU spikes at production scale.

Checklist

• I have updated the documentation via a pull request in the fleet-product-docs repository.

[Copilot]

Pull request overview

This PR reduces synchronized (“thundering herd”) cluster status check-ins from agents by adding a per-agent randomized startup offset before the periodic cluster-status ticker loop begins, helping smooth fleet-controller load during mass agent startups/restarts.

Changes:

• Add a random jitter delay before starting the periodic cluster-status ticker loop.
• Add tests covering context cancellation behavior and multi-agent start-time spreading.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
internal/cmd/agent/clusterstatus/ticker.go	Adds randomized pre-ticker delay to spread periodic status patches over the check-in interval.
internal/cmd/agent/clusterstatus/ticker_test.go	Adds/updates Ginkgo tests for cancellation behavior and multi-agent patch-time spread.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[d3flex]

The new rand.N(checkinInterval) thing only staggers the periodic ticks that come after the startup. If the production incident was a startup burst rather than middle lifecycle drift, this doesn't cover it. is that intentionally out of scope?

[himabindugit]

The new rand.N(checkinInterval) thing only staggers the periodic ticks that come after the startup. If the production incident was a startup burst rather than middle lifecycle drift, this doesn't cover it. is that intentionally out of scope?

Good catch — you're right, the startup burst was not covered.

More details:
Fixed: the startup check-in goroutine now sleeps ClusterRegisterDelay + rand.N(checkinInterval) instead of a fixed ClusterRegisterDelay. This spreads initial check-ins across the same interval window as periodic ticks, covering the scenario where many agents restart simultaneously (e.g. after a fleet-controller recovery). Context cancellation is also handled properly during the extended delay.

[d3flex]

Same as #5445 (review). commit history needs some work, if you do not mind

[himabindugit]

Same as #5445 (review). commit history needs some work, if you do not mind

Done — consolidated into 2 logical commits with bodies and signatures:

• fix(agent): add jitter to cluster status ticker to prevent thundering herd — periodic ticker jitter + test fixes
• fix(agent): add startup jitter to spread initial check-ins on agent restart — startup goroutine jitter covering the mass-restart scenario