Correct the completeness signal, and fix the interval drift it exposed

[alexey-milovidov]

Follow-up to #1. Two claims made there were wrong, and measuring properly exposed a scheduling bug.

What was wrong

A single response from /events is an unordered sample of the recent window, not the strict newest 100. Two adjacent responses can therefore barely overlap while their union still covers everything. So:

• the per-poll "all 100 entries are new" check added in Fix the event capture regression: poll all three pages of /events #1 fired constantly on page 1 without any events actually being lost;
• the claim that pushes "cannot be captured completely at any affordable polling rate" came from that same misreading, and is false.

What the aggregate actually shows

Page 1, 20 second windows:

interval	requests/s	events/s captured	overlap	largest timestamp gap
0.5 s	1.80	100.0	46%	1.0 s
2.0 s	0.50	50.0	0%	2.0 s
4.0 s	0.25	25.0	0%	4.0 s

Unique events scale linearly with the length of the window (1100 in 10 s, 3101 in 30 s) and timestamps within the covered period are contiguous, so this is a complete stream at about 100 events/second — and two polls a second capture all of it.

More clients cannot help. Four concurrent workers at 7.7 requests/second found one extra event over a single worker at 1.9 requests/second across the same window, while the already-seen fraction rose from 44% to 87% — the extra requests only re-fetched what was already captured. Extra tokens buy the quota to sustain the poll rate, not extra coverage.

The right signal, and the bug it found

The check is now the overlap between polls, aggregated per page per minute. Healthy is comfortably above zero; a page sitting at 0% for a whole minute means consecutive polls never met and the difference was dropped.

Reporting that immediately turned up a bug in #1: the next poll was scheduled from the moment the response came back, so every interval was silently stretched by a round trip and page 1 ran at 0.85 requests/second instead of 2. It now schedules from the moment the request goes out.

before: 103 polls/minute, overlap p1:0%  p2:21% p3:94%, warning on page 1
after:  193 polls/minute, overlap p1:42% p2:57% p3:97%, no warning

42% matches the 46% measured independently at 1.8 requests/second, so all three pages are now fully covered. Latest run: 10681 events, 16 types, 0 duplicates, 72% push / 28% non-push.

🤖 Generated with Claude Code