Fix the event capture regression: poll all three pages of /events

[alexey-milovidov]

The crawler polled only page 1 of the Events API. GitHub does not spread event types evenly over the pages: page 1 carries the push / create / delete stream, and everything else — pull requests, issues, comments, stars, reviews, releases, forks — is only reachable on pages 2 and 3. Reading page 1 alone captured pushes and silently dropped nearly everything else.

This is the degradation reported upstream in igrigorik/gharchive.org#310 and #320.

Evidence

Of 4654 unique events observed on the live API over 78 seconds, grouped by the page each first appeared on:

type	total	page 1	page 2	page 3
PushEvent	1996	1995	1	0
PullRequestEvent	944	2	747	195
IssueCommentEvent	434	1	346	87
IssuesEvent	382	1	299	82
WatchEvent	225	0	179	46
PullRequestReviewEvent	211	0	160	51
others	154	0	122	32

Before and after, against the live API

Previous code, 20 seconds — 300 events, 4 types, no stars, no pull requests, no comments:

283 PushEvent   13 CreateEvent   3 DeleteEvent   1 IssuesEvent

This branch, 2 minutes — 16248 events, 16 types, zero duplicates, 67% push / 33% non-push:

10894 PushEvent          436 PullRequestReviewEvent         47 ForkEvent
 2054 PullRequestEvent   385 CreateEvent                    33 PublicEvent
  868 IssueCommentEvent  349 PullRequestReviewCommentEvent  21 CommitCommentEvent
  583 IssuesEvent        328 WatchEvent                     15 MemberEvent
                         118 ReleaseEvent                    4 DiscussionEvent
                         110 DeleteEvent                      3 GollumEvent

Star capture goes from about 0.03 to 2.7 events/second, which matches what the API actually serves.

Changes

• Poll pages 1..3, each on its own schedule, so a busy page 1 cannot delay the others. Page 4 is not fetched — the API refuses it with pagination is limited for this resource, so 300 events per poll is the ceiling.
• De-duplicate against a bounded set of recently seen event ids. The previous code compared only against the immediately preceding response, so an event that briefly dropped out of the window was archived twice.
• Restore the missed-records alarm. It tested new_events.size >= PAGE_LIMIT, and f1f4200 set PAGE_LIMIT to 500 while the API clamps per_page to 100 — so the condition could never be true and the crawler ran saturated and silent for 15 months. It is now per page, and rate-limited to one line a minute because page 1 is expected to saturate.
• Accept several tokens separated by commas, used round-robin. The default polling rate costs 14400 requests/hour, more than the 5000/hour of one user token; a GitHub App installation token (15000/hour) also works.
• Back off when the remaining quota runs behind the window, and honour Retry-After on 403/429 instead of hammering through a throttle.
• Conditional GETs with per-page ETags — from #317 by @emalaj-seq.
• Report to StatHat only when STATHATKEY is set, instead of raising on every cycle.
• Fix the git:// remotes in the Gemfile, also from #317 — GitHub disabled that protocol in 2022, so bundle install could not resolve them.

Known limitation

Page 1 alone runs at over 190 events/second — 100 entries replaced in under 0.53 s — which is faster than the API will serve to a single client. Pushes therefore cannot be captured completely at any affordable polling rate; pages 2 and 3 can. The crawler now reports when this happens rather than hiding it.

🤖 Generated with Claude Code