Skip to content

Rotate ENS lookups across free RPCs, refreshed on a cron into KV - #21

Merged
frolic merged 5 commits into
mainfrom
rpc-pool
Jul 15, 2026
Merged

Rotate ENS lookups across free RPCs, refreshed on a cron into KV#21
frolic merged 5 commits into
mainfrom
rpc-pool

Conversation

@frolic

@frolic frolic commented Jul 14, 2026

Copy link
Copy Markdown
Owner

Brings #10 (RPC rotation) and #11 (chainlist-backed healthy list) forward onto the v2 monorepo API worker, rebuilt around a cron + KV.

How it works

  • ethereumTransport — viem fallback across the free RPCs in random order, paid endpoint (ETHEREUM_RPC_URL) last. viem retries each transport zero times and advances on any non-user error, so a 429 rolls over transparently — keeping paid usage, and cost, minimal. Shuffling spreads load (a stateless Worker can't round-robin).
  • Cron (0 * * * *) — health-checks the chainlist candidates and writes the survivors to KV, once for the whole fleet.
  • Request path — one KV read:
    const healthy = (await env.RPCS.get<string[]>(RPCS_KEY, "json")) ?? [];
    transport: ethereumTransport(healthy, env.ETHEREUM_RPC_URL)
    A cold/empty list just means straight to the paid RPC — the transport already handles that.

Why cron + KV rather than refreshing on the request path

caches.default is per-colo. Refreshing lazily meant every data center re-ran the full 41-endpoint health-check pass hourly — roughly 2k check requests/hour against the free RPCs the rotation exists to lean on, which is how you get rate-limited by them. The cron does ~41/hour, once. And because a cron runs in a single colo, the list has to live somewhere global — hence KV, not the Cache API.

What that deleted

Falling back to the paid RPC on a cold KV removed the seed list, and with it most of the machinery:

  • rpcUrls.ts (7-endpoint seed list), getHealthyRpcs.ts (cache + refresh + staleness), scripts/verify-rpcs.ts, and a dead alchemy.run esbuild stub in the tests.
  • Gone as concepts: waitUntil, the Age check, the refreshing dedupe flag, and the two competing TTLs.

Verified on a preview stage

  • Cron registered: ['0 * * * *']; per-stage KV namespace created.
  • A triggered refresh wrote the health-checked list to KV (nodereal, publicnode, mevblocker, regional blxrbdn, …).
  • Cold KV → resolves via the paid RPC. Warm → resolves via the KV list.
  • Typecheck clean; 15/15 tests, including the cold-KV→paid path in both the transport unit test and the miniflare integration test (which now runs with an empty KV).

Note: the deploy token needed Workers KV Storage: Edit added.

Supersedes #10 and #11.

🤖 Generated with Claude Code

Brings #10 (rotation) and #11 (chainlist pool + /rpcs) forward onto the v2
monorepo API worker.

- ethereumTransport: viem `fallback` across the free RPCs in random order,
  with the paid endpoint (ETHEREUM_RPC_URL) last. viem advances on any
  non-user error, so a 429 rolls over transparently — keeping paid usage,
  and cost, to a minimum. Shuffling spreads load (a stateless Worker can't
  round-robin).
- getRpcPool/getHealthyRpcs: the known-good pool, cached in the Cache API and
  refreshed in the background via waitUntil (chainlist candidates + the
  committed seed list, health-checked). Cold/stale cache returns the seed
  immediately, so requests never block on a health-check pass.
- GET /rpcs exposes the pool (and rides the worker's response cache).
- The resolver now builds its client on the rotating transport.

Verified on a preview stage: /rpcs cold returns the 7 seeds, then the
background pass checked 41 candidates and kept 16 healthy; resolution works
through the pool. 15/15 tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions
github-actions Bot temporarily deployed to preview/pr-21/site July 14, 2026 22:47 Destroyed
@github-actions
github-actions Bot temporarily deployed to preview/pr-21/api July 14, 2026 22:47 Destroyed
The pool tracked its own freshness (generatedAt + a 5min FRESH_MS check) on
top of a cache entry with max-age=86400 — two competing TTLs, with the
hand-rolled one doing the real work. Collapse it to one signal: cache the
health-checked list with a max-age, and let `cache.match` missing be what
triggers a refresh. Drops the Pool type, generatedAt/checked metadata, and
the staleness comparison.

The /rpcs endpoint existed only to get the list cached; the cached method
does that directly, without a subrequest per resolve, so the endpoint goes
too. A worker *can* fetch its own hostname, but it costs a subrequest, adds
latency, and re-enters the worker on a miss — the Cache API gives the same
caching in-process.

Behavior note: on expiry the next request now gets the committed seed list
(not the previous list) while the refresh runs. The seeds are verified-good
and the transport falls through anyway.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@frolic frolic changed the title Rotate ENS lookups across a self-refreshing pool of free RPCs (+ /rpcs) Rotate ENS lookups across a self-refreshing pool of free RPCs Jul 15, 2026
@github-actions
github-actions Bot temporarily deployed to preview/pr-21/site July 15, 2026 08:36 Destroyed
@github-actions
github-actions Bot temporarily deployed to preview/pr-21/api July 15, 2026 08:36 Destroyed
Expiring the cache entry at the freshness threshold meant every request in
the refresh window fell back to the 7 committed seeds — a narrower pool, so
more 429 fallthrough and more paid RPC hits, once an hour.

Cloudflare's Cache API ignores `stale-while-revalidate` (measured: an entry
is dropped the moment max-age passes) but does stamp `Age` on every hit. So
keep the entry alive for 24h and use the cache's own Age to decide when to
revalidate: past an hour we refresh in the background and still serve the
cached list. Requests now only see the seed list on a genuinely cold cache,
and never wait on a health-check pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions
github-actions Bot temporarily deployed to preview/pr-21/site July 15, 2026 08:45 Destroyed
@github-actions
github-actions Bot temporarily deployed to preview/pr-21/api July 15, 2026 08:45 Destroyed
The pool was refreshed lazily on the request path into the Cache API, which
is per-colo: every data center re-ran the whole 41-endpoint health-check pass
hourly (~2k check requests/hour) — hammering the free RPCs the rotation
exists to lean on. A cron does the pass once for the fleet (~41/hour), but a
cron runs in one colo, so the list has to live somewhere global: KV.

Falling back to the paid endpoint on a cold KV removes the seed list, and
with it most of the machinery. The request path is now a KV read; the
transport already routes an empty free list straight to the paid RPC.

Deleted: rpcUrls.ts (seed list), getHealthyRpcs.ts (cache/refresh/staleness
logic), scripts/verify-rpcs.ts, and a dead alchemy.run esbuild stub in the
tests. Gone with them: waitUntil, the Age check, the refreshing flag, and the
two TTLs.

Verified on a preview stage: cron registered (0 * * * *), per-stage KV
namespace created, a triggered refresh wrote the health-checked list to KV,
cold KV resolves via the paid RPC, and warm resolves use the list.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@frolic frolic changed the title Rotate ENS lookups across a self-refreshing pool of free RPCs Rotate ENS lookups across free RPCs, refreshed on a cron into KV Jul 15, 2026
@github-actions
github-actions Bot temporarily deployed to preview/pr-21/site July 15, 2026 09:09 Destroyed
@github-actions
github-actions Bot temporarily deployed to preview/pr-21/api July 15, 2026 09:09 Destroyed
The Env interface duplicated the binding list by hand — adding RPCS meant
editing the config and the interface. Alchemy infers it: hoist the api
worker's definition to module scope so there's a `typeof` to feed
Cloudflare.InferEnv, and derive Env from that.

Verified the inference is real, not vacuous: a non-existent binding and a
wrong-typed binding both fail typecheck, while ETHEREUM_RPC_URL (string),
RATE_LIMITER and RPCS resolve to their proper types. The type import erases —
the worker bundle contains no config code — so no esbuild stub is needed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions
github-actions Bot temporarily deployed to preview/pr-21/site July 15, 2026 09:17 Destroyed
@github-actions
github-actions Bot temporarily deployed to preview/pr-21/api July 15, 2026 09:18 Destroyed
@frolic
frolic merged commit 93fcca2 into main Jul 15, 2026
2 checks passed
@frolic
frolic deleted the rpc-pool branch July 15, 2026 09:26
frolic added a commit that referenced this pull request Jul 15, 2026
**Draft — this is the live API cutover.** `api.ensideas.com` serves
millions of hits/month. The domain move happens by script (below);
**merging is the last step, not the trigger.**

## Scope
Only **`api.ensideas.com`** moves. `api.instantens.com` is left alone on
`instant-ens-api` — it's unused (abandoned migration prototype), so
there's no reason to put it in the blast radius.

## What this changes
- The **production** api worker gets `domain: ["api.ensideas.com"]`
(previews stay on workers.dev).
- Adds `scripts/point-api-domains.sh` — the cutover *and* its rollback,
as one symmetric command.

## Why a runbook (merging alone won't work)
`api.ensideas.com` is a **worker custom domain on the live
`instant-ens-api`** (managed from the separate `instant-ens-alchemy`
repo). Alchemy's attach **hard-fails** here — it checks `listDomains`
and dies with *"already attached to Worker X … detach it first"*.
`adopt: true` does **not** cover it. It must be detached first.

## Pre-flight (verified ✅)
- `ens-ideas-api-production` is **already deployed and serving**,
running current `main` (#21): cron `0 * * * *` registered, KV-backed RPC
rotation live. The cutover is *only* a domain move — no deploy in the
critical path.
- **The site does not call the API** (only a doc example URL in
`about.tsx`) — blast radius is external API consumers only.
- `instant-ens-api` is intact and untouched (last modified 07-04) — the
rollback target exists, and it still owns `api.instantens.com`.

## Cutover
```bash
./scripts/point-api-domains.sh ens-ideas-api-production
```
Detaches from the old worker and immediately re-attaches to the new one
— that gap is the only downtime (**seconds**). Refuses to detach unless
the target worker exists, skips if already on target, prints ownership +
a live check at the end.

Then, **only once verified**, merge this PR so `main`'s config carries
the domain — otherwise the next production deploy would *detach* it.

---

# Rollback plan

### Roll back if, after the cutover script runs:
- `api.ensideas.com` doesn't return **200** on
`/ens/resolve/vitalik.eth`, or
- responses are wrong/malformed, or 429s appear at normal traffic, or
- error rate climbs on the Workers dashboard for
`ens-ideas-api-production`.

### How (one command, ~seconds)
```bash
./scripts/point-api-domains.sh instant-ens-api
```
Same script, old target. `instant-ens-api` is never modified or deleted
by any of this — it stays deployed and ready the whole time, so rollback
is just moving the hostname back.

**This exact command has already been run against production** (as a
no-op, while the hostname was still on `instant-ens-api`) to prove its
lookup, target check and verification work.

### ⚠️ The merge is the point of no *easy* return
| When | Rollback |
|---|---|
| **Before merging** (recommended window) | Run the script with
`instant-ens-api`. Done — config never changed. |
| **After merging** | Run the script **and revert this PR**. Otherwise
the next production deploy re-attaches the domain and silently
re-cuts-over. |

So: cut over, verify, sit on it, *then* merge.

### Verify the rollback
The script prints ownership and hits the hostname. Ownership is the
definitive check — both workers return identical JSON:
```
api.ensideas.com -> instant-ens-api
```

### Do not, while this is in flight
- Delete or redeploy `instant-ens-api` (it's the rollback target).
- Deploy the **`instant-ens-alchemy`** repo — its config still lists
`api.ensideas.com` and would fight for it.

## ⚠️ Required follow-up (after the cutover sticks)
Remove **`api.ensideas.com`** from `instant-ens-alchemy`'s config so a
stray deploy can't steal it back. It keeps `api.instantens.com`, so that
repo/worker still has a valid config.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant