refactor(demo): run native demo without Docker#4708
Conversation
Replace the five Docker-backed process-compose services with native processes so `just demo-native` needs no Docker daemon: - demo-l1-network: custom geth-l1 image -> native anvil (from foundry) - espresso-node-db-0/1: docker postgres -> scripts/run-postgres - keydb: docker keydb -> native redis-server - telemetry-collector (vector) and block-explorer: dropped flake.nix adds redis and bumps postgresql_16 -> postgresql_18. cleanup-process-compose drops the docker compose teardown and reaps anvil/postgres/redis-server.
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. |
|
Claude finished @sveitser's task in 3m 22s —— View job Review: run native demo without DockerNice cleanup — replacing the Docker-backed services with native processes makes
Findings (all minor / non-blocking)
Nothing here blocks merge. Assuming |
| createdb -h 127.0.0.1 -p "${port}" -U root espresso | ||
|
|
||
| # Double wait: the first returns as soon as the trap fires; the second blocks until |
There was a problem hiding this comment.
Startup-time shutdown can hang this script. The trap is installed before this readiness loop, but if SIGTERM/SIGINT arrives while we're still in the until pg_isready loop, the trap fires kill -INT on postgres and postgres starts shutting down — after which pg_isready will never succeed, so this loop spins forever and the wait lines below are never reached. process-compose then has to SIGKILL on teardown.
This is a narrow window (only during startup, before the DB is ready), so it's minor, but you can make shutdown deterministic by breaking out when a shutdown was requested, e.g. set a flag in the trap and check it:
shutting_down=0
trap 'shutting_down=1; kill -INT "${pg_pid}" 2>/dev/null' TERM INT
until pg_isready -h 127.0.0.1 -p "${port}" >/dev/null 2>&1; do
[[ "${shutting_down}" -eq 1 ]] && break
sleep 0.5
done| OPTIONS: | ||
| -h, --help Show this help |
There was a problem hiding this comment.
Minor doc nit: the usage says DATADIR Existing empty data directory to initialize and use, but the script actually relies on initdb creating the directory (the caller passes ${ESPRESSO_BASE_STORAGE_PATH}/pg0, which doesn't exist yet). Only the parent needs to exist. Consider rewording to "Data directory to initialize (created if missing; must be empty if it exists)" so it's clear the dir need not pre-exist — and that re-running against a populated dir would fail initdb.
- run-postgres: break readiness loop on shutdown signal so a SIGTERM during startup does not spin (pg_isready never succeeds against a shutting-down server); skip createdb when shutting down - run-postgres: correct DATADIR docs (initdb creates the dir) - bind anvil and redis to 127.0.0.1 instead of all interfaces
The native demo previously ran the L1, CDN store, and node DBs as Docker containers, so the test-integration and demo-native jobs only needed process-compose. Now those run natively (anvil, redis-server, postgres), which were absent on the runner and broke every native demo job with 'anvil: command not found'. Add install-native-demo-deps: installs foundry (anvil), redis-server and postgresql, stops the auto-started redis/postgres services so the demo can bind its own ports, and puts the postgres server binaries on PATH.
The Push-CDN broker heartbeat uses KeyDB's EXPIREMEMBER command, which plain redis does not implement, so brokers never registered and the CDN was dead (the demo limped along on the libp2p secondary; the non-PoS->PoS epoch upgrade tests stalled). The standalone cdn-broker/cdn-marshal pick their discovery client at compile time (Redis), so SQLite/embedded discovery is not selectable. KeyDB is not in nixpkgs and has no clean macOS build, so nix/keydb.nix repackages prebuilt binaries: the official keydb-tools .deb relocated via autoPatchelfHook on Linux, and the Homebrew bottle (openssl relocated + re-signed) on macOS aarch64. CI extracts the .deb directly.
autoPatchelfHook failed on the keydb-tools .deb for libbz2.so.1.0, liblz4.so.1, libz.so.1 and libsystemd.so.0, which broke the default nix devShell build and every nix-develop CI job (contracts, etc.). Add the full DT_NEEDED set.
imabdulbasit
left a comment
There was a problem hiding this comment.
I haven't looked through all of the code but since the native demo works, so I'm approving it.
Replace the five Docker-backed process-compose services with native processes so
just demo-nativeneeds no Docker daemon:flake.nix adds redis and bumps postgresql_16 -> postgresql_18. cleanup-process-compose drops the docker compose teardown and reaps anvil/postgres/redis-server.