Summary
.env.example does not include CLICKHOUSE_USER or CLICKHOUSE_PASSWORD, and docs/envs.md does not document them either. An operator who sets CLICKHOUSE_PASSWORD (e.g. following an inline docker-compose hint or copying from a peer's config) without also setting CLICKHOUSE_USER will silently break every GraphQL transactions() resolver query while leaving everything else looking healthy.
Took ~1 hour of investigation on a production gateway today to find this; surfacing so others don't repeat it.
Symptoms
GET /ar-io/healthcheck ✅ ok
GET /raw/<id> ✅ works
- GraphQL
blocks(...) ✅ works
- GraphQL
transactions(ids:[...]) or transactions(tags:[...]) ❌ 5s timeout, INTERNAL_SERVER_ERROR
- Core log fills with:
warn: Failed to read ClickHouse max height; skipping boundary optimization Timeout error. (~3-4 per minute)
- CH text_log fills with:
Code: 194. DB::Exception: default: Authentication failed: password is incorrect, or there is no user with such name
clickhouse-auto-import succeeds and continues to import partitions normally (this is the misleading bit — looks like CH is "fine")
Root cause
docker-compose.yaml passes the env through unchanged:
```
- CLICKHOUSE_USER=${CLICKHOUSE_USER:-}
- CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD:-}
- CLICKHOUSE_USER=${CLICKHOUSE_USER:-} # defaults to 'default' in scripts
```
The comment even hints at the asymmetry:
clickhouse-auto-import is a bash script that does \${CLICKHOUSE_USER:-default} shell-default at use site → works when env is empty.
- The core (Node.js,
@clickhouse/client) does NOT apply the same default. It sends an empty/wrong username, CH rejects with code 194, the resolver's pre-flight SELECT MAX(height) FROM transactions times out, and the whole transactions() resolver chain returns a 5s timeout.
Net result: clickhouse-auto-import works (writes to CH), but every GraphQL read against the transactions table fails. Because the symptoms don't include any CH auth message in the core log (auth failures land in CH's own log), this is very hard to diagnose without running SELECT * FROM system.text_log WHERE message LIKE '%Authentication failed%' against the CH container.
Proposed fix
Primary: update .env.example with the connection config and an inline comment about the bash-vs-Node asymmetry:
```bash
==============================================================================
ClickHouse Connection (required when streaming pipeline / GraphQL transactions
resolver is enabled)
==============================================================================
ClickHouse server URL (default 'http://clickhouse:8123' when running with
the bundled docker-compose).
Username. MUST be set explicitly to 'default' (or whatever user matches your
CLICKHOUSE_PASSWORD) even though CH ships with only a 'default' user.
Leaving this empty silently breaks the core's @clickhouse/client auth on
every GraphQL transactions(...) query — auto-import will keep working
because it's a bash script that shell-defaults to 'default' at use site,
but the Node core client does not apply the same fallback.
CLICKHOUSE_USER=default
Password. Set during ClickHouse initialization. Must match the user above.
CLICKHOUSE_PASSWORD=
```
Secondary: add the same to docs/envs.md under a new "## ClickHouse Connection" section, sitting before the existing "## ClickHouse TTL Rules" / "## ClickHouse Auto-Import Daemon" / "## ClickHouse / SQLite GraphQL Boundary" sections.
Tertiary (optional, separate change): make the core's ClickHouse client default username to 'default' when the env var is absent or empty, matching the bash-script behavior. This would make the trap impossible going forward, but it's a behavior change so the .env.example + docs fix is the safer immediate win.
Reproduction (anyone can verify)
```bash
In an .env with CLICKHOUSE_PASSWORD set but CLICKHOUSE_USER absent/empty:
docker compose --profile clickhouse up -d --force-recreate core
Wait for healthcheck, then:
curl -sf -m 10 -X POST http://localhost:4000/graphql
-H 'Content-Type: application/json'
-d '{"query":"{ transactions(first:1) { edges { node { id } } } }"}'
→ 5s timeout, INTERNAL_SERVER_ERROR
docker exec clickhouse-client -q
"SELECT count() FROM system.text_log WHERE event_time > now() - INTERVAL 1 MINUTE AND message LIKE '%Authentication failed%'"
→ non-zero (typically 5-15)
Set CLICKHOUSE_USER=default, recreate core, retry the same curl:
→ sub-second response with data
```
Happy to submit a PR with the .env.example + docs/envs.md change if it's helpful.
Summary
.env.exampledoes not includeCLICKHOUSE_USERorCLICKHOUSE_PASSWORD, anddocs/envs.mddoes not document them either. An operator who setsCLICKHOUSE_PASSWORD(e.g. following an inline docker-compose hint or copying from a peer's config) without also settingCLICKHOUSE_USERwill silently break every GraphQLtransactions()resolver query while leaving everything else looking healthy.Took ~1 hour of investigation on a production gateway today to find this; surfacing so others don't repeat it.
Symptoms
GET /ar-io/healthcheck✅ okGET /raw/<id>✅ worksblocks(...)✅ workstransactions(ids:[...])ortransactions(tags:[...])❌ 5s timeout,INTERNAL_SERVER_ERRORwarn: Failed to read ClickHouse max height; skipping boundary optimization Timeout error.(~3-4 per minute)Code: 194. DB::Exception: default: Authentication failed: password is incorrect, or there is no user with such nameclickhouse-auto-importsucceeds and continues to import partitions normally (this is the misleading bit — looks like CH is "fine")Root cause
docker-compose.yamlpasses the env through unchanged:```
- CLICKHOUSE_USER=${CLICKHOUSE_USER:-}
- CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD:-}
- CLICKHOUSE_USER=${CLICKHOUSE_USER:-} # defaults to 'default' in scripts
```
The comment even hints at the asymmetry:
clickhouse-auto-importis a bash script that does\${CLICKHOUSE_USER:-default}shell-default at use site → works when env is empty.@clickhouse/client) does NOT apply the same default. It sends an empty/wrong username, CH rejects with code 194, the resolver's pre-flightSELECT MAX(height) FROM transactionstimes out, and the wholetransactions()resolver chain returns a 5s timeout.Net result:
clickhouse-auto-importworks (writes to CH), but every GraphQL read against thetransactionstable fails. Because the symptoms don't include any CH auth message in the core log (auth failures land in CH's own log), this is very hard to diagnose without runningSELECT * FROM system.text_log WHERE message LIKE '%Authentication failed%'against the CH container.Proposed fix
Primary: update
.env.examplewith the connection config and an inline comment about the bash-vs-Node asymmetry:```bash
==============================================================================
ClickHouse Connection (required when streaming pipeline / GraphQL transactions
resolver is enabled)
==============================================================================
ClickHouse server URL (default 'http://clickhouse:8123' when running with
the bundled docker-compose).
CLICKHOUSE_URL=http://clickhouse:8123
Username. MUST be set explicitly to 'default' (or whatever user matches your
CLICKHOUSE_PASSWORD) even though CH ships with only a 'default' user.
Leaving this empty silently breaks the core's @clickhouse/client auth on
every GraphQL transactions(...) query — auto-import will keep working
because it's a bash script that shell-defaults to 'default' at use site,
but the Node core client does not apply the same fallback.
CLICKHOUSE_USER=default
Password. Set during ClickHouse initialization. Must match the user above.
CLICKHOUSE_PASSWORD=
```
Secondary: add the same to
docs/envs.mdunder a new "## ClickHouse Connection" section, sitting before the existing "## ClickHouse TTL Rules" / "## ClickHouse Auto-Import Daemon" / "## ClickHouse / SQLite GraphQL Boundary" sections.Tertiary (optional, separate change): make the core's ClickHouse client default
usernameto'default'when the env var is absent or empty, matching the bash-script behavior. This would make the trap impossible going forward, but it's a behavior change so the.env.example+ docs fix is the safer immediate win.Reproduction (anyone can verify)
```bash
In an .env with CLICKHOUSE_PASSWORD set but CLICKHOUSE_USER absent/empty:
docker compose --profile clickhouse up -d --force-recreate core
Wait for healthcheck, then:
curl -sf -m 10 -X POST http://localhost:4000/graphql
-H 'Content-Type: application/json'
-d '{"query":"{ transactions(first:1) { edges { node { id } } } }"}'
→ 5s timeout, INTERNAL_SERVER_ERROR
docker exec clickhouse-client -q
"SELECT count() FROM system.text_log WHERE event_time > now() - INTERVAL 1 MINUTE AND message LIKE '%Authentication failed%'"
→ non-zero (typically 5-15)
Set CLICKHOUSE_USER=default, recreate core, retry the same curl:
→ sub-second response with data
```
Happy to submit a PR with the .env.example + docs/envs.md change if it's helpful.