Skip to content

docs: .env.example missing CLICKHOUSE_USER causes silent GraphQL transactions() timeout #781

Description

@vilenarios

Summary

.env.example does not include CLICKHOUSE_USER or CLICKHOUSE_PASSWORD, and docs/envs.md does not document them either. An operator who sets CLICKHOUSE_PASSWORD (e.g. following an inline docker-compose hint or copying from a peer's config) without also setting CLICKHOUSE_USER will silently break every GraphQL transactions() resolver query while leaving everything else looking healthy.

Took ~1 hour of investigation on a production gateway today to find this; surfacing so others don't repeat it.

Symptoms

  • GET /ar-io/healthcheck ✅ ok
  • GET /raw/<id> ✅ works
  • GraphQL blocks(...) ✅ works
  • GraphQL transactions(ids:[...]) or transactions(tags:[...])5s timeout, INTERNAL_SERVER_ERROR
  • Core log fills with: warn: Failed to read ClickHouse max height; skipping boundary optimization Timeout error. (~3-4 per minute)
  • CH text_log fills with: Code: 194. DB::Exception: default: Authentication failed: password is incorrect, or there is no user with such name
  • clickhouse-auto-import succeeds and continues to import partitions normally (this is the misleading bit — looks like CH is "fine")

Root cause

docker-compose.yaml passes the env through unchanged:
```
- CLICKHOUSE_USER=${CLICKHOUSE_USER:-}
- CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD:-}
- CLICKHOUSE_USER=${CLICKHOUSE_USER:-} # defaults to 'default' in scripts
```

The comment even hints at the asymmetry:

  • clickhouse-auto-import is a bash script that does \${CLICKHOUSE_USER:-default} shell-default at use site → works when env is empty.
  • The core (Node.js, @clickhouse/client) does NOT apply the same default. It sends an empty/wrong username, CH rejects with code 194, the resolver's pre-flight SELECT MAX(height) FROM transactions times out, and the whole transactions() resolver chain returns a 5s timeout.

Net result: clickhouse-auto-import works (writes to CH), but every GraphQL read against the transactions table fails. Because the symptoms don't include any CH auth message in the core log (auth failures land in CH's own log), this is very hard to diagnose without running SELECT * FROM system.text_log WHERE message LIKE '%Authentication failed%' against the CH container.

Proposed fix

Primary: update .env.example with the connection config and an inline comment about the bash-vs-Node asymmetry:

```bash

==============================================================================

ClickHouse Connection (required when streaming pipeline / GraphQL transactions

resolver is enabled)

==============================================================================

ClickHouse server URL (default 'http://clickhouse:8123' when running with

the bundled docker-compose).

CLICKHOUSE_URL=http://clickhouse:8123

Username. MUST be set explicitly to 'default' (or whatever user matches your

CLICKHOUSE_PASSWORD) even though CH ships with only a 'default' user.

Leaving this empty silently breaks the core's @clickhouse/client auth on

every GraphQL transactions(...) query — auto-import will keep working

because it's a bash script that shell-defaults to 'default' at use site,

but the Node core client does not apply the same fallback.

CLICKHOUSE_USER=default

Password. Set during ClickHouse initialization. Must match the user above.

CLICKHOUSE_PASSWORD=

```

Secondary: add the same to docs/envs.md under a new "## ClickHouse Connection" section, sitting before the existing "## ClickHouse TTL Rules" / "## ClickHouse Auto-Import Daemon" / "## ClickHouse / SQLite GraphQL Boundary" sections.

Tertiary (optional, separate change): make the core's ClickHouse client default username to 'default' when the env var is absent or empty, matching the bash-script behavior. This would make the trap impossible going forward, but it's a behavior change so the .env.example + docs fix is the safer immediate win.

Reproduction (anyone can verify)

```bash

In an .env with CLICKHOUSE_PASSWORD set but CLICKHOUSE_USER absent/empty:

docker compose --profile clickhouse up -d --force-recreate core

Wait for healthcheck, then:

curl -sf -m 10 -X POST http://localhost:4000/graphql
-H 'Content-Type: application/json'
-d '{"query":"{ transactions(first:1) { edges { node { id } } } }"}'

→ 5s timeout, INTERNAL_SERVER_ERROR

docker exec clickhouse-client -q
"SELECT count() FROM system.text_log WHERE event_time > now() - INTERVAL 1 MINUTE AND message LIKE '%Authentication failed%'"

→ non-zero (typically 5-15)

Set CLICKHOUSE_USER=default, recreate core, retry the same curl:

→ sub-second response with data

```

Happy to submit a PR with the .env.example + docs/envs.md change if it's helpful.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions