Skip to content

Add ClickHouse provider #1

Description

@pietrodicaprio

Context

go-wormhole currently ships SQL, Mongo, Slipstream, and MemDoc providers (docs/05-providers.md). ClickHouse is not in that set, but it is the natural fit for high-volume append-only telemetry workloads (logs, metrics, time-series events, traces) where Code-First migrations and the differ are still desirable.

Adding it brings the Code-First workflow to ClickHouse-targeted projects that today either fall back to hand-written DDL or pick up a separate migration tool for the ClickHouse side only. Both deviate from the "single source of truth in Go struct definitions" promise that motivates go-wormhole.

Scope

Add a ClickHouse provider that satisfies the same Provider interface as the existing SQL provider, with extensions to the schema-migration vocabulary for ClickHouse-specific DDL features.

Driver

Native protocol via ch-go (pure Go, no CGO). clickhouse-go (database/sql) is the fallback if ch-go does not expose a needed surface.

Schema features the differ must emit

Feature Used by Notes
MergeTree engine every telemetry table Default. Other engines (e.g. ReplicatedMergeTree) added on demand.
PARTITION BY <expr> every telemetry table Most common: toDate(timestamp).
ORDER BY (col, col, ...) every telemetry table Mandatory in ClickHouse; the differ must require it for the entity.
DateTime64(precision) columns high-precision timestamps e.g. time.Time -> DateTime64(9) for nanosecond precision.
CODEC(...) column hints timestamp / numeric columns e.g. CODEC(Delta, ZSTD). Field-level tag.
LowCardinality(String) enum-like columns Field-level tag.
TTL <expr> retention Per-table TTL for retention policy enforcement.

Tag surface

Field-level tag extensions:

type Event struct {
    Timestamp time.Time `db:"type:datetime64(9); codec:delta,zstd"`
    Region    string    `db:"type:lowcardinality(string)"`
    Status    string    `db:"type:lowcardinality(string)"`
    // ...
}

Entity-level (registration-side) declarations for engine, partitionBy, orderBy, and ttl, since these are not single-column properties.

Capability matrix entry

To be added in docs/05-providers.md:

Capability ClickHouse
Transactions partial
Partial update
Sorting
Offset pagination
Nested filters
Relation includes
Schema migrations
Schema evolution

The partial-update gap is the most significant divergence from SQL provider behaviour. Suggested approach: the provider rejects Update() calls on ClickHouse-backed entities with a clear error, directing the caller to insert a new row instead. The differ still handles schema changes (add column, drop column, alter codec) because those use ALTER TABLE, which ClickHouse supports synchronously for DDL.

Out of scope (v1)

  • JOIN / Include semantics.
  • UPDATE / DELETE on individual rows (the architectural fit for
    ClickHouse is append-only).
  • Distributed-table support (Distributed engine, replicated engines).
    Single-node ClickHouse only.

Acceptance

  • ClickHouseProvider implements the Provider interface.
  • Field tags codec:, type:lowcardinality(...), type:datetime64(N)
    are recognised and emitted by the differ.
  • Entity-level registration accepts engine, partitionBy, orderBy,
    and ttl declarations.
  • wormhole migrations add <Name> against a ClickHouse-backed entity
    generates a migrations/*.go file with a CreateTableOp carrying
    the engine, partition key, and order key.
  • wormhole database update applies the migration against a real
    ClickHouse instance via the native ch-go driver.
  • docs/05-providers.md updated with the capability matrix row and a
    ClickHouse subsection.
  • Integration test against a ClickHouse container in CI.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions