Document from_clickhouse

src/content/docs/guides/collecting/index.mdx

@@ -69,6 +69,17 @@ this = string(message).parse_json()
 See the [message broker guide](/guides/collecting/read-from-message-brokers)
 for broker-specific configurations.
 
+### Data stores
+
+Query external data stores such as MySQL and ClickHouse:
+
+```tql
+from_clickhouse sql="SELECT * FROM events WHERE severity >= 3", tls=false
+```
+
+See the [data store guide](/guides/collecting/read-from-data-stores) for table
+reads, SQL pushdown, metadata inspection, and live MySQL polling.
+
 ### Network data
 
 Receive data over TCP or UDP sockets, or capture packets from network
@@ -98,5 +109,6 @@ connections.
 - <Guide>collecting/read-and-watch-files</Guide>
 - <Guide>collecting/fetch-via-http-and-apis</Guide>
 - <Guide>collecting/read-from-message-brokers</Guide>
+- <Guide>collecting/read-from-data-stores</Guide>
 - <Guide>collecting/get-data-from-the-network</Guide>
 - <Guide>routing/send-to-destinations</Guide>

src/content/docs/guides/collecting/read-from-data-stores.mdx

@@ -0,0 +1,146 @@
+---
+title: Read from data stores
+---
+
+This guide shows you how to read from external data stores with TQL. You'll learn how to read full tables, push filters into SQL, inspect metadata, and stream new rows from MySQL.
+
+Today, this guide focuses on <Op>from_mysql</Op> and <Op>from_clickhouse</Op>.
+As Tenzir adds more data store integrations, the same patterns will apply.
+
+## Read a full table
+
+Use `table=...` when you want to fetch all rows from a table.
+
+Read a ClickHouse table:
+
+```tql
+from_clickhouse table="security.events",
+                host="clickhouse.example.com",
+                password=secret("CLICKHOUSE_PASSWORD"),
+                tls=false
+```
+
+Or use a ClickHouse URI:
+
+```tql
+from_clickhouse uri="clickhouse://default:secret@clickhouse.example.com:9000/security",
+                table="events",
+                tls=false
+```
+
+Read a MySQL table:
+
+```tql
+from_mysql table="users",
+           host="mysql.example.com",
+           user="tenzir",
+           password=secret("MYSQL_PASSWORD"),
+           database="identity"
+```
+
+Use this mode when you want Tenzir to treat the table as the source of truth and
+apply filtering later in the pipeline.
+
+## Push filters and projections into SQL
+
+Use `sql=...` when the data store should do the filtering, sorting, or column
+selection before Tenzir receives the rows.
+
+Filter in ClickHouse:
+
+```tql
+from_clickhouse sql="SELECT time, host, severity, message FROM events WHERE severity >= 3 ORDER BY time DESC",
+                host="clickhouse.example.com",
+                password=secret("CLICKHOUSE_PASSWORD"),
+                tls=false
+```
+
+Filter in MySQL:
+
+```tql
+from_mysql sql="SELECT id, user, last_login FROM users WHERE active = 1 ORDER BY last_login DESC",
+           host="mysql.example.com",
+           user="tenzir",
+           password=secret("MYSQL_PASSWORD"),
+           database="identity"
+```
+
+This pattern reduces network traffic and lets you use the source system's query
+planner and indexes.
+
+## Inspect metadata
+
+Both operators can return metadata instead of table rows.
+
+List ClickHouse tables in a database:
+
+```tql
+from_clickhouse sql="SHOW TABLES FROM security",
+                host="clickhouse.example.com",
+                password=secret("CLICKHOUSE_PASSWORD"),
+                tls=false
+```
+
+Show the columns of a MySQL table:
+
+```tql
+from_mysql table="users",
+           show="columns",
+           host="mysql.example.com",
+           user="tenzir",
+           password=secret("MYSQL_PASSWORD"),
+           database="identity"
+```
+
+Use metadata queries when you want to discover available tables, inspect a
+schema, or validate assumptions before you run a larger pipeline. In
+ClickHouse, prefer regular SQL such as `SHOW`, `DESCRIBE`, or queries against
+system catalogs.
+
+## Poll for new rows from MySQL
+
+MySQL supports a live polling mode for tables with a monotonically increasing
+integer tracking column.
+
+```tql
+from_mysql table="audit_log",
+           live=true,
+           tracking_column="id",
+           host="mysql.example.com",
+           user="tenzir",
+           password=secret("MYSQL_PASSWORD"),
+           database="security"
+where severity == "high"
+```
+
+Use this mode when you want to turn a database table into a continuously polled
+event source.
+
+`from_clickhouse` does not currently provide a comparable live mode. It runs a
+query, emits the result, and then finishes.
+
+## Shape rows after reading
+
+Both operators produce structured events, so you can transform the result right
+away.
+
+```tql
+from_clickhouse sql="SELECT host, severity, message FROM events",
+                host="clickhouse.example.com",
+                password=secret("CLICKHOUSE_PASSWORD"),
+                tls=false
+where severity >= 3
+message = message.to_upper()
+```
+
+This lets you treat database rows like any other event stream in Tenzir.
+
+## See also
+
+- <Guide>collecting/read-from-message-brokers</Guide>
+- <Guide>transformation/filter-and-select-data</Guide>
+- <Guide>routing/send-to-destinations</Guide>
+- <Op>from_clickhouse</Op>
+- <Op>from_mysql</Op>
+- <Integration>clickhouse</Integration>
+- <Integration>mysql</Integration>

src/content/docs/guides/routing/send-to-destinations.mdx

@@ -3,13 +3,14 @@ title: Send to destinations
 ---
 
 This guide shows you how to send data to various destinations using TQL output
-operators. You'll learn about destination operators, file output patterns, and
-expression-based serialization.
+operators. You'll learn about message destinations, data stores, file output
+patterns, and expression-based serialization.
 
 ## Destination operators
 
-TQL provides `to_*` operators for sending events to various destinations. These
-operators accept expressions for flexible serialization.
+TQL provides `to_*` operators for sending events to various destinations.
+Message-oriented operators accept expressions for flexible serialization, while
+data store operators write structured rows directly.
 
 ### Message brokers
 
@@ -51,6 +52,34 @@ subscribe "security-events"
 to_amazon_kinesis "security-events"
 ```
 
+### Data stores
+
+Send events to data stores like <Integration>clickhouse</Integration> and
+<Integration>snowflake</Integration>.
+
+Send structured events to ClickHouse:
+
+```tql
+subscribe "security-events"
+to_clickhouse table="alerts", primary=time, mode="create_append", tls=false
+```
+
+Write batches to Snowflake with bulk ingestion:
+
+```tql
+export
+to_snowflake \
+  account_identifier="org-account",
+  user_name="tenzir_user",
+  password=secret("SNOWFLAKE_PASSWORD"),
+  database="SECURITY",
+  schema="PUBLIC",
+  table="EVENTS"
+```
+
+These operators preserve event structure instead of requiring a `message`
+expression.
+
 ### Analytics platforms
 
 Send data to platforms like [Splunk](/integrations/splunk),
@@ -76,8 +105,8 @@ to_opensearch "https://opensearch.example.com:9200",
 
 ### Cloud services
 
-Route events to cloud destinations like <Integration>amazon/sqs</Integration>
-and <Integration>google/cloud-pubsub</Integration>.
+Route events to cloud destinations like [Amazon SQS](/integrations/amazon/sqs)
+and [Google Cloud Pub/Sub](/integrations/google/cloud-pubsub).
 
 Send to SQS:
 
@@ -90,7 +119,9 @@ Send to Pub/Sub:
 
 ```tql
 subscribe "events"
-to_google_cloud_pubsub project_id="my-project", topic_id="events", message=this.print_ndjson()
+to_google_cloud_pubsub project_id="my-project",
+  topic_id="events",
+  message=this.print_json()
 ```
 
 ## File output
@@ -139,9 +170,10 @@ to_tcp "collector.example.com:12201" {
 }
 ```
 
-## Expression-based serialization
+## Expression-based serialization for message destinations
 
-Destination operators use expressions for flexible message formatting:
+Message-oriented destination operators use expressions for flexible message
+formatting:
 
 ### Serialize the entire event
 
@@ -179,7 +211,7 @@ Route events to different destinations based on content:
 to_kafka f"events.{event_type}"
 ```
 
-## See Also
+## See also
 
 - <Op>fork</Op>
 - <Op>to_kafka</Op>
@@ -190,5 +222,8 @@ to_kafka f"events.{event_type}"
 - <Op>write_delimited</Op>
 - <Op>write_ndjson</Op>
 - <Fn>print_ndjson</Fn>
+- <Guide>collecting/read-from-data-stores</Guide>
 - <Guide>routing/load-balance-pipelines</Guide>
 - <Guide>routing/split-and-merge-streams</Guide>
+- <Op>to_clickhouse</Op>
+- <Op>to_snowflake</Op>