snowflake: add snowflake_adapter for table/column comments and tags#3964
Open
usbrandon wants to merge 1 commit into
Open
snowflake: add snowflake_adapter for table/column comments and tags#3964usbrandon wants to merge 1 commit into
usbrandon wants to merge 1 commit into
Conversation
Mirrors the pattern shipped for Databricks in dlt-hub#2625 (table_comment + table_tags + column_tags adapter hints + post-CREATE DDL emission). Closes a gap reported repeatedly when comparing dlt-Snowflake against Databricks/BigQuery — until now the only way to land COMMENTs on Snowflake was the column-level `description` hint; tables and tag objects had no path. New API (mirrors `databricks_adapter` / `bigquery_adapter` shape): from dlt.destinations.adapters import snowflake_adapter snowflake_adapter( resource, table_comment="Shopify orders — one row per checkout", table_tags={"governance.domain": "commerce"}, column_comments={"customer_email": "Buyer's email (PII)"}, column_tags={"customer_email": {"governance.pii_class": "internal"}}, ) Hints applied by the adapter: - x-snowflake-table-comment → COMMENT ON TABLE <name> IS '...' (falls back to the generic `description` table hint, matching the existing column-level fallback at `_get_column_def_sql`) - x-snowflake-table-tags → ALTER TABLE <name> SET TAG <k> = '<v>' per pair - x-snowflake-column-tags → ALTER TABLE <name> ALTER COLUMN <c> SET TAG <k> = '<v>' per (col, tag) pair - x-snowflake-column-comment → already existed; the adapter exposes it via `column_comments=` for symmetry Differences from databricks_adapter: - Tags are dicts ({name: value}), not the Databricks list-of-strings shape — Snowflake tags are first-class catalog objects with values and the loose-string form doesn't map. - No `table_properties` / `cluster` / `partition` knobs. Snowflake doesn't have TBLPROPERTIES; clustering is configured via the destination-level `create_indexes` knob; iceberg tables live behind a catalog integration not a CREATE TABLE option. - Adapter does NOT create tag objects — `CREATE TAG <name>` must already have been run by governance tooling. Keeps the loader role's privilege footprint small (no CREATE TAG ACCOUNT-level grant needed). SQL emission integrated into `SnowflakeClient._get_table_update_sql` via a new `_add_comment_and_tag_sql()` helper that runs after the existing `_add_cluster_sql()`. Each comment/tag becomes its own statement so attribution stays clean on partial failures (a single bad tag doesn't invalidate the CREATE/ALTER). Tests: - tests/load/snowflake/test_snowflake_adapter.py — adapter API validation (9 cases) covering shape rejection, fallback behavior, chaining with existing hints. - tests/load/snowflake/test_snowflake_table_builder.py — 5 new SQL emission tests covering table_comment, description-fallback, table_tags, column_tags, and Snowflake literal-escape on quotes. - All 29 tests in test_snowflake_table_builder.py + test_snowflake_adapter.py pass. Docs: `docs/website/docs/dlt-ecosystem/destinations/snowflake.md` gains a "Snowflake adapter" section under "Additional destination options" showing the adapter call, argument semantics, and the Databricks-divergence list.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
snowflake_adaptermirroring the pattern shipped for Databricks in #2625. Closes the gap where the dlt-Snowflake destination only honored column-levelCOMMENT(via thedescriptioncolumn hint) — tables, table tags, and column tags had no documented path.What's included
dlt/destinations/impl/snowflake/snowflake_adapter.py— new adapter withtable_comment,table_tags,column_comments,column_tagsarguments. Mirrors thedatabricks_adapter/bigquery_adapterAPI shape.dlt/destinations/impl/snowflake/snowflake.py— new_add_comment_and_tag_sql()helper called from_get_table_update_sql. Emits:COMMENT ON TABLE <name> IS '…'from thex-snowflake-table-commenthint, with fallback to the genericdescriptiontable hint (matches the existing column-level fallback).ALTER TABLE <name> SET TAG <k> = '<v>'per pair fromx-snowflake-table-tags.ALTER TABLE <name> ALTER COLUMN <c> SET TAG <k> = '<v>'fromx-snowflake-column-tags.dlt/destinations/adapters.py— exportssnowflake_adapteralongside the others.tests/load/snowflake/test_snowflake_adapter.py— 9 unit tests for the adapter API (shape validation, fallback behavior, chaining with existing hints).tests/load/snowflake/test_snowflake_table_builder.py— 5 SQL emission tests (table_comment, description fallback, table_tags, column_tags, literal-escape).docs/website/docs/dlt-ecosystem/destinations/snowflake.mdgains a "Snowflake adapter" section before the existing Query tagging section.Differences from
databricks_adapter{name: value}), not the Databricks list-of-strings shape. Snowflake tags are first-class catalog objects with values; the loose-string form doesn't map.table_properties/cluster/partitionarguments. Snowflake doesn't have TBLPROPERTIES; clustering is configured via the destination-levelcreate_indexes+ cluster column hint; iceberg tables live behind a separate catalog integration not a CREATE TABLE option.CREATE TAG <name>must already have been run by governance/Terraform tooling. Keeps the loader role's privilege footprint small (no CREATE TAG ACCOUNT-level grant needed).Test plan
test_snowflake_adapter.py).test_snowflake_table_builder.py).databricks_adapter-style `@pytest.mark.parametrize("destination_config", destinations_configs(default_sql_configs=True, subset=["snowflake"]))` pipeline test if you'd like it before merge).Context
Originally surfaced by a downstream user needing per-table cost-allocation tags on a Snowflake warehouse + table-level documentation that PowerBI / Snowsight can surface via DESCRIBE TABLE. The Databricks pattern from #2625 fit cleanly; this PR ports it.
🤖 Generated with Claude Code