feature/add_audit_trail_events

.quickstart/quickstart.yml

@@ -10,6 +10,8 @@ table_variables:
     - user
   fivetran_platform_using_connection:
     - connection
+  fivetran_platform_using_audit_trail:
+    - audit_trail
   fivetran_platform_using_transformations:
     - transformation_runs
   fivetran_platform_using_connector_sdk_log:
@@ -28,7 +30,8 @@ public_models: [
   "fivetran_platform__connection_status",
   "fivetran_platform__usage_history",
   "fivetran_platform__audit_table",
+  "fivetran_platform__audit_trail_enriched",
   "fivetran_platform__mar_table_history",
-  "fivetran_platform__transformation_run_log"
+  "fivetran_platform__transformation_run_log",
   "fivetran_platform__errors_and_warnings"
 ]

CHANGELOG.md

@@ -1,4 +1,15 @@
 # dbt_fivetran_log v2.6.0
+
+[PR #197](https://github.com/fivetran/dbt_fivetran_log/pull/197) includes the following updates:
+
+## Schema/Data Change
+**2 total changes • 0 possible breaking changes**
+
+| Data Model(s) | Change type | Old | New | Notes |
+| ------------- | ----------- | --- | --- | ----- |
+| [fivetran_platform__audit_trail_enriched](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__audit_trail_enriched) | New final model |  |  | Enriches each audit trail event with the names of the primary and secondary resources involved (resolved for `CONNECTION`, `DESTINATION`, `ACCOUNT`, and `USER` resources) and the details of the user who performed the action. Only available for customers on the Enterprise plan and above. |
+| [stg_fivetran_platform__audit_trail](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.stg_fivetran_platform__audit_trail) | New staging model |  |  | Records user actions performed within your Fivetran account (what was changed, by whom, and how). Only available for customers on the Enterprise plan and above. |
+
 [PR #193](https://github.com/fivetran/dbt_fivetran_log/pull/193) includes the following updates:
 
 ## Schema/Data Change
@@ -10,6 +21,7 @@
 
 ## Under the Hood
 - Creates `fivetran_log_connection_ids_unnested` macro for warehouse-specific JSON array unnesting logic in `fivetran_platform__transformation_run_log` model.
+
 [PR #196](https://github.com/fivetran/dbt_fivetran_log/pull/196) includes the following updates:
 
 ## Schema/Data Change

DECISIONLOG.md

@@ -50,3 +50,6 @@ Not all customers have the `connector_sdk_log` source table, as it is only popul
 vars:
   fivetran_platform_using_connector_sdk_log: false ## Default is true. Set to false if the connector_sdk_log source table is not present in your destination.
 ```
+
+## Resolving connection names in `fivetran_platform__audit_trail_enriched`
+The `audit_trail` source identifies connections by `connection_id`, but `stg_fivetran_platform__connection` is unique on `(connection_name, destination_id)`, so one `connection_id` can span multiple rows (e.g. after a rename or move). We deduplicate the connection lookup to the active, most recently set up record per `connection_id` to avoid fanning out audit events. A connection therefore resolves to its current name; no history is lost, as renames are still captured in `old_values`/`new_values`.

README.md

@@ -5,7 +5,7 @@ This dbt package transforms data from the Fivetran Platform connector into analy
 
 ## Resources
 
-- Number of materialized models¹: 22
+- Number of materialized models¹: 23
 - Connector documentation
   - [Fivetran Platform connector documentation](https://fivetran.com/docs/logs/fivetran-platform)
   - [Fivetran Platform ERD](https://fivetran.com/docs/logs/fivetran-platform#schemainformation)
@@ -40,6 +40,7 @@ By default, this package materializes the following final tables:
 | [fivetran_platform__schema_changelog](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__schema_changelog) | Documents all schema changes made to your connections including table alterations, table creations, schema creations, and configuration changes with detailed metadata about each event to track data structure evolution and troubleshoot schema-related issues. |
 | [fivetran_platform__audit_table](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__audit_table) | Replaces the deprecated [`_fivetran_audit` table](https://fivetran.com/docs/getting-started/system-columns-and-tables#audittables) and tracks each table receiving data during connection syncs with comprehensive timestamps for connection and table-level sync progress plus detailed counts of records inserted, replaced, updated, and deleted to monitor data processing and sync performance. |
 | [fivetran_platform__audit_user_activity](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__audit_user_activity) | Records all user-triggered actions within your Fivetran account to provide a comprehensive audit trail that helps you trace user activities to specific [log events](https://fivetran.com/docs/logs#logeventlist) such as schema changes, sync frequency updates, manual syncs, connection failures, and other operational events for compliance and troubleshooting purposes. |
+| [fivetran_platform__audit_trail_enriched](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__audit_trail_enriched) | Enriches each account-level audit trail event with the human-readable names of the primary and secondary resources involved (resolved for `CONNECTION`, `DESTINATION`, `ACCOUNT`, and `USER` resources) and the details of the user who performed the action, so you can audit who changed what without manually resolving resource IDs. Only available for customers on the Enterprise plan and above. |
 | [fivetran_platform__transformation_run_log](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__transformation_run_log) | Surfaces each transformation run event from the Fivetran log, including start and end timestamps, formatted duration, per-step model success and failure counts, and overall transformation status to monitor and troubleshoot your transformation pipeline performance. Requires the `transformation_runs` source table. |
 | [fivetran_platform__errors_and_warnings](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__errors_and_warnings) | Consolidates error and warning events from both standard connections and Connector SDK connections into a single feed, enriched with connection details, so you can monitor and triage the severity of issues across your data pipelines. Excludes `INFO`-level events. |
 
@@ -114,6 +115,13 @@ vars:
     fivetran_platform_using_connector_sdk_log: false # Default is true. This will disable the connector_sdk_log source and exclude Connector SDK events from fivetran_platform__errors_and_warnings
 ```
 
+The `audit_trail` source table is only available on the Enterprise plan and above. It powers the `stg_fivetran_platform__audit_trail` and `fivetran_platform__audit_trail_enriched` models. By default, the `stg_fivetran_platform__audit_trail` model is enabled only if the `audit_trail` table is present in your schema, and Quickstart automatically skips both models when the source table is absent. For core dbt users, the `fivetran_platform__audit_trail_enriched` model is enabled by default. You may explicitly enable or disable both models by adding the following variable to your root `dbt_project.yml` file:
+
+```yml
+vars:
+    fivetran_platform_using_audit_trail: true # Disable by setting to false if you do not have the audit_trail source table
+```
+
 #### Leveraging `CONNECTION` vs `CONNECTOR`  
 In Q1 2025, the `CONNECTOR` source table was deprecated and replaced by `CONNECTION`, and `CONNECTION` is now the default source.
 

dbt_project.yml

@@ -25,3 +25,4 @@ vars:
     user: "{{ source('fivetran_platform', 'user') }}"
     usage_cost: "{{ source('fivetran_platform', 'usage_cost') }}"
     transformation_runs: "{{ source('fivetran_platform','transformation_runs') }}"
+    audit_trail: "{{ source('fivetran_platform','audit_trail') }}"

integration_tests/dbt_project.yml

@@ -94,6 +94,12 @@ seeds:
         updated_at: "{{ 'datetime2' if target.type == 'sqlserver' else 'timestamp' }}"
         model_runs: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
         _fivetran_synced: "{{ 'datetime2' if target.type == 'sqlserver' else 'timestamp' }}"
+    audit_trail:
+      +column_types:
+        _fivetran_synced: "{{ 'datetime2' if target.type == 'sqlserver' else 'timestamp' }}"
+        captured_at: "{{ 'datetime2' if target.type == 'sqlserver' else 'timestamp' }}"
+        old_values: "{{ 'string' if target.type in ('bigquery', 'spark', 'databricks') else 'varchar' if target.type != 'sqlserver' else 'varchar(256)' }}"
+        new_values: "{{ 'string' if target.type in ('bigquery', 'spark', 'databricks') else 'varchar' if target.type != 'sqlserver' else 'varchar(256)' }}"
     usage_cost:
       +column_types:
         _fivetran_synced: "{{ 'datetime2' if target.type == 'sqlserver' else 'timestamp' }}"

integration_tests/seeds/audit_trail.csv

@@ -0,0 +1,5 @@
+id,captured_at,user_id,action,interaction_method,primary_resource_type,primary_resource_id,secondary_resource_type,secondary_resource_id,old_values,new_values,_fivetran_synced
+aud_1aZ8kQ,2024-12-01 10:15:03.120,barista_fibula,CREATE,WEB_UI,CONNECTION,this_connection,DESTINATION,ups_trampoline,"{}","{""connection_name"":""this_connection""}",2024-12-01 12:00:00.000
+aud_2bY7lR,2024-12-02 14:22:47.880,backlog_jack,EDIT,API,CONNECTION,this_connection,,,"{""paused"":true}","{""paused"":false}",2024-12-02 15:00:00.000
+aud_3cX6mS,2024-12-03 09:05:11.450,barista_fibula,EDIT,WEB_UI,USER,backlog_jack,,,"{""role"":""Account Reviewer""}","{""role"":""Account Administrator""}",2024-12-03 10:00:00.000
+aud_4dW5nT,2024-12-04 18:40:59.000,backlog_jack,DELETE,SYSTEM,DESTINATION,ups_trampoline,,,"{""name"":""s_test""}","{}",2024-12-04 19:00:00.000

integration_tests/tests/integrity/row_count__audit_trail_enriched.sql

@@ -0,0 +1,25 @@
+
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+-- the enriched model resolves resource names via left joins and must remain 1:1 with the staging audit trail
+with end_model as (
+
+    select count(*) as row_count
+    from {{ ref('fivetran_platform__audit_trail_enriched') }}
+),
+
+staging_model as (
+
+    select count(*) as row_count
+    from {{ ref('stg_fivetran_platform__audit_trail') }}
+)
+
+select
+    end_model.row_count as end_model_row_count,
+    staging_model.row_count as staging_model_row_count
+from end_model
+cross join staging_model
+where end_model.row_count != staging_model.row_count

macros/staging/get_audit_trail_columns.sql

@@ -0,0 +1,20 @@
+{% macro get_audit_trail_columns() %}
+
+{% set columns = [
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "action", "datatype": dbt.type_string()},
+    {"name": "captured_at", "datatype": dbt.type_timestamp()},
+    {"name": "id", "datatype": dbt.type_string()},
+    {"name": "interaction_method", "datatype": dbt.type_string()},
+    {"name": "new_values", "datatype": dbt.type_string()},
+    {"name": "old_values", "datatype": dbt.type_string()},
+    {"name": "primary_resource_id", "datatype": dbt.type_string()},
+    {"name": "primary_resource_type", "datatype": dbt.type_string()},
+    {"name": "secondary_resource_id", "datatype": dbt.type_string()},
+    {"name": "secondary_resource_type", "datatype": dbt.type_string()},
+    {"name": "user_id", "datatype": dbt.type_string()}
+] %}
+
+{{ return(columns) }}
+
+{% endmacro %}

models/fivetran_platform.yml

@@ -283,6 +283,63 @@ models:
       - name: log_id
         description: Unique ID of the log record.
 
+  - name: fivetran_platform__audit_trail_enriched
+    description: >
+      Table of user actions performed within your Fivetran account, enriched with the human-readable
+      names of the resources involved and the user who performed each action. Each event references a
+      primary resource (the container, e.g. an `ACCOUNT`) and an optional secondary resource (the member,
+      e.g. a `USER` added to a `TEAM`). This model resolves both resource references to their names when
+      the resource type is a `CONNECTION`, `DESTINATION`, `USER`, or `ACCOUNT`. This table is only
+      available on the Enterprise plan and above.
+    columns:
+      - name: unique_audit_trail_key
+        description: Surrogate key hashed on `audit_trail_id` and `captured_at` to uniquely identify each audit trail event.
+        tests:
+          - unique
+          - not_null
+      - name: audit_trail_id
+        description: Identifier of the audit trail event.
+      - name: captured_at
+        description: Timestamp when the action was performed.
+      - name: user_id
+        description: Foreign key referencing the `user` who performed the action.
+      - name: actor_email
+        description: Account email of the user who performed the action. Only available when the `user` source table is enabled.
+      - name: actor_first_name
+        description: Given name of the user who performed the action. Only available when the `user` source table is enabled.
+      - name: actor_last_name
+        description: Surname of the user who performed the action. Only available when the `user` source table is enabled.
+      - name: action
+        description: The type of operation performed. Possible values include `CREATE`, `EDIT`, and `DELETE`.
+      - name: interaction_method
+        description: How the action was performed. Possible values include `WEB_UI`, `API`, and `SYSTEM`.
+      - name: primary_resource_type
+        description: >
+          The main type of resource (the container) affected by the action (e.g., `ACCOUNT`, `DESTINATION`, `CONNECTION`,
+          `TEAM`, `USER`, `TRANSFORMATION`, `ROLE`, `EXTERNAL_LOG_SERVICE`).
+      - name: primary_resource_id
+        description: Identifier of the primary resource affected by the action.
+      - name: primary_resource_name
+        description: >
+          Name of the primary resource, resolved from `primary_resource_id` when `primary_resource_type` is a
+          `CONNECTION`, `DESTINATION`, `ACCOUNT`, or `USER` (the user's email). Null for resource types without a
+          name lookup or when the resource no longer exists.
+      - name: secondary_resource_type
+        description: The type of secondary resource (the member) affected by the action, if applicable.
+      - name: secondary_resource_id
+        description: Identifier of the secondary resource affected by the action, if applicable.
+      - name: secondary_resource_name
+        description: >
+          Name of the secondary resource, resolved from `secondary_resource_id` when `secondary_resource_type` is a
+          `CONNECTION`, `DESTINATION`, `ACCOUNT`, or `USER` (the user's email). Null for resource types without a
+          name lookup or when the resource no longer exists.
+      - name: old_values
+        description: JSON-like string of the resource's attributes before the action.
+      - name: new_values
+        description: JSON-like string of the resource's attributes after the action.
+      - name: _fivetran_synced
+        description: Timestamp when the record was last synced.
+
   - name: fivetran_platform__errors_and_warnings
     description: >
       Table of connector error and warning events across both standard Fivetran connections (from the `log` table) and
@@ -305,4 +362,4 @@ models:
           Indicates which source the event originated from. Possible values are `standard_connector` (from the `log` table)
           and `connector_sdk` (from the `connector_sdk_log` table).
       - name: sync_id
-        description: Identifies which sync the event is associated with.
+        description: Identifies which sync the event is associated with.

models/fivetran_platform__audit_trail_enriched.sql

@@ -0,0 +1,137 @@
+{{ config(enabled=var('fivetran_platform_using_audit_trail', true)) }}
+
+with audit_trail as (
+
+    select *
+    from {{ ref('stg_fivetran_platform__audit_trail') }}
+),
+
+connection as (
+
+    -- dedupe to one row per connection_id (not unique in staging) so the join doesn't fan out audit events
+    select
+        connection_id,
+        connection_name
+    from (
+        select
+            connection_id,
+            connection_name,
+            row_number() over (
+                partition by connection_id
+                order by is_deleted asc, set_up_at desc
+            ) as nth_connection_record
+        from {{ ref('stg_fivetran_platform__connection') }}
+    ) deduped
+    where nth_connection_record = 1
+),
+
+destination as (
+
+    select
+        destination_id,
+        destination_name
+    from {{ ref('stg_fivetran_platform__destination') }}
+),
+
+account as (
+
+    select
+        account_id,
+        account_name
+    from {{ ref('stg_fivetran_platform__account') }}
+),
+
+{% if var('fivetran_platform_using_user', true) %}
+users as (
+
+    select
+        user_id,
+        email,
+        first_name,
+        last_name
+    from {{ ref('stg_fivetran_platform__user') }}
+),
+{% endif %}
+
+final as (
+
+    select
+        audit_trail.unique_audit_trail_key,
+        audit_trail.audit_trail_id,
+        audit_trail.captured_at,
+        audit_trail.user_id,
+{%- if var('fivetran_platform_using_user', true) %}
+        actor.email as actor_email,
+        actor.first_name as actor_first_name,
+        actor.last_name as actor_last_name,
+{% endif %}
+        audit_trail.action,
+        audit_trail.interaction_method,
+        audit_trail.primary_resource_type,
+        audit_trail.primary_resource_id,
+        coalesce(
+            primary_connection.connection_name,
+            primary_destination.destination_name,
+            primary_account.account_name
+            {%- if var('fivetran_platform_using_user', true) %},
+            primary_user.email
+            {%- endif %}
+        ) as primary_resource_name,
+        audit_trail.secondary_resource_type,
+        audit_trail.secondary_resource_id,
+        coalesce(
+            secondary_connection.connection_name,
+            secondary_destination.destination_name,
+            secondary_account.account_name
+            {%- if var('fivetran_platform_using_user', true) %},
+            secondary_user.email
+            {%- endif %}
+        ) as secondary_resource_name,
+        audit_trail.old_values,
+        audit_trail.new_values,
+        audit_trail._fivetran_synced
+
+    from audit_trail
+
+{%- if var('fivetran_platform_using_user', true) %}
+
+    -- actor who performed the action
+    left join users as actor
+        on audit_trail.user_id = actor.user_id
+{% endif %}
+
+    -- resolve the primary resource (the container) to its name
+    left join connection as primary_connection
+        on audit_trail.primary_resource_id = primary_connection.connection_id
+        and audit_trail.primary_resource_type = 'CONNECTION'
+    left join destination as primary_destination
+        on audit_trail.primary_resource_id = primary_destination.destination_id
+        and audit_trail.primary_resource_type = 'DESTINATION'
+    left join account as primary_account
+        on audit_trail.primary_resource_id = primary_account.account_id
+        and audit_trail.primary_resource_type = 'ACCOUNT'
+{%- if var('fivetran_platform_using_user', true) %}
+    left join users as primary_user
+        on audit_trail.primary_resource_id = primary_user.user_id
+        and audit_trail.primary_resource_type = 'USER'
+{% endif %}
+
+    -- resolve the secondary resource (the member) to its name
+    left join connection as secondary_connection
+        on audit_trail.secondary_resource_id = secondary_connection.connection_id
+        and audit_trail.secondary_resource_type = 'CONNECTION'
+    left join destination as secondary_destination
+        on audit_trail.secondary_resource_id = secondary_destination.destination_id
+        and audit_trail.secondary_resource_type = 'DESTINATION'
+    left join account as secondary_account
+        on audit_trail.secondary_resource_id = secondary_account.account_id
+        and audit_trail.secondary_resource_type = 'ACCOUNT'
+{%- if var('fivetran_platform_using_user', true) %}
+    left join users as secondary_user
+        on audit_trail.secondary_resource_id = secondary_user.user_id
+        and audit_trail.secondary_resource_type = 'USER'
+{% endif %}
+)
+
+select *
+from final