bugfix/cast_strings_as_floats

[fivetran-savage]

PR Overview

Package version introduced in this PR:

• 0.9.2

This PR addresses the following Issue/Feature(s):

• GA-1023403

Summary of changes:

• Fixes an issue in stg_workday__worker_history where annual_currency_summary_primary_compensation_basis, annual_currency_summary_total_base_pay, and annual_currency_summary_total_salary_and_allowances could surface as string datatypes by casting as float.

Submission Checklist

• Alignment meeting with the reviewer (if needed)
  • Timeline and validation requirements discussed
• Provide validation details:
  • Validation Steps: Check for unintentional effects (e.g., add/run consistency & integrity tests)
  • Testing Instructions: Confirm the change addresses the issue(s)
  • Focus Areas: Complex logic or queries that need extra attention
• Merge any relevant open PRs into this PR

Changelog

• Draft changelog for PR
• Final changelog for release review

[fivetran-avinash]

@fivetran-savage Sent some notes back, let me know if you have questions!

[fivetran-avinash]

I think there is a silent failure when running against databricks here. Variables set inside a {% for %} loop don't propagate out of it in Jinja2. So {% set is_str = true %} inside the {% for stype in string_dtypes %} loop has no effect, and is_str always stays false after the loop ends. This means string-typed compensation columns on Databricks are never cast to float and silently land as strings in the output table.

I tested with this configuration in integration_tests/dbt_project.yml

        annual_currency_summary_primary_compensation_basis: "{{ 'varchar(100)' if target.type in ('redshift', 'postgres') else 'string' }}"
        annual_currency_summary_total_base_pay: "{{ 'varchar(100)' if target.type in ('redshift', 'postgres') else 'string' }}"
        annual_currency_summary_total_salary_and_allowances: "{{ 'varchar(100)' if target.type in ('redshift', 'postgres') else 'string' }}" 

And when running

dbt compile -s stg_workday__worker_history -t databricks --vars '{"employee_history_enabled": true}'

You end up seeing that these fields are not cast to floats and remain as strings.

So the Databricks case isn't quite solved as of yet. I wonder if we can use the Jinja namespace capability, to do the data type check, and then change the value to true that way. Might be worth testing.

[fivetran-avinash]

Looks like the latest changes fixed this!

[fivetran-savage]

namespace was the key.

[fivetran-avinash]

These fields are also present in the base stg_workday__worker model. Should we apply the macro there as well?

[fivetran-savage]

per this comment in the Jira ticket, the decision was to only apply to the worker_history table.

[fivetran-avinash]

I don't think 'text' is a string data type in Databricks. It's harmless if it's there, but might be worth double-checking if it's safe to remove.

[fivetran-savage]

Removed.

[fivetran-avinash]

@fivetran-savage Approved with a small fix.