Skip to content

[BUGFIX] ExpectColumnValuesToBeOfType returns full map-format result for all backends#11868

Closed
EshwarCVS wants to merge 0 commit into
fivetran:developfrom
EshwarCVS:develop
Closed

[BUGFIX] ExpectColumnValuesToBeOfType returns full map-format result for all backends#11868
EshwarCVS wants to merge 0 commit into
fivetran:developfrom
EshwarCVS:develop

Conversation

@EshwarCVS

@EshwarCVS EshwarCVS commented May 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #11076ExpectColumnValuesToBeOfType was returning only
{"observed_value": <type>} for non-object Pandas dtypes, SQLAlchemy,
and Spark backends instead of the standard ColumnMapExpectation result
structure documented in the expectation.

Also confirms #11026 (accented characters in row_condition strings) is
already resolved via alphas8bit in legacy_row_conditions.py.

Changes

  • expect_column_values_to_be_of_type.py

    • Added table.row_count and column_values.nonnull.unexpected_count
      to get_validation_dependencies for all backends so missing/null
      stats are always available.
    • Added _build_map_result() helper that wraps aggregate type-check
      results through _format_map_output, computing nonnull_count from
      the null metric and preserving observed_value.
    • Updated _validate() to route non-map paths through _build_map_result.
  • test_expect_column_values_to_be_of_type.py

    • Added tests verifying the full result schema on success, failure, and
      columns containing nulls.

Before / After

Before:

{ "success": false, "result": { "observed_value": "StringType" } }

After:

{
  "success": false,
  "result": {
    "element_count": 5,
    "unexpected_count": 4,
    "unexpected_percent": 100.0,
    "partial_unexpected_list": [],
    "missing_count": 1,
    "missing_percent": 20.0,
    "unexpected_percent_total": 80.0,
    "unexpected_percent_nonmissing": 100.0,
    "observed_value": "StringType"
  }
}

@netlify

netlify Bot commented May 6, 2026

Copy link
Copy Markdown

👷 Deploy request for niobium-lead-7998 pending review.

Visit the deploys page to approve it

Name Link
🔨 Latest commit 4c202a1

@EshwarCVS

EshwarCVS commented May 6, 2026

Copy link
Copy Markdown
Contributor Author

Didn't see #11855
Similar changes in this PR can be ignored

@joshua-stauffer

Copy link
Copy Markdown
Collaborator

thanks for the PR @EshwarCVS . This expectation is unusual in that the Pandas implementation behaves as a row level check, while the Spark and SQL implementations behave as a schema level check. We'll address the gap, but haven't settled on a solution yet. I'll leave this open, but will hold off reviewing for the time being.

@github-actions

Copy link
Copy Markdown
Contributor

Is this PR still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity.

It will be closed if no further activity occurs. Thank you for your contributions 🙇

@github-actions github-actions Bot added the stale Stale issues and PRs label Jun 11, 2026
@netlify

netlify Bot commented Jun 30, 2026

Copy link
Copy Markdown

👷 Deploy request for niobium-lead-7998 pending review.

Visit the deploys page to approve it

Name Link
🔨 Latest commit d8813fe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale Stale issues and PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ExpectColumnValuesToBeOfType dq rule is not returning expected result format

2 participants