[Compatibility] Spark result mismatches found by differential fuzzing (single-SQL repros)

[zhangxffff]

Reference Engine

Spark (vanilla Spark 3.5.5 vs. the same Spark session with Gluten + Bolt, spark.gluten.enabled off/on)

Affected Function / Operator

IN-list scan filter, min/max (TIMESTAMP, DOUBLE, FLOAT), round, from_json, unhex, unix_timestamp/to_unix_timestamp, to_utc_timestamp/from_utc_timestamp, filter/exists (lambda), TRY_CAST/CAST, bit_count, initcap, parse_url, regexp_replace, get_json_object, substring_index, array_intersect, array_position/array_contains (NaN), CAST(date AS STRING)/array_join/to_json (date formatting), pow, to_timestamp, json_object_keys, array_contains (empty array), map_contains_key/map_concat (map()), rpad/lpad, date_format/to_timestamp (patterns), to_json (map key order), json_array_length, ascii

Reproduction (Query & Data)

These are the findings of a ~11.5h differential fuzz campaign (expression-level Bolt vs Spark 3.2.4, plus query-level Spark 3.5 + Gluten(Bolt) on/off; 443k generated queries, 485k expression iterations) that could afterwards be reproduced with one self-contained SQL statement each. Bolt commit 9efd689d (current main), built into the Gluten Bolt backend and run through scripts/launch-spark.sh.

Every statement reads its inputs from a small table. Note that the inputs must come from a real scan (here: a parquet file written from the VALUES rows), because Spark constant-folds projections over an inline VALUES relation before Gluten sees the plan and Bolt would never execute the expression. The exact script that produced every row below is scripts/spark-fuzz/single_sql_verify.py (materializes each VALUES clause as parquet, runs the statement with spark.gluten.enabled=false and =true, and diffs the rows); it can be re-run with --only <id>.

-- data for the IN-list cases
CREATE TABLE fz_scan USING parquet AS
SELECT * FROM VALUES (0, ''), (1, 'a'), (2, 'b'), (CAST(NULL AS INT), CAST(NULL AS STRING)) AS t(k, s);
-- fz_scan2: 400 rows, k = id % 2 (NULL every 10th row), s = '' / 'a' (NULL every 10th row)

All other statements are written as SELECT ... FROM VALUES (...) AS v(...); read them as "the VALUES rows written to parquet and queried".

Result Comparison

Legend: Spark = vanilla Spark 3.5.5, Bolt = same session with Gluten + Bolt. Rows are listed in the order of the VALUES rows.

A. Wrong rows / values returned silently

A1. WHERE col IN (NULL, ...) pushed into the Parquet scan returns rows equal to the type's default value (0 / '')

SELECT k FROM fz_scan WHERE k IN (CAST(NULL AS INT), 5);
-- Spark: 0 rows            Bolt: (0)
SELECT s FROM fz_scan WHERE s IN (CAST(NULL AS STRING), 'zzz');
-- Spark: 0 rows            Bolt: ('')
SELECT k, count(*) FROM fz_scan2 WHERE k IN (CAST(NULL AS INT), 1) GROUP BY k;
-- Spark: (1, 160)          Bolt: (0, 150), (1, 160)
SELECT k, count(*) FROM fz_scan2 WHERE k IN (CAST(NULL AS INT), 535, 25) GROUP BY k;
-- Spark: 0 rows            Bolt: (1, 40)
SELECT s, count(*) FROM fz_scan2 WHERE s IN (CAST(NULL AS STRING), 'zzz', 'yyy') GROUP BY s;
-- Spark: 0 rows            Bolt: ('', 200)
-- control: the same predicate on an expression is correct
SELECT k FROM fz_scan WHERE (k + 0) IN (CAST(NULL AS INT), 5);   -- both: 0 rows

Only the scan-column form is affected ((k + 0) IN (...), VALUES sub-queries and SELECT k IN (...) are all correct); spark.sql.parquet.filterPushdown=false does not change it. It looks like the NULL literal is turned into the default value when the IN list becomes a subfield filter.

Spark 3.2: reproduced, but integer scan-filter results were non-deterministic: A1.1 differed in 2/3 runs; A1.3 and A1.4 differed in 3/3 with varying Bolt counts.

A2. min/max on TIMESTAMP truncate microseconds to milliseconds (global aggregate, GROUP BY and window; first/collect_list are fine)

SELECT max(t), min(t) FROM VALUES (TIMESTAMP'2049-05-08 06:35:54.977576'), (TIMESTAMP'2000-01-01 00:00:00.123456') AS v(t);
-- Spark: 2049-05-08 06:35:54.977576, 2000-01-01 00:00:00.123456
-- Bolt : 2049-05-08 06:35:54.977000, 2000-01-01 00:00:00.123000
SELECT max(t) OVER () FROM VALUES (TIMESTAMP'2049-05-08 06:35:54.977576'), (TIMESTAMP'2000-01-01 00:00:00.123456') AS v(t);
-- Spark: ...54.977576 (x2)    Bolt: ...54.977000 (x2)

Cause: bolt/functions/prestosql/aggregates/MinMaxAggregates.cpp has a MinMaxAggregate<Timestamp>::extractValues specialization that does Timestamp::fromMillis(ts.toMillis()) (Presto semantics), and the Spark min/max reuse it.

A3. min/max on DOUBLE/FLOAT lose ±Infinity

SELECT max(x), min(x) FROM VALUES (CAST('-inf' AS DOUBLE)), (CAST('-inf' AS DOUBLE)) AS v(x);
-- Spark: -Infinity, -Infinity      Bolt: -1.7976931348623157e+308, -Infinity
SELECT min(x), max(f) FROM VALUES (CAST('inf' AS DOUBLE), CAST('-inf' AS FLOAT)) AS v(x, f);
-- Spark: Infinity, -Infinity       Bolt: Infinity, -3.4028234663852886e+38

A4. round(double, n) is inexact and overflows near DBL_MAX

SELECT round(x, 5), round(x, 2) FROM VALUES (1e15), (1234567.125) AS v(x);
-- Spark: (1000000000000000.0, 1000000000000000.0), (1234567.125, 1234567.13)
-- Bolt : (1000000000000000.1, 1000000000000000.1), (1234567.125, 1234567.13)
SELECT round(x, 5), floor(round(x, 3)) FROM VALUES (1.7976931348623157e308), (9007199254740992.0) AS v(x);
-- Spark: (1.7976931348623157e+308, 9223372036854775807), (9007199254740992.0, 9007199254740992)
-- Bolt : (Infinity, 9223372036854775807),                (9007199254740994.0, 9007199254740994)

Spark 3.2: both statements matched Spark in all three runs, although round was Bolt-native.

A5. from_json on malformed input

SELECT from_json(s, 'a INT, b STRING') FROM VALUES (' lead'), ('{"a":1}'), ('{bad') AS v(s);
-- Spark: {null, null}, {1, null}, {null, null}
-- Bolt : {null, ''},   {1, null}, NULL
SELECT from_json(s, 'a INT, b STRING') FROM VALUES ('{"a":'), ('{"a":1,"b":"x",'), ('{') AS v(s);
-- Spark: {null, null} for all three         Bolt: NULL for all three

Spark (PERMISSIVE) returns an all-NULL struct; Bolt returns either a struct whose STRING field is '' or a NULL struct. (Related: #902.)

Spark 3.2: both from_json cases fell back to Spark, so no mismatch was observed.

A6. unhex on odd-length input

SELECT unhex(s) FROM VALUES ('abc'), ('zz'), ('ab') AS v(s);
-- Spark: X'0ABC', NULL, X'AB'      Bolt: X'BC00', NULL, X'AB'

Spark 3.2: both modes returned X'BC00' for 'abc'; unhex was Bolt-native.

A7. unix_timestamp / to_unix_timestamp of a pre-1970 fractional timestamp (truncation instead of floor)

SELECT unix_timestamp(t), to_unix_timestamp(t), unix_seconds(t) FROM VALUES (TIMESTAMP'1969-12-31 23:59:59.5') AS v(t);
-- Spark: 0, 0, -1        Bolt: -1, -1, -1

(Spark's unix_timestamp truncates toward zero here; unix_seconds floors. Bolt floors for all three.)

A8. Timezone conversion offsets (same root cause as #965, listed for completeness)

SELECT to_utc_timestamp(t, 'Europe/Berlin') FROM VALUES (TIMESTAMP'2059-05-28 23:26:11.59779'), (TIMESTAMP'2020-05-28 23:26:11') AS v(t);
-- Spark: 2059-05-28 21:26:11.59779, 2020-05-28 21:26:11
-- Bolt : 2059-05-28 22:26:11.59779, 2020-05-28 21:26:11
SELECT from_utc_timestamp(t, 'Pacific/Chatham'), from_utc_timestamp(t, 'Australia/Lord_Howe'), from_utc_timestamp(t, 'America/Los_Angeles')
FROM VALUES (TIMESTAMP'2050-07-01 00:00:00') AS v(t);
-- Spark: 2050-07-01 12:45:00, 2050-07-01 10:30:00, 2050-06-30 17:00:00
-- Bolt : 2050-07-01 13:45:00, 2050-07-01 11:00:00, 2050-06-30 16:00:00

A9. filter with a lambda that does not reference the element drops NULL elements

SELECT filter(a, x -> y IS NOT NULL), size(filter(a, x -> true)) FROM VALUES (array(1, NULL, 2), 5) AS v(a, y);
-- Spark: [1, null, 2], 3        Bolt: [1, 2], 3

A10. TRY_CAST / CAST of out-of-range doubles

SELECT TRY_CAST(x AS BIGINT), TRY_CAST(x AS INT), TRY_CAST(x AS SMALLINT) FROM VALUES (CAST('NaN' AS DOUBLE)) AS v(x);
-- Spark: NULL, NULL, NULL        Bolt: 0, 0, 0
SELECT year(TRY_CAST(x AS TIMESTAMP)) FROM VALUES (1e16) AS v(x);
-- Spark: NULL                    Bolt: 294247
SELECT CAST(x AS BIGINT) FROM VALUES (CAST(9223372036854775807L AS DOUBLE)) AS v(x);   -- x = 2^63 exactly
-- Spark: 9223372036854775807     Bolt: -9223372036854775808

Spark 3.2: the NaN-to-integral TRY_CAST fell back and matched Spark; TRY_CAST(DOUBLE AS TIMESTAMP) was rejected during analysis in both modes.

A11. bit_count on narrow negative integers

SELECT bit_count(CAST(x AS INT)), bit_count(CAST(x AS SMALLINT)), bit_count(CAST(x AS TINYINT)), bit_count(CAST(x AS BIGINT)) FROM VALUES (-1) AS v(x);
-- Spark: 64, 64, 64, 64          Bolt: 32, 16, 8, 64

(Spark promotes to long before counting.)

A12. initcap treats non-space characters as word delimiters

SELECT initcap(s) FROM VALUES ('a,b,c'), ('asia/shanghai'), ('{"a":1}') AS v(s);
-- Spark: A,b,c   Asia/shanghai   {"a":1}
-- Bolt : A,B,C   Asia/Shanghai   {"A":1}

A13. parse_url on a string that is not a URL

SELECT parse_url(s, 'PATH'), parse_url(s, 'HOST'), parse_url(s, 'FILE') FROM VALUES ('[1,2]') AS v(s);
-- Spark: NULL, NULL, NULL        Bolt: '[1,2]', NULL, '[1,2]'

A14. regexp_replace when the pattern matches the empty string

SELECT regexp_replace(s, '^$', 'X'), regexp_replace(s, 'x*', '-') FROM VALUES ('') AS v(s);
-- Spark: 'X', '-'                Bolt: '', ''

('abc' with 'x*' gives -a-b-c- on both.)

Spark 3.2: both modes returned '', ''; regexp_replace was Bolt-native.

A15. get_json_object with $.* on an array (related: #688)

SELECT get_json_object(s, '$.*'), get_json_object(s, '$[*]') FROM VALUES ('[1,2]') AS v(s);
-- Spark: NULL, '[1,2]'           Bolt: '[1,2]', '[1,2]'

A16. substring_index with an empty delimiter

SELECT substring_index(s, '', -3), substring_index(s, '', 2) FROM VALUES ('m.,') AS v(s);
-- Spark: '', ''                  Bolt: 'm.,', ''

A17. array_intersect element order with NULL

SELECT array_intersect(a, b), array_intersect(b, a) FROM VALUES (array('', NULL, 'x'), array(NULL, '')) AS v(a, b);
-- Spark: ['', null], [null, '']  Bolt: ['', null], ['', null]

A18. NaN equality in array_position / array_contains

SELECT array_position(array(CAST(NULL AS DOUBLE), x, x), x), array_contains(array(x), x) FROM VALUES (CAST('NaN' AS DOUBLE)) AS v(x);
-- Spark: 2, true                 Bolt: 0, false

A19. Date to string: years < 1000 not zero-padded, years > 9999 without + (date_format was fixed in #948; the cast / array_join / to_json paths were not)

SELECT CAST(d AS STRING), CAST(DATE_FROM_UNIX_DATE(i) AS STRING), array_join(array(d), ','), to_json(struct(d))
FROM VALUES (DATE'0001-01-01', 92489428) AS v(d, i);
-- Spark: 0001-01-01, +255197-06-15, 0001-01-01, {"d":"0001-01-01"}
-- Bolt : 1-01-01,    255197-06-15,  1-01-01,    {"d":"1-01-01"}

A20. pow special values follow C, not Java

SELECT pow(x, y) FROM VALUES (-1.0, CAST('inf' AS DOUBLE)), (-1.0, CAST('-inf' AS DOUBLE)), (1.0, CAST('NaN' AS DOUBLE)) AS v(x, y);
-- Spark: NaN, NaN, NaN           Bolt: 1.0, 1.0, 1.0

A21. to_timestamp with yy / yyyy patterns

SELECT to_timestamp(s, 'yy'), to_timestamp(s, 'yyyy') FROM VALUES ('99'), ('2020') AS v(s);
-- Spark: (2099-01-01 00:00:00, NULL),             (NULL, 2020-01-01 00:00:00)
-- Bolt : (1999-01-01 00:00:00, 0099-01-01 00:00:00), (2020-01-01 00:00:00, 2020-01-01 00:00:00)

A22. to_json sorts map keys after map_concat

SELECT to_json(map_concat(m, map())), to_json(m) FROM VALUES (map('b', 1, 'a', 2)) AS v(m);
-- Spark: {"b":1,"a":2}, {"b":1,"a":2}     Bolt: {"a":2,"b":1}, {"b":1,"a":2}

A23. json_array_length on trailing garbage, ascii on invalid UTF-8

SELECT json_array_length(s) FROM VALUES ('[1,2][1,2]x') AS v(s);      -- Spark: 2      Bolt: NULL
SELECT ascii(CAST(b AS STRING)) FROM VALUES (X'FF') AS v(b);          -- Spark: 65533  Bolt: -1

Spark 3.2: ascii(X'FF') returned -1 in both modes; ascii was Bolt-native.

B. Bolt raises where Spark returns a value

-- B1 json_object_keys on non-object / truncated JSON (Spark: NULL). Related: #742
SELECT json_object_keys(s) FROM VALUES ('{"a":'), ('[1,2]'), ('"str"'), ('{') AS v(s);
-- Bolt: TAPE_ERROR: The JSON document has an improper structure ...
-- Spark 3.2: both modes returned NULL for all four rows; json_object_keys was Bolt-native.

-- B2 array_contains / IN over an empty array (Spark: false)
SELECT array_contains(CAST(array() AS ARRAY<INT>), x), array_contains(array_intersect(array(0), array(1)), x) FROM VALUES (1) AS v(x);
-- Bolt: (0 vs. 0) IN list must not be empty
-- Spark 3.2: the expression fell back; both modes returned false, false.

-- B3 map_contains_key / map_concat with an empty map() (Spark: false / {a -> 1})
SELECT map_contains_key(map(), s), map_concat(map(), map('a', 1)) FROM VALUES ('a') AS v(s);
-- Bolt: (0 vs. 0) IN list must not be empty
-- Spark 3.2: map_contains_key is unavailable, so this exact query failed analysis in both modes.

-- B4 rpad / lpad with an empty pad string (Spark: 'ab', 'ab')
SELECT rpad(s, 3, ''), lpad(s, 3, '') FROM VALUES ('ab') AS v(s);
-- Bolt: padString must not be empty

C. Spark raises where Bolt returns a value

-- C1 date/time patterns rejected by Spark 3 (DATETIME_PATTERN_RECOGNITION)
SELECT to_timestamp(s, 'u'), date_format(t, 'u'), unix_timestamp(s2, 'EEE, MMM d yyyy')
FROM VALUES ('1', TIMESTAMP'2020-01-01 00:00:00', 'Wed, Jan 1 2020') AS v(s, t, s2);
-- Spark: SparkUpgradeException [INCONSISTENT_BEHAVIOR_CROSS_VERSION.DATETIME_PATTERN_RECOGNITION]
-- Bolt : NULL, '3', 1577836800

Bolt Version / Commit ID

9efd689d (main, 2026-09-05: "fix(type): fix decimal cast error formatting (#970)")

Reference Engine Version

Spark 3.5.5 (vanilla, same session with spark.gluten.enabled=false); the expression-level fuzz lane also compared against vanilla Spark 3.2.4 and agrees on every item above.

Important Configurations (Critical)

spark.sql.session.timeZone=UTC
spark.sql.ansi.enabled=false
spark.sql.legacy.timeParserPolicy=CORRECTED
spark.sql.parquet.datetimeRebaseModeInRead/Write=CORRECTED
spark.sql.parquet.int96RebaseModeInRead/Write=CORRECTED
spark.sql.adaptive.enabled=false          (only so that explain shows the Gluten plan)
spark.gluten.enabled=false | true         (the only difference between the two columns)

Every "Bolt" row above was checked to run in a Gluten *Transformer operator (offloaded), not a fallback.

Additional context

• Full campaign report with counts, the re-verification procedure (3x re-runs per finding group, Gluten replay of the expression-level findings) and the items that could only be reproduced on fuzz datasets: scripts/spark-fuzz/reports/long-20260906/REPORT.md (branch with the fuzz framework; not merged yet). The per-statement table this issue is derived from is single_sql_verification.md in the same directory.
• Not listed here because the single-SQL check showed both engines raising (only the message differs): unbase64 on invalid input, NULL_MAP_KEY, DUPLICATED_MAP_KEY, element_at(arr, 0), url_decode on a bad escape, add_months overflow.
• Findings that only reproduced on fuzz datasets so far (kept out of this issue): map_from_entries with NULL values returning NULL, count(DISTINCT) of NaN/-0.0, var_pop/corr etc. returning 0.0/inf instead of NaN, overlay, from_unixtime, array_remove on arrays with NULL, date_trunc with invalid units.