HIVE-29516 Fix NPE in StatsUtils.updateStats when column statistics a… by shubhluck · Pull Request #6382 · apache/hive

shubhluck · 2026-03-19T21:50:15Z

…re unavailable

Added null checks before iterating over column statistics in:

StatsUtils.updateStats()
StatsUtils.getColStatisticsUpdatingTableAlias()
StatsRulesProcFactory (JOIN statistics) This prevents query compilation failures during semijoin optimization when column-level statistics are incomplete, commonly seen with large TPC-DS datasets (100GB+).

What changes were proposed in this pull request?

This PR adds null checks before iterating over column statistics in three locations to prevent NullPointerException:

StatsUtils.updateStats() - Added null check for stats.getColumnStats() before the for-each loop, defaulting to empty list when null
StatsUtils.getColStatisticsUpdatingTableAlias() - Added null check with early return of empty list when parent column stats are null
StatsRulesProcFactory (JOIN statistics computation) - Added null check before iterating over column stats during join statistics calculation
The root cause is that Statistics.getColumnStats() returns null (not an empty list) when no column statistics are available:

public List<ColStatistics> getColumnStats() {
    if (columnStats != null) {
        return Lists.newArrayList(columnStats.values());
    }
    return null;  // Returns null, causing NPE in for-each loops
}

Why are the changes needed?

Query compilation fails with NullPointerException during semijoin optimization when column statistics are unavailable:

java.lang.NullPointerException
    at org.apache.hadoop.hive.ql.stats.StatsUtils.updateStats(StatsUtils.java:2067)
    at org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1982)
    at org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:539)
    at org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:238)
    ...

This issue is particularly prevalent with:

Large TPC-DS datasets (100GB+) where statistics collection may be incomplete
Tables where column-level statistics have not been computed
Complex queries where intermediate operators lack column statistics
The fix ensures graceful handling when column statistics are unavailable, allowing the optimizer to continue using row-based statistics instead of failing.

Does this PR introduce any user-facing change?

No. This is a bug fix that prevents query compilation failures. Previously failing queries will now compile and execute successfully. There is no change to query results or behavior for queries that were already working.

How was this patch tested?

Reproduced the issue with TPC-DS queries at 100GB scale where column statistics were incomplete
Verified that queries failing with NPE now compile and execute successfully after the fix
Verified that queries with complete column statistics continue to work correctly and produce the same results
Existing unit tests pass without modification

To reproduce the original issue:

Generate TPC-DS dataset at 100GB+ scale
Do not compute column statistics (or ensure they are incomplete)
Run queries involving semijoin optimizations
Observe NPE during compilation

soumyakanti3578

This doesn't address the root cause mentioned in the PR description: getColumnStats.

Ideally it should return an empty list instead of returning null. But I understand that it is difficult to change it there as it's used in many other places. One solution could be to add a new method similar to getColumnStats:

public List<ColStatistics> getColumnStatsOrEmpty() {
    if (columnStats != null) {
        return Lists.newArrayList(columnStats.values());
    }
    return Collections.emptyList();
}

This method can be used in the three places that were changed in the PR.

Another thing that I noticed in updateStats is because now colStats is not null, we will set empty column stats in:

      stats.setColumnStats(colStats);
      long newDataSize = StatsUtils.getDataSizeFromColumnStats(newNumRows, colStats);
      stats.setDataSize(StatsUtils.getMaxIfOverflow(newDataSize));

This is a subtle change of this PR as earlier we would get an NPE instead. Do you think we should at least add a LOG.warn when useColStats = true and colStats is empty? Maybe it's a better idea to go to the else block when this happens?

There might be other changes like this which we need to be wary of.

Additionally, I looked into the other places which calls the method. I think there is a risk of NPE from SemanticAnalyzer.getMaterializedTableStats() too, so maybe we can call the new method there too. I think the other places are fine but it would help if you could investigate those too.

Finally, I think we should add some unit tests for the methods with changes.

shubhluck · 2026-03-24T05:24:09Z

@soumyakanti3578 Thanks for the review! Addressed all feedback:

Added getColumnStatsOrEmpty() - returns empty list instead of null. Now used in all 4 locations (3 original + SemanticAnalyzer.getMaterializedTableStats()).

Fixed updateStats() behavior - when useColStats=true but stats are empty, we now fall back to ratio-based calculation instead of setting empty stats. Added LOG.warn for visibility.

Investigated other callers - all safe: SortedDynPartitionOptimizer, StatsRulesProcFactory (UDTF), GenericUDAFBloomFilter, GroupingSetOptimizer, and ConvertJoinMapJoin already have null handling.

Added unit tests for getColumnStatsOrEmpty() and the updated behavior in updateStats()/getColStatisticsUpdatingTableAlias().

zabetak

The bug appears during a very specific use-case where semijoin removal takes place. Ideally, it would be nice to have a .q test that can reproduce the NPE. Note, that you can fake statistics by using ALTERA TABLE xx UPDATE STATISTICS commands (examples in reduce_deduplicate_stat_based.q).

ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java

ql/src/test/queries/clientpositive/semijoin_stats_missing_colstats.q

ql/src/test/org/apache/hadoop/hive/ql/plan/TestStatistics.java

ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java

ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java

ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java

zabetak

Now that the fix in TezCompiler is in place the only thing pending is getting the semijoin_stats_missing_colstats.q test right so that it actually goes through the problematic code.

ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java

ql/src/test/queries/clientpositive/semijoin_stats_missing_colstats.q

…unavailable Check column stats availability before passing useColStats=true in TezCompiler.removeSemijoinOptimizationByBenefit() to avoid NPE when column statistics are not present.

In order to trigger the NPE there are two important requirements: * `hive.stats.fetch.column.stats` must be set to false * the big table must be partitioned All configuration properties that are not necessary or using the default values are dropped.

zabetak

LGTM, pending tests.

sonarqubecloud · 2026-04-10T17:40:56Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

asf-ci-hive added tests pending tests passed and removed tests pending labels Mar 19, 2026

soumyakanti3578 reviewed Mar 24, 2026

View reviewed changes

asf-ci-hive added tests pending tests failed tests unstable and removed tests passed tests pending tests failed labels Mar 24, 2026

zabetak reviewed Mar 27, 2026

View reviewed changes

ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java Outdated Show resolved Hide resolved

asf-ci-hive added tests pending tests failed and removed tests unstable tests pending labels Mar 29, 2026

zabetak requested changes Mar 30, 2026

View reviewed changes

asf-ci-hive added tests pending tests unstable and removed tests failed tests pending tests unstable labels Mar 30, 2026

asf-ci-hive added tests pending tests unstable and removed tests pending labels Apr 1, 2026

zabetak requested changes Apr 2, 2026

View reviewed changes

ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java Show resolved Hide resolved

ql/src/test/queries/clientpositive/semijoin_stats_missing_colstats.q Outdated Show resolved Hide resolved

shubhluck force-pushed the HIVE-29516 branch from f642f53 to 732913f Compare April 4, 2026 15:24

asf-ci-hive added tests pending and removed tests unstable labels Apr 4, 2026

shubhluck force-pushed the HIVE-29516 branch 2 times, most recently from 963e305 to f9074cc Compare April 4, 2026 15:38

asf-ci-hive added tests failed and removed tests pending labels Apr 4, 2026

shubhluck requested a review from zabetak April 4, 2026 16:26

asf-ci-hive added tests pending tests unstable tests passed and removed tests failed tests pending tests unstable labels Apr 4, 2026

shubhluck and others added 3 commits April 10, 2026 17:41

HIVE-29516: Fix NPE in StatsUtils.updateStats when column statistics …

217c58e

…unavailable Check column stats availability before passing useColStats=true in TezCompiler.removeSemijoinOptimizationByBenefit() to avoid NPE when column statistics are not present.

HIVE-29516 Fix whitespace related to test failure

5439f62

zabetak force-pushed the HIVE-29516 branch from fa29d9f to 7328e0f Compare April 10, 2026 16:05

asf-ci-hive added tests pending and removed tests passed labels Apr 10, 2026

zabetak approved these changes Apr 10, 2026

View reviewed changes

asf-ci-hive added tests passed and removed tests pending labels Apr 10, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HIVE-29516 Fix NPE in StatsUtils.updateStats when column statistics a…#6382

HIVE-29516 Fix NPE in StatsUtils.updateStats when column statistics a…#6382
shubhluck wants to merge 3 commits intoapache:masterfrom
shubhluck:HIVE-29516

shubhluck commented Mar 19, 2026

Uh oh!

soumyakanti3578 left a comment

Uh oh!

shubhluck commented Mar 24, 2026

Uh oh!

zabetak left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zabetak left a comment

Uh oh!

Uh oh!

Uh oh!

zabetak left a comment

Uh oh!

sonarqubecloud bot commented Apr 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

shubhluck commented Mar 19, 2026

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

soumyakanti3578 left a comment

Choose a reason for hiding this comment

Uh oh!

shubhluck commented Mar 24, 2026

Uh oh!

zabetak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zabetak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zabetak left a comment

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Apr 10, 2026

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants