Skip to content

feat(benchmarks): Add velox_tpch_data_gen utility for SF=N Parquet generation - #1

Open
kgpai wants to merge 3 commits into
mainfrom
add-tpch-data-gen-utility
Open

kgpai wants to merge 3 commits into
mainfrom
add-tpch-data-gen-utility

Conversation

@kgpai

@kgpai kgpai commented May 1, 2026

Copy link
Copy Markdown
Owner

Summary

Adds velox_tpch_data_gen, a standalone CLI that generates TPC-H Parquet files at a configurable scale factor using Velox's own TpchConnector + Parquet writer. Output layout is Hive-style, matching what velox_tpch_benchmark --data_path=<dir> already expects, so the same directory is consumable by both the benchmark and other engines (DuckDB, Spark, etc.) for cross-engine comparison.

The implementation lifts the pattern from ParquetTpchTest::saveTpchTablesAsParquet but streams scan -> write through a multi-split pipeline so SF=10 (~10 GB) fits in memory. The original test pattern collects rows into a single in-memory RowVector, which only works at SF=0.01.

Why Velox-driven generation (vs. dbgen / DuckDB)

When this data feeds a Velox-vs-DuckDB perf comparison, having one engine write the files would bias the read path — DuckDB's writer might emit Parquet tuned for DuckDB's reader (page sizes, encodings, dictionary thresholds calibrated to its own decoder). Using Velox's writer keeps the read path neutral: both engines read files written by the engine under test, and we additionally exercise Velox's writer in the process.

Flags

Flag Default Notes
--output_dir required Created if missing. One subdirectory per table.
--scale_factor 1.0 Must be > 0. SF=1 → ~1 GB, SF=10 → ~10 GB.
--num_splits_per_table 8 Scan parallelism per table.
--num_drivers 4 Driver concurrency for the scan-then-write pipeline.

Usage

velox_tpch_data_gen \
  --output_dir=/data/tpch10 \
  --scale_factor=10 \
  --num_splits_per_table=8 \
  --num_drivers=8

Produces:

/data/tpch10/customer/*.parquet
/data/tpch10/lineitem/*.parquet
/data/tpch10/nation/*.parquet
/data/tpch10/orders/*.parquet
/data/tpch10/part/*.parquet
/data/tpch10/partsupp/*.parquet
/data/tpch10/region/*.parquet
/data/tpch10/supplier/*.parquet

Then:

velox_tpch_benchmark --data_path=/data/tpch10 --data_format=parquet

Test plan

  • Builds clean on Linux with -DVELOX_ENABLE_BENCHMARKS=ON
  • SF=1 generation produces files with row counts matching TPC-H spec (lineitem ~6M, orders ~1.5M, etc.)
  • velox_tpch_benchmark --data_path=<dir> successfully reads the generated SF=1 files and runs all 22 queries
  • DuckDB CLI can read the generated Parquet (interop check) — SELECT count(*) FROM read_parquet('/data/tpch1/lineitem/*.parquet')
  • SF=10 generation succeeds within ~10 minutes on a 30GB-RAM / 8-core machine

kgpai added 3 commits May 1, 2026 10:09
…neration

Standalone CLI that produces TPC-H Parquet files at a configurable scale
factor using Velox's own TpchConnector + Parquet writer. Output layout
matches what velox_tpch_benchmark expects, so the same directory can be
consumed by both the benchmark and other engines (e.g. DuckDB) for
cross-engine comparison.

Lifts the pattern from ParquetTpchTest::saveTpchTablesAsParquet but
streams scan->write through a multi-split pipeline so SF=10 (~10GB)
fits in memory; the test pattern collects rows into a single RowVector
which only works at SF=0.01.

Flags:
  --output_dir            Required. Output directory (created if missing).
  --scale_factor          Required > 0. SF=1 -> ~1GB, SF=10 -> ~10GB.
  --num_splits_per_table  Default 8. Scan parallelism per table.
  --num_drivers           Default 4. Driver concurrency.

Test plan:
- Builds locally and on gpu1
- SF=1 round-trip (gen -> velox_tpch_benchmark reads it -> DuckDB reads it)
- SF=10 generation succeeds and row counts match TPC-H spec
Without dwio::common::registerFileSinks(), the LocalFileSink isn't in the
factory registry, and tableWrite fails with 'FileSink is not registered for
<path>' once it tries to open the first output file.
task_writer_count defaults to 4 in core::QueryConfig, so tableWrite uses
4 writer drivers regardless of maxDrivers. With --num_drivers=8 we want
8 output files (one per driver) so all cores stay busy and downstream
scans get fine-grained split assignment. Override the config to numDrivers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant