Conversation
…neration Standalone CLI that produces TPC-H Parquet files at a configurable scale factor using Velox's own TpchConnector + Parquet writer. Output layout matches what velox_tpch_benchmark expects, so the same directory can be consumed by both the benchmark and other engines (e.g. DuckDB) for cross-engine comparison. Lifts the pattern from ParquetTpchTest::saveTpchTablesAsParquet but streams scan->write through a multi-split pipeline so SF=10 (~10GB) fits in memory; the test pattern collects rows into a single RowVector which only works at SF=0.01. Flags: --output_dir Required. Output directory (created if missing). --scale_factor Required > 0. SF=1 -> ~1GB, SF=10 -> ~10GB. --num_splits_per_table Default 8. Scan parallelism per table. --num_drivers Default 4. Driver concurrency. Test plan: - Builds locally and on gpu1 - SF=1 round-trip (gen -> velox_tpch_benchmark reads it -> DuckDB reads it) - SF=10 generation succeeds and row counts match TPC-H spec
Without dwio::common::registerFileSinks(), the LocalFileSink isn't in the factory registry, and tableWrite fails with 'FileSink is not registered for <path>' once it tries to open the first output file.
task_writer_count defaults to 4 in core::QueryConfig, so tableWrite uses 4 writer drivers regardless of maxDrivers. With --num_drivers=8 we want 8 output files (one per driver) so all cores stay busy and downstream scans get fine-grained split assignment. Override the config to numDrivers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
velox_tpch_data_gen, a standalone CLI that generates TPC-H Parquet files at a configurable scale factor using Velox's ownTpchConnector+ Parquet writer. Output layout is Hive-style, matching whatvelox_tpch_benchmark --data_path=<dir>already expects, so the same directory is consumable by both the benchmark and other engines (DuckDB, Spark, etc.) for cross-engine comparison.The implementation lifts the pattern from
ParquetTpchTest::saveTpchTablesAsParquetbut streamsscan -> writethrough a multi-split pipeline so SF=10 (~10 GB) fits in memory. The original test pattern collects rows into a single in-memoryRowVector, which only works at SF=0.01.Why Velox-driven generation (vs. dbgen / DuckDB)
When this data feeds a Velox-vs-DuckDB perf comparison, having one engine write the files would bias the read path — DuckDB's writer might emit Parquet tuned for DuckDB's reader (page sizes, encodings, dictionary thresholds calibrated to its own decoder). Using Velox's writer keeps the read path neutral: both engines read files written by the engine under test, and we additionally exercise Velox's writer in the process.
Flags
--output_dir--scale_factor1.0--num_splits_per_table8--num_drivers4Usage
Produces:
Then:
Test plan
-DVELOX_ENABLE_BENCHMARKS=ONvelox_tpch_benchmark --data_path=<dir>successfully reads the generated SF=1 files and runs all 22 queriesSELECT count(*) FROM read_parquet('/data/tpch1/lineitem/*.parquet')