Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1010 commits
Select commit Hold shift + click to select a range
4bc810a
remove debug print, style fix
karthikeyann Mar 12, 2025
affc476
stylefix
karthikeyann Mar 12, 2025
cf3da81
fix defaults in benchmarks
karthikeyann Mar 12, 2025
84effce
Update ToCudf.cpp
karthikeyann Mar 13, 2025
e0ab0aa
Merge pull request #80 from karthikeyann/fea-addFilter
karthikeyann Mar 13, 2025
7279af1
Merge branch 'velox-cudf' into local-all_PRs_merged1
karthikeyann Mar 13, 2025
9dc09e1
update cmake on centos
devavret Mar 13, 2025
e0ccfbd
Add our team to codeowners
devavret Mar 13, 2025
d00db4f
Check off some todos
devavret Mar 13, 2025
5accb50
remove commented tests
devavret Mar 13, 2025
4bc9e28
Merge pull request #88 from bdice/use-adapters-cuda
karthikeyann Mar 13, 2025
5fe2575
Ignore known warnings just for cudf_exec target
devavret Mar 17, 2025
c9b2c1a
Misc review fixes:
devavret Mar 17, 2025
ebaa994
Revert erroneous hardcoding of `numSplitsPerFile`
mhaseeb123 Mar 17, 2025
ef95aa8
Revert erroneous hardcoding of `numSplitsPerFile`
mhaseeb123 Mar 17, 2025
ddb5fb6
Merge branch 'velox-cudf' into pr/karthikeyann/71
devavret Mar 17, 2025
5529c0a
Merge pull request #71 from karthikeyann/fea-more_hashjoins
devavret Mar 17, 2025
8e696fc
A abrely working eager groupby impl that only works for q18 (groupby …
devavret Mar 18, 2025
44b8d02
Merge remote-tracking branch 'origin/velox-cudf' into local-all_PRs_m…
devavret Mar 18, 2025
9bc7263
add estimateFlatSize to CudfVector
karthikeyann Mar 19, 2025
5a75b9e
Misc review changes requested by @bdice
devavret Mar 19, 2025
3f4ca09
Remove only cudf adapter
devavret Mar 19, 2025
6f7d72e
Add clang format to our subdir
devavret Mar 19, 2025
2d679a4
Fix style
devavret Mar 20, 2025
2fd784e
use stream_c
karthikeyann Mar 20, 2025
9f63a02
Merge branch 'enh-memory_cudfvector' of github.com:karthikeyann/velox…
karthikeyann Mar 20, 2025
757bce9
Fix naming
devavret Mar 21, 2025
436d721
Fix more style
devavret Mar 21, 2025
5538888
Error out when cuda architecture is less than 70
devavret Mar 21, 2025
5d6e583
Use streaming aggregation in partial step and disable in final step f…
devavret Mar 23, 2025
a88a331
use non-kernel method
karthikeyann Mar 24, 2025
31caae7
Merge branch 'velox-cudf' of github.com:rapidsai/velox-private into f…
karthikeyann Mar 24, 2025
f3b1d23
clean to split to 2 PRs
karthikeyann Mar 24, 2025
203561c
move remainingFilterExpr to new PR
karthikeyann Mar 24, 2025
5979144
rename cudfIsRegistered to isCudfRegistered
karthikeyann Mar 25, 2025
71624fa
Misc. review changes
devavret Mar 25, 2025
6ccdb57
Misc review changes
devavret Mar 25, 2025
ab399f2
Prevent merging vectors whose total size exceeds vector_size_t max
devavret Mar 25, 2025
327f718
Misc review changes
devavret Mar 25, 2025
3c1554d
address review comments
karthikeyann Mar 25, 2025
08e2d68
Merge pull request #57 from karthikeyann/fea-cudfTableScan_TPCH
karthikeyann Mar 25, 2025
ea70e4b
remove unused use_arrow_schema
karthikeyann Mar 25, 2025
6d67a56
Merge branch 'velox-cudf' of github.com:rapidsai/velox-private into l…
karthikeyann Mar 25, 2025
3d43de8
remove unused include
karthikeyann Mar 25, 2025
fa2ebea
cleanup parquet filter to separate PR
karthikeyann Mar 25, 2025
d4d119a
cleanup Utilities.h
karthikeyann Mar 25, 2025
201c800
remove unused include
karthikeyann Mar 25, 2025
d28820b
include cleanup
karthikeyann Mar 26, 2025
1038579
style fix
karthikeyann Mar 26, 2025
5216246
cleanup cmakelists.txt
karthikeyann Mar 26, 2025
5754b3c
replace env variable with gflags and CudfOptions
karthikeyann Mar 26, 2025
538398e
replace gpu batch size env variable with a QueryConfig entry
karthikeyann Mar 26, 2025
7570d56
Add back optional debug printing of plans
devavret Mar 26, 2025
702cebe
Add a table sizing method that does not need stream
devavret Mar 26, 2025
e633371
Misc review changes
devavret Mar 26, 2025
9ef1dff
replace env variables with gflags
karthikeyann Mar 31, 2025
05c4465
cleanup
karthikeyann Mar 31, 2025
b5c9901
Merge pull request #92 from karthikeyann/enh-memory_cudfvector
devavret Mar 31, 2025
ca349b9
rename to cudfTableScanEnabled
karthikeyann Mar 31, 2025
86738ed
Add clang tidy
devavret Mar 31, 2025
7b18528
remove aacidental flags added to all of velox
devavret Mar 31, 2025
b9e56ae
Merge remote-tracking branch 'rapids/cmake-upstreaming' into cleanup-…
karthikeyann Apr 1, 2025
bcf1d88
cleanup more includes
karthikeyann Apr 1, 2025
a1cc5fa
Merge branch 'velox-cudf' of github.com:rapidsai/velox-private into c…
karthikeyann Apr 1, 2025
7483674
revert local changes
karthikeyann Apr 1, 2025
fcf6afc
remove re2
karthikeyann Apr 1, 2025
6d48542
cleanup
karthikeyann Apr 1, 2025
5114e07
style fix
karthikeyann Apr 1, 2025
8192488
Trim extra items from nvtx range name
devavret Apr 2, 2025
5ce0269
fix bug of sticky payload
devavret Apr 2, 2025
934e8fd
Add plan node id to range string
devavret Apr 2, 2025
074aa5e
Move parquet connector range to nvtx domain and display table name
devavret Apr 2, 2025
737464c
README updates
bdice Apr 2, 2025
aa750f8
Fix sentence
bdice Apr 2, 2025
c0c5851
Fix typo in comment.
bdice Apr 2, 2025
780da70
Merge pull request #95 from karthikeyann/cleanup-utilities
karthikeyann Apr 2, 2025
fd34c6b
add support to use non-output column names in filter
karthikeyann Apr 2, 2025
b43b6f9
Merge branch 'velox-cudf' into TableScanFilter
karthikeyann Apr 2, 2025
dd7d1fb
fix merge issue
karthikeyann Apr 2, 2025
199ec92
Merge branch 'velox-cudf' into enh-nvtx-cleanup
karthikeyann Apr 3, 2025
850d7ae
Merge pull request #97 from rapidsai/enh-nvtx-cleanup
karthikeyann Apr 3, 2025
e00a541
Merge pull request #96 from bdice/velox-cudf-readme
karthikeyann Apr 3, 2025
d8dbbf2
Merge branch 'velox-cudf' of github.com:rapidsai/velox-private into T…
karthikeyann Apr 3, 2025
76d313f
change args to benchmark
karthikeyann Apr 3, 2025
5f063ae
update cudf upto PR #18395
karthikeyann Apr 3, 2025
1584c38
fix merge issue
karthikeyann Apr 3, 2025
1b8654b
add comments
karthikeyann Apr 3, 2025
e2c3a8b
revert add support to use non-output column names in filter
karthikeyann Apr 3, 2025
d0b9f4f
style fix
karthikeyann Apr 3, 2025
531c71c
disable related unit test
karthikeyann Apr 3, 2025
61f5368
update kvikio, rmm, rapids-cmake
karthikeyann Apr 3, 2025
5358a1e
Add ARM support.
bdice Apr 3, 2025
41e32c6
revert cmake
karthikeyann Apr 3, 2025
59cf040
Merge pull request #98 from karthikeyann/update_cudf_host_decompress_Fix
karthikeyann Apr 4, 2025
7f723db
cmake min required
devavret Apr 7, 2025
bee2494
Make sure last operator from task produces velox RowVector
devavret Apr 7, 2025
7d5e04a
Merge pull request #100 from bdice/arm-support
bdice Apr 7, 2025
39e1816
Merge pull request #94 from karthikeyann/TableScanFilter
karthikeyann Apr 7, 2025
e1d0b55
Rename codeowners team.
bdice Apr 7, 2025
5575205
Cudf driver adapter without storing plan nodes
devavret Apr 8, 2025
01f1aaf
re-fix conversion to RowVector in sink
devavret Apr 8, 2025
a6c99ff
add link scope to velox_cudf_exec
devavret Apr 9, 2025
8b7992c
Merge branch 'cmake-upstreaming' into compilestate-fix-gluten
devavret Apr 9, 2025
5fce74a
Cleanup CudfVector.cpp and the most obvious bad comment I had
devavret Apr 9, 2025
f2075f3
Cleanup comments in LocalPartition
devavret Apr 9, 2025
d6d6029
Cleanup comments in CudfHashAggregation
devavret Apr 9, 2025
aac3984
Add comments in ExpressionEvaluator
devavret Apr 9, 2025
b7fcf6d
Missed tostring change in CudfLocalPartition.
devavret Apr 9, 2025
46e2db7
Merge pull request #101 from bdice/velox-cudf-codeowners
bdice Apr 9, 2025
e5b33ae
fix compile
devavret Apr 9, 2025
68aac81
fix cmake dependency
karthikeyann Apr 9, 2025
0eb3354
add velox_cudf_vector to velox_cudf_exec
karthikeyann Apr 10, 2025
d7bc13e
newline
karthikeyann Apr 10, 2025
993dde4
Merge pull request #102 from rapidsai/compilestate-fix-gluten
devavret Apr 10, 2025
649e8dd
Apply suggestions from code review
bdice Apr 10, 2025
e4bfa30
Merge pull request #104 from rapidsai/cleanup-docs-comments
bdice Apr 10, 2025
cdc6944
Add missed stream usage in scalar accessors causing use of default st…
devavret Apr 14, 2025
4892a81
Merge pull request #2 from rapidsai/scalar-stream
devavret Apr 15, 2025
4369cf3
Merge remote-tracking branch 'upstream/main' into update-velox-2025-0…
bdice Apr 18, 2025
8ee066e
Add Operator.h (needed for compilation to pass).
bdice Apr 18, 2025
8a1a1d7
Merge branch 'velox-cudf' into eager-groupby
devavret Apr 19, 2025
da0be93
Style
bdice Apr 20, 2025
903ccdf
Temporarily disable restore action.
bdice Apr 20, 2025
aba48bc
Merge branch 'update-velox-2025-04-17' into eager-groupby
devavret Apr 21, 2025
c74cdf0
Run clang tidy on all files in experimental/cudf
devavret Apr 21, 2025
35a6fd8
Merge branch 'clangtidy-cleanups-0' into eager-groupby
devavret Apr 21, 2025
7ed1fc5
Re-add stash restore
bdice Apr 21, 2025
2926e7e
Revert "Re-add stash restore"
bdice Apr 21, 2025
92ce4ef
add nvtx range
devavret Apr 21, 2025
5ec5b0b
Merge pull request #3 from bdice/update-velox-2025-04-17
bdice Apr 21, 2025
859c8ff
enable avg agg in streaming partial groupby
devavret Apr 22, 2025
fe47687
Merge branch 'velox-cudf' into clangtidy-cleanups-0
devavret Apr 23, 2025
fc51acf
Merge pull request #4 from rapidsai/clangtidy-cleanups-0
devavret Apr 23, 2025
05e8830
Fix issue with reordering of results. Issue exposed with Q1 which has…
devavret Apr 24, 2025
1e7700b
Use stream again instead of default
devavret Apr 24, 2025
0c6684c
Make partial output a CudfVectorPtr so it can store its stream
devavret Apr 24, 2025
4fe7aa8
cleanup and reorganize addInput code
devavret Apr 24, 2025
6cf35b8
cleanup final groupby handling
devavret Apr 24, 2025
a8ba91f
Fix for when grouping keys are moved around
devavret Apr 24, 2025
0a53fea
Fix issue where we were using groupby for distinct
devavret Apr 25, 2025
a05f9f0
Re-enable progressive compaction for distinct that we got for free wh…
devavret Apr 25, 2025
f25fcfc
Fix the same output reordering issue in distinct
devavret Apr 25, 2025
18ec27d
Support spark operator and companion function agg name
jinchengchenghh Apr 29, 2025
30c71f8
fix avg name
jinchengchenghh Apr 29, 2025
eab0108
address comments for map
jinchengchenghh Apr 29, 2025
134426c
Merge pull request #5 from jinchengchenghh/rapidvelox
karthikeyann Apr 30, 2025
6b829cc
Merge remote-tracking branch 'upstream/main' into update-velox-2025-0…
bdice Apr 30, 2025
2f53b21
Merge remote-tracking branch 'upstream/main' into update-velox-2025-0…
bdice Apr 30, 2025
16196a6
Update target name
bdice Apr 30, 2025
4ad6fdc
add batching in CudfToVelox
karthikeyann Apr 30, 2025
8814526
copy and enable OrderByTest.outputBatchRows
karthikeyann Apr 30, 2025
b71dc62
style fix
karthikeyann Apr 30, 2025
b3ff196
update the deprecated includes
karthikeyann Apr 30, 2025
ff07e8b
Merge pull request #8 from karthikeyann/fea-add_to_velox_batching
karthikeyann May 1, 2025
d860452
fix merge issue, remove duplicated lines
karthikeyann May 1, 2025
4566a32
Merge pull request #7 from bdice/update-velox-2025-04-30
bdice May 2, 2025
20f95af
Support query config for partial aggregation memory
devavret May 2, 2025
78fa582
Add stats and gtest
devavret May 2, 2025
9b42812
Merge branch 'velox-cudf' into eager-groupby
devavret May 2, 2025
9672458
Cleanup debug prints and comments
devavret May 2, 2025
ed34ab4
Allow setting partial agg memory limit in query benchmarks
devavret May 5, 2025
01828f6
Comment cleanup
devavret May 5, 2025
38dfbb3
Fix Q13
devavret May 5, 2025
48372d1
Add debug utility function to print the cudf table
jinchengchenghh May 6, 2025
38177fc
address comments
jinchengchenghh May 6, 2025
e65941f
fix
jinchengchenghh May 6, 2025
4b29b5b
Merge pull request #12 from jinchengchenghh/rapidvelox
bdice May 7, 2025
d7af6c9
Merge pull request #6 from rapidsai/eager-groupby
devavret May 8, 2025
9409ee9
add recordOutputStats to CudfLocalPartition
May 12, 2025
773208d
Fix the companion function changes step behavior (#13)
jinchengchenghh May 21, 2025
7968a10
Merge branch 'update-velox-2025-05-20' into merge-velox-2025-05-20
karthikeyann May 22, 2025
c6d655c
Merge pull request #20 from karthikeyann/merge-velox-2025-05-20
karthikeyann May 22, 2025
d0eb8e2
Merge branch 'velox-cudf' into velox-cudf
karthikeyann May 22, 2025
8ed8a84
Merge pull request #17 from VinithKrishnan/velox-cudf
karthikeyann May 22, 2025
a3a2b6f
Fix replacement of localexchange when followed by non-cudf operator (…
devavret May 27, 2025
44b93bd
Fix null behavior issue in hashJoin (#15)
karthikeyann May 27, 2025
917dafa
Merge branch 'main-meta' into merge-from-upstream-28-05-2025
devavret May 28, 2025
5fede81
Restore ccache
bdice May 28, 2025
437d0da
Use non-hidden directory for ccache artifacts.
bdice May 28, 2025
344f9e7
Revert "Restore ccache"
bdice May 28, 2025
8f2db54
Try running builds and tests on GPU.
bdice May 28, 2025
312e17e
Reapply "Restore ccache"
bdice May 28, 2025
d0acbb8
Enable actions: read permission.
bdice May 28, 2025
55963e8
Revert "Provide token."
bdice May 28, 2025
313a93d
Revert actions: read permissions.
bdice May 28, 2025
1ce567e
Merge pull request #22 from bdice/restore-ccache
bdice May 28, 2025
aa2c04a
Merge branch 'velox-cudf' into gpu-ci
bdice May 28, 2025
17e97db
Merge branch 'velox-cudf' into merge-from-upstream-28-05-2025
bdice May 28, 2025
a8f645d
Merge pull request #25 from devavret/merge-from-upstream-28-05-2025
bdice May 28, 2025
5a00d43
Merge branch 'velox-cudf' into gpu-ci
bdice May 28, 2025
24e41be
Merge branch 'main-rapids-new' into merge-from-upstream-04-06-2025
devavret Jun 9, 2025
adfd186
fix TPCH Q12 switch
jinchengchenghh Jun 11, 2025
c53d6aa
Merge pull request #28 from devavret/merge-from-upstream-04-06-2025
GregoryKimball Jun 11, 2025
6a06f56
add nulls for apply_filter condition
karthikeyann Jun 12, 2025
7fb8295
apply review comments
karthikeyann Jun 12, 2025
be1bb46
address review comments
karthikeyann Jun 12, 2025
b07d7ed
use IILE
karthikeyann Jun 16, 2025
c305ce0
Merge branch 'velox-cudf' into fix-cudfFilterProject
karthikeyann Jun 16, 2025
5d42f26
Merge branch 'velox-cudf' into gpu-ci
bdice Jun 16, 2025
194df79
Fix try_cast to pass TPCH Q20 (#32)
jinchengchenghh Jun 18, 2025
fc41d87
Merge branch 'velox-cudf' into gpu-ci
bdice Jun 18, 2025
3d157b9
Merge commit '0c4aedd0d0722788cb246cbf5bea577bf11e2e1c' of github.com…
karthikeyann Jun 18, 2025
9e55e2f
fix yamllint errors
karthikeyann Jun 18, 2025
57aa6e7
fix missed yamllint error
karthikeyann Jun 18, 2025
45a03c0
Merge pull request #36 from karthikeyann/fix-build-dep
karthikeyann Jun 19, 2025
5a3079e
Merge branch 'velox-cudf' into fix-cudfFilterProject
karthikeyann Jun 19, 2025
525ceca
Merge branch 'velox-cudf' into fix_switch
karthikeyann Jun 19, 2025
7bc1f16
Merge branch 'velox-cudf' into gpu-ci
bdice Jun 19, 2025
56d1d8d
Merge pull request #34 from karthikeyann/fix-cudfFilterProject
karthikeyann Jun 24, 2025
23aa0b4
Merge pull request #33 from jinchengchenghh/fix_switch
karthikeyann Jun 24, 2025
c3b5ef5
Fix TPCH Q7 year output type (#35)
jinchengchenghh Jun 25, 2025
b228ff3
Fix TPCH Q1 col not found (#31)
jinchengchenghh Jun 25, 2025
81cf777
Merge branch 'velox-cudf' into gpu-ci
bdice Jun 25, 2025
e13af59
Apply pre-commit fixes
bdice Jun 25, 2025
4f9df66
Rerun CI
bdice Jun 26, 2025
f3a4a53
Merge pull request #23 from bdice/gpu-ci
bdice Jul 11, 2025
5dcff93
feat(Stats): added totalScanTime and totalRemainingFilterTime stats t…
VinithKrishnan Jul 11, 2025
a541905
Merge branch 'velox_2025_07_18' into update-to-2025_07_18
karthikeyann Jul 19, 2025
8011cc0
Fix shellcheck warning: remove unused CUDA_ARCH variable
karthikeyann Jul 19, 2025
b75bb24
fix merge issues
karthikeyann Jul 29, 2025
98655a6
cleanup after HashJoinTest.cpp update
karthikeyann Jul 29, 2025
8b9678a
Merge pull request #40 from karthikeyann/update-to-2025_07_18
karthikeyann Jul 29, 2025
fc5fb17
Merge pull request #37 Fixes to enable gluten TPCH runs
karthikeyann Jul 30, 2025
01b0df7
add startswith, endswith, contains in expression
karthikeyann Jul 30, 2025
4761892
support importFromArrow for large utf-8 string
karthikeyann Jul 30, 2025
e07e9cd
Merge pull request #41 from karthikeyann/ep-velox-gluten-tpch-fixes
karthikeyann Jul 31, 2025
51fd261
Merge upstream/main 2025_08_31
karthikeyann Aug 31, 2025
36efb77
use ColumnStatsSpec instead of Aggregation node - fix CudfPlanBuilder
karthikeyann Aug 31, 2025
d850fcd
pre-commit failure fixes
karthikeyann Sep 1, 2025
2b5d1b4
update linux build from change from linux build base yml
karthikeyann Sep 1, 2025
c6324f2
Merge pull request #45 from karthikeyann/update-to-2025_08_31
karthikeyann Sep 1, 2025
67678c2
Merge remote-tracking branch 'upstream/main' into update_to_2025_09_01
karthikeyann Sep 2, 2025
53455e3
Merge pull request #47 from karthikeyann/update_to_2025_09_01
karthikeyann Sep 2, 2025
94698da
Merge branch 'velox-cudf' into 20250902-velox-update
devavret Sep 4, 2025
73ecb88
Merge pull request #55 from devavret/20250902-velox-update
devavret Sep 4, 2025
153cc4e
Merge commit '2cac6c6857deb62bd3cda239d54c03cc431e250c' into update-v…
karthikeyann Sep 6, 2025
44173ec
Merge branch 'main' into update-velox-2025_09_05
karthikeyann Sep 6, 2025
708ab1a
cleanup
karthikeyann Sep 6, 2025
1e6e7aa
Merge pull request #58 from karthikeyann/update-velox-2025_09_05
karthikeyann Sep 6, 2025
849ba03
local partition round robin batch (#60)
karthikeyann Sep 8, 2025
1588eb4
Large string support for importFromArrow (#59)
karthikeyann Sep 8, 2025
43dd371
Empty input agg avg (#52)
devavret Sep 10, 2025
5c7fdf4
Add support for coalesce expression (#56)
devavret Sep 10, 2025
d91f7e7
Merge branch 'main' of github.com:facebookincubator/velox into update…
karthikeyann Sep 10, 2025
d4d5963
fix merge, build issues
karthikeyann Sep 10, 2025
ca0f398
Merge pull request #66 from karthikeyann/update-velox-2025_09_10
karthikeyann Sep 10, 2025
4238051
Parquet connector refactor (#67)
karthikeyann Sep 11, 2025
5f4281d
Parquet connector refactor (#50)
devavret Sep 11, 2025
0b375b2
Upgrade cudf to 25.10 (commit 28 Aug 2025) (#65)
karthikeyann Sep 11, 2025
c7d0e59
Merge upstream main into velox-cudf
bdice May 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 19 additions & 13 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -22,31 +22,37 @@
# Only users that have write access to the repo can be added as owners.
# See the official docs for more details on syntax and precedence of rules: https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners#example-of-a-codeowners-file

# Velox-cuDF codeowners
* @rapidsai/velox-cudf-codeowners

# Build & CI
CMake/ @assignUser @majetideepak
/CMakeLists.txt @assignUser
*.cmake @majetideepak
**/CMakeLists.txt @majetideepak
scripts/ @assignUser @majetideepak
.github/ @assignUser @majetideepak
# CMake/ @assignUser @majetideepak
# /CMakeLists.txt @assignUser
# *.cmake @majetideepak
# **/CMakeLists.txt @majetideepak
# scripts/ @assignUser @majetideepak
# .github/ @assignUser @majetideepak

# Breeze
velox/experimental/breeze @dreveman
#velox/experimental/breeze @dreveman

# cuDF
velox/experimental/cudf @bdice @karthikeyann @devavret

# cuDF
velox/experimental/cudf/ @bdice @karthikeyann @devavret @mhaseeb123

# Parquet
velox/dwio/parquet/ @majetideepak
#velox/dwio/parquet/ @majetideepak

# Storage Adapters
velox/connectors/hive/storage_adapters/ @majetideepak
#velox/connectors/hive/storage_adapters/ @majetideepak

# Connectors
velox/connectors/ @majetideepak
#velox/connectors/ @majetideepak

# Caching
velox/common/caching/ @majetideepak
# velox/common/caching/ @majetideepak

# Spark Functions
velox/functions/sparksql/ @jinchengchenghh @rui-mo @zhli1142015
# # Spark Functions
# velox/functions/sparksql/ @jinchengchenghh @rui-mo @zhli1142015
17 changes: 17 additions & 0 deletions .github/copy-pr-bot.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Copyright (c) Facebook, Inc. and its affiliates.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Configuration file for `copy-pr-bot` GitHub App
# https://docs.gha-runners.nvidia.com/apps/copy-pr-bot/

enabled: true
224 changes: 224 additions & 0 deletions .github/disabled-workflows/benchmark.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,224 @@
# Copyright (c) Facebook, Inc. and its affiliates.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

name: Ubuntu Benchmark

on:
pull_request:
paths:
- .github/workflows/benchmark.yml
- scripts/ci/benchmark-requirements.txt
- scripts/setup-ubuntu.sh

push:
branches: [main]

permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.sha }}
cancel-in-progress: true

defaults:
run:
shell: bash

jobs:
benchmark:
if: github.repository == 'facebookincubator/velox'
runs-on: 8-core-ubuntu-22.04
env:
CCACHE_DIR: ${{ github.workspace }}/ccache/
CCACHE_BASEDIR: ${{ github.workspace }}
BINARY_DIR: ${{ github.workspace }}/benchmarks/
CONTENDER_OUTPUT_PATH: ${{ github.workspace }}/benchmark-results/contender/
INSTALL_PREFIX: ${{ github.workspace }}/dependencies
steps:

- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
path: velox

persist-credentials: false

- name: Restore Dependencies
uses: actions/cache/restore@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4
id: restore-deps
with:
path: ${{ env.INSTALL_PREFIX }}
key: dependencies-benchmark-${{ hashFiles('velox/scripts/setup-ubuntu.sh') }}

- name: Install apt dependencies
env:
UV_TOOL_BIN_DIR: /usr/local/bin
run: |
source velox/scripts/setup-ubuntu.sh
install_apt_deps

- name: Install compiled dependencies
if: ${{ steps.restore-deps.outputs.cache-hit != 'true' }}
env:
CCACHE_DISABLE: 'true'
run: |
source velox/scripts/setup-ubuntu.sh
run_and_time install_fmt
run_and_time install_protobuf
run_and_time install_boost
run_and_time install_fast_float
run_and_time install_folly
run_and_time install_stemmer
run_and_time install_thrift
run_and_time install_arrow

- name: Save Dependencies
if: ${{ steps.restore-deps.outputs.cache-hit != 'true' }}
uses: actions/cache/save@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4
with:
path: ${{ env.INSTALL_PREFIX }}
key: dependencies-benchmark-${{ hashFiles('velox/scripts/setup-ubuntu.sh') }}

- name: Restore ccache
uses: apache/infrastructure-actions/stash/restore@3354c1565d4b0e335b78a76aedd82153a9e144d4
id: restore-cache
with:
clean: true
path: ccache
key: ccache-benchmark

- name: Clear CCache Statistics
run: |
ccache -sz

- name: Build Contender Benchmarks
working-directory: velox
run: |
n_cores=$(nproc)
make benchmarks-basic-build NUM_THREADS=$n_cores MAX_HIGH_MEM_JOBS=$n_cores MAX_LINK_JOBS=$n_cores
mkdir -p ${BINARY_DIR}/contender/
cp -r --verbose _build/release/velox/benchmarks/basic/* ${BINARY_DIR}/contender/

- name: CCache after
run: |
ccache -vs

- name: Save ccache"
uses: apache/infrastructure-actions/stash/save@3354c1565d4b0e335b78a76aedd82153a9e144d4
with:
path: ccache
key: ccache-benchmark

- name: Install benchmark dependencies
run: |
python3 -m pip install -r velox/scripts/ci/benchmark-requirements.txt

- name: Run Benchmarks - Contender
working-directory: velox
run: |
make benchmarks-basic-run \
EXTRA_BENCHMARK_FLAGS="--binary_path ${BINARY_DIR}/contender/ --output_path ${CONTENDER_OUTPUT_PATH}"

- name: Upload result artifact
uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
with:
path: benchmark-results
name: benchmark-results
retention-days: 5

upload:
runs-on: ubuntu-latest
needs: benchmark
if: false && github.event_name == 'push'
permissions:
actions: read
statuses: write
steps:

- name: Download artifacts
uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5.0.0
with:
merge-multiple: true
path: /tmp/artifacts/

- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
path: velox
persist-credentials: false

- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: '3.10'
cache: pip
cache-dependency-path: velox/scripts/*

- name: Install dependencies
run: pip install -r velox/scripts/ci/benchmark-requirements.txt

- name: Upload results
env:
CONBENCH_URL: https://velox-conbench.voltrondata.run/
CONBENCH_MACHINE_INFO_NAME: GitHub-runner-8-core
CONBENCH_EMAIL: ${{ secrets.CONBENCH_EMAIL }}
CONBENCH_PASSWORD: ${{ secrets.CONBENCH_PASSWORD }}
CONBENCH_PROJECT_REPOSITORY: ${{ github.repository }}
CONBENCH_PROJECT_COMMIT: ${{ github.sha }}
RUN_ID: GHA-${{ github.run_id }}-${{ github.run_attempt }}
run: |
./velox/scripts/ci/benchmark-runner.py upload \
--run_id "$RUN_ID" \
--sha "$CONBENCH_PROJECT_COMMIT" \
--output_dir "/tmp/artifacts/contender/"

- name: Check the status of the upload
# Status functions like failure() only work in `if:`
if: failure()
id: status
run: echo "failed=true" >> $GITHUB_OUTPUT

- name: Create a GitHub Status on the contender commit (whether the upload was successful)
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
if: ${{ !cancelled() && steps.extract.conclusion != 'failure' }}
with:
script: |
let url = 'https://github.com/${{github.repository}}/actions/runs/${{ github.run_id }}'
let state = 'success'
let description = 'Result upload succeeded!'

if(${{ steps.status.outputs.failed == 'true' && true || false }}) {
state = 'failure'
description = 'Result upload failed!'
}

github.rest.repos.createCommitStatus({
owner: context.repo.owner,
repo: context.repo.repo,
sha: context.sha,
state: state,
target_url: url,
description: description,
context: 'Benchmark Result Upload'
})

- name: Create a GitHub Check benchmark report on the merged PR
env:
CONBENCH_URL: https://velox-conbench.voltrondata.run/
GITHUB_APP_ID: ${{ secrets.GH_APP_ID }}
GITHUB_APP_PRIVATE_KEY: ${{ secrets.GH_APP_PRIVATE_KEY }}
COMMIT_MESSAGE: ${{ github.event.head_commit.message }}
CONTENDER_SHA: ${{ github.sha }}
run: |
./velox/scripts/ci/benchmark-alert.py \
--contender-sha "$CONTENDER_SHA" \
--merge-commit-message "$COMMIT_MESSAGE" \
--z-score-threshold 50
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loading
Loading