-
Notifications
You must be signed in to change notification settings - Fork 80
Add a protocol buffer decode kernel #4107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
thirtiseven
wants to merge
112
commits into
NVIDIA:main
Choose a base branch
from
thirtiseven:protocol_buffer_jni_dev
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+9,494
−209
Open
Changes from 1 commit
Commits
Show all changes
112 commits
Select commit
Hold shift + click to select a range
4aa7dbb
ai draft
thirtiseven a319d10
style
thirtiseven 458b583
address comments
thirtiseven 5c1bbf4
address comments
thirtiseven 0445daa
address comments
thirtiseven 8ddbf96
address comments
thirtiseven c9eea59
copyrights
thirtiseven f27fd6e
Merge branch 'main' into protocol_buffer_jni_dev
thirtiseven 617471d
update, added more tests
thirtiseven bdfac06
Merge branch 'protocol_buffer_jni_dev' of https://github.com/thirtise…
thirtiseven 3701cdf
style
thirtiseven 1bf7660
address comments
thirtiseven b012421
multi column processing
thirtiseven ecd3a38
fix merge
thirtiseven c86f78c
Support enum, required and default values
thirtiseven 3b37d28
style
thirtiseven 46d2130
Support nested types
thirtiseven 5f89e60
performance optimization
thirtiseven e311b4b
single-pass-kernel, with debug log, met unbreakable wall
thirtiseven c52782c
delete debug log
thirtiseven 47973a7
check point before schema projection option A, PROTOBUF_SINGLE_PASS=1…
thirtiseven d2595b1
Merge flat and nested api for from_protobuf
thirtiseven aa3c852
Fix nested type support
thirtiseven 7e0e77d
add a tests
thirtiseven ac39a4e
Kernal code clean up
thirtiseven fd7ec66
style change
thirtiseven 828e3c0
clean up code
thirtiseven ca5a921
address comments
thirtiseven edbfd98
ai self review and comment addressed
thirtiseven 34bcf0b
ai self review and comment addressed
thirtiseven 632448b
address comments
thirtiseven 46ef3b0
bug fixs
thirtiseven 5438a17
clean up code
thirtiseven c009f62
style
thirtiseven 6152733
address comments
thirtiseven 48204d5
clean up and split big files
thirtiseven 89e6e8c
style
thirtiseven f37d3c9
style
thirtiseven 78b1c60
address cc comments
thirtiseven 661f085
codex review and address
thirtiseven 8e09a47
gemini review and address
thirtiseven aa2586f
Merge branch 'main' into protocol_buffer_jni_dev
thirtiseven 3789cfc
adapt new api
thirtiseven 33ddacd
add micro benchmark
thirtiseven c56ed5b
merge reduce+scan, defer error check, combined occ scan
thirtiseven 2b113c7
O1 field number lookup in nested
thirtiseven ce408d6
Remove perf-results
thirtiseven 38348cc
Batched scalar extraction (2D grid kernel)
thirtiseven 3647299
style
thirtiseven 6af1577
address greptile comments
thirtiseven 0d9d105
address comments
thirtiseven e861f40
address greptile comments
thirtiseven 2ff75a9
address greptile comments
thirtiseven f6ebffe
address greptile comments
thirtiseven 7a312d4
address greptile comments
thirtiseven 4fa362c
address greptile comments
thirtiseven 54fda94
style
thirtiseven aeaf50c
address greptile comments
thirtiseven d50f089
address greptile comments
thirtiseven 63bc13a
address greptile comments
thirtiseven 8316f75
address greptile comments
thirtiseven 8ae9547
address greptile comments
thirtiseven 756010d
address greptile comments
thirtiseven 64de422
address greptile comments
thirtiseven aae6bca
address comments
thirtiseven 6e1c4d2
address greptile comments
thirtiseven 9c6dd70
address greptile comments
thirtiseven 6b2f494
address greptile comments
thirtiseven e40d5a7
Lookup table for repeated message child scan
thirtiseven 4b4f6f9
reflection refactor
thirtiseven 66daed1
reflection refactor
thirtiseven cd1763b
bug fixes
thirtiseven 0213721
comment address
thirtiseven 5357378
comment address
thirtiseven c4b1507
address comments
thirtiseven 3852f53
address comments
thirtiseven ce29c11
address comments
thirtiseven 96b20e0
address comments
thirtiseven 87c5e99
address comments
thirtiseven fca9ea7
address comments
thirtiseven 8e5473c
address comments
thirtiseven 95e2586
address comments
thirtiseven 67f2db8
address greptile comments
thirtiseven 06c15ee
address greptile comments
thirtiseven 14c6644
bugfix
thirtiseven 7a06053
address comments
thirtiseven 809daa1
nits
thirtiseven 3067f0b
address comments
thirtiseven 8aa0bdf
style
thirtiseven a13433f
style
thirtiseven e0cd7f5
fix bug
thirtiseven 5ef84e9
address comments
thirtiseven 4817d06
address comments
thirtiseven bdd2e40
address comments
thirtiseven 6101a4b
address human comments
thirtiseven 0213433
port back refactor from pr 0
thirtiseven 6bcac2f
style
thirtiseven ff83290
address comments from part0
thirtiseven 35b1cb8
nghia style self-check
thirtiseven 82bf02a
backport suggestions
thirtiseven fe73ada
address comments
thirtiseven d01a25b
copyright
thirtiseven 54223ca
fix
thirtiseven 70a214b
apply comments suggestions
thirtiseven 6e00831
apply refactor on headers
thirtiseven 1754032
Merge remote-tracking branch 'origin/main' into protocol_buffer_jni_dev
thirtiseven 02b7303
Merge origin/main into protocol_buffer_jni_dev
thirtiseven 8ea16b5
Remove stale mr parameter from call sites
thirtiseven 2e91499
Apply part0 brace style to dev-only code
thirtiseven c1f13c2
Replace copy_varlen_data_kernel with cub::DeviceMemcpy::Batched
thirtiseven 33e30f7
Remove unused threads/blocks variables
thirtiseven 7508a80
style
thirtiseven File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| /* | ||
| * Copyright (c) 2025, NVIDIA CORPORATION. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| #include "cudf_jni_apis.hpp" | ||
| #include "dtype_utils.hpp" | ||
| #include "protobuf_simple.hpp" | ||
|
|
||
| #include <cudf/column/column_view.hpp> | ||
|
|
||
| extern "C" { | ||
|
|
||
| JNIEXPORT jlong JNICALL Java_com_nvidia_spark_rapids_jni_ProtobufSimple_decodeToStruct( | ||
| JNIEnv* env, jclass, jlong binary_input_view, jintArray field_numbers, jintArray type_ids, jintArray type_scales) | ||
| { | ||
| JNI_NULL_CHECK(env, binary_input_view, "binary_input_view is null", 0); | ||
| JNI_NULL_CHECK(env, field_numbers, "field_numbers is null", 0); | ||
| JNI_NULL_CHECK(env, type_ids, "type_ids is null", 0); | ||
| JNI_NULL_CHECK(env, type_scales, "type_scales is null", 0); | ||
|
|
||
| JNI_TRY | ||
| { | ||
| cudf::jni::auto_set_device(env); | ||
| auto const* input = reinterpret_cast<cudf::column_view const*>(binary_input_view); | ||
| cudf::jni::native_jintArray n_field_numbers(env, field_numbers); | ||
| cudf::jni::native_jintArray n_type_ids(env, type_ids); | ||
| cudf::jni::native_jintArray n_type_scales(env, type_scales); | ||
| if (n_field_numbers.size() != n_type_ids.size() || n_field_numbers.size() != n_type_scales.size()) { | ||
| JNI_THROW_NEW(env, | ||
| cudf::jni::ILLEGAL_ARG_EXCEPTION_CLASS, | ||
| "fieldNumbers/typeIds/typeScales must be the same length", | ||
| 0); | ||
| } | ||
|
|
||
| std::vector<int> field_nums(n_field_numbers.begin(), n_field_numbers.end()); | ||
| std::vector<cudf::data_type> out_types; | ||
| out_types.reserve(n_type_ids.size()); | ||
| for (int i = 0; i < n_type_ids.size(); ++i) { | ||
| out_types.emplace_back(cudf::jni::make_data_type(n_type_ids[i], n_type_scales[i])); | ||
| } | ||
|
|
||
| auto result = | ||
| spark_rapids_jni::decode_protobuf_simple_to_struct(*input, field_nums, out_types); | ||
| return cudf::jni::release_as_jlong(result); | ||
| } | ||
| JNI_CATCH(env, 0); | ||
| } | ||
|
|
||
| } // extern "C" | ||
|
|
||
|
|
||
|
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.