Add GBK charset decoding kernel for StringDecode

[thirtiseven]

Add a GPU kernel that decodes GBK-encoded binary data to UTF-8 strings, matching Java's GBK CharsetDecoder behavior for use with Spark's StringDecode expression.

• Lookup table generated from Java's GBK CharsetDecoder (REPORT mode), with generator script included for reproducibility
• Two-pass kernel using cudf::strings::detail::make_strings_children
• Invalid byte sequences replaced with U+FFFD
• Second byte 0x7F excluded from pair consumption (matches Java)
• Second byte 0xFF consumed as pair but mapped to U+FFFD (matches Java)
• Java unit test with full 24066 byte-pair end-to-end verification

[greptile-apps]

Greptile Summary

This PR implements a two-pass CUDA kernel that decodes GBK-encoded LIST<UINT8> columns to UTF-8 strings, backed by a 24 066-entry lookup table generated from Java's GBK CharsetDecoder. The implementation correctly handles ASCII pass-through, invalid lead bytes, truncated sequences, and the 0x7F/0xFF second-byte edge cases documented in the PR description.

Confidence Score: 5/5

PR is safe to merge; all prior review concerns are resolved and the only remaining finding is a P2 test coverage gap.

All P0/P1 concerns from prior rounds (byte 0x80 mapping, static singleton lifecycle, Java-side validation, REPORT vs REPLACE mode) have been addressed or confirmed as non-issues on the target JDK. The single remaining gap — no explicit test for {valid_lead, 0xFF} second byte — is P2 and does not block merge.

src/test/java/com/nvidia/spark/rapids/jni/CharsetDecodeTest.java — missing second=0xFF test case in testAllGbkDoubleBytePairs

Important Files Changed

Filename	Overview
src/main/cpp/src/charset_decode.cu	Core decode kernel with correct two-pass GBK logic; handles ASCII, double-byte pairs, 0x7F exclusion, 0xFF special case, and truncated sequences
src/test/java/com/nvidia/spark/rapids/jni/CharsetDecodeTest.java	Good test coverage including null rows, invalid bytes, and 24 066 pair sweep, but missing explicit test for second byte 0xFF and creates redundant decoder objects in inner loop
src/main/cpp/scripts/GenerateGbkTable.java	Correct REPORT-mode table generator; 0x7F entry is generated but never consulted by kernel (harmless, but minor inconsistency)
src/main/cpp/src/CharsetDecodeJni.cpp	Thin JNI bridge with correct cast and null check; delegates entirely to decode_charset
src/main/cpp/src/charset_decode.hpp	Clean header exposing charset_type enum and decode_charset declaration
src/main/cpp/src/gbk_to_unicode_table.inc	Generated 24 066-entry uint16_t lookup table; correct index layout matching kernel constants
src/main/java/com/nvidia/spark/rapids/jni/CharsetDecode.java	Java API with GBK constant and input validation guard before JNI call
src/main/cpp/CMakeLists.txt	Adds charset_decode.cu to the build; low-risk build change

Sequence Diagram

sequenceDiagram
    participant Java as CharsetDecode.decode()
    participant JNI as CharsetDecodeJni.cpp
    participant CPP as charset_decode.cu
    participant Once as get_gbk_table (once_flag)
    participant GPU as gbk_decode_fn kernel

    Java->>JNI: decodeNative(nativeColumnView, charset=GBK)
    JNI->>CPP: decode_charset(input, GBK, stream, mr)
    CPP->>Once: get_gbk_table(stream)
    Once-->>CPP: device ptr to 24066-entry table (copied once)
    CPP->>GPU: make_strings_children pass 1 (d_chars=null, compute sizes)
    GPU->>GPU: per-row: walk bytes, lookup table, accumulate output_size
    GPU-->>CPP: d_sizes[]
    CPP->>GPU: make_strings_children pass 2 (d_chars=allocated, write bytes)
    GPU->>GPU: per-row: walk bytes, lookup table, write UTF-8
    GPU-->>CPP: output chars buffer
    CPP->>CPP: make_strings_column(offsets, chars, null_count, copy_bitmask)
    CPP-->>JNI: unique_ptr<cudf::column>
    JNI-->>Java: ColumnVector (UTF-8 strings)

Loading

_{Reviews (3): Last reviewed commit: "address comments" | Re-trigger Greptile}

[greptile-apps]

Byte 0x80 not mapped to U+20AC (€) — mismatch with Java GBK

Java's "GBK" charset in both Oracle JDK and OpenJDK is implemented as CP936/Windows-936, which maps the single byte 0x80 to U+20AC (Euro sign, €). The current kernel treats 0x80 as an invalid lead byte (it falls outside [GBK_FIRST_BYTE_MIN, GBK_FIRST_BYTE_MAX] = [0x81, 0xFE]) and emits U+FFFD instead.

This causes a behavioral divergence from Spark's CPU-side StringDecode("GBK") on any JVM where GBK maps 0x80 → € (i.e., practically all modern JVMs). The testByte0x80 unit test acknowledges this JDK dependency — the test itself will fail on any JVM that maps 0x80 → € because the GPU always returns FFFD.

To match Java's GBK decoder, byte 0x80 should be decoded as U+20AC:

if (byte == 0x80) {
    // CP936/Windows-936 extension: 0x80 → U+20AC (EURO SIGN)
    int utf8_len = codepoint_to_utf8(0x20AC, out_ptr);
    if (out_ptr) { out_ptr += utf8_len; }
    output_size += utf8_len;
    i += 1;
} else if (byte <= 0x7F) {
    // ASCII ...

[thirtiseven]

We verified this on the actual build JDK (OpenJDK 1.8.0_482): new String(new byte[]{(byte)0x80}, "GBK") returns U+FFFD, not U+20AC. We also tried adding a 0x80 → U+20AC special case in the kernel — the integration test (GPU vs CPU comparison with random binary data) failed immediately because the CPU side produces FFFD.

The testByte0x80 test uses decodeGbkJava() which dynamically queries the JDK behavior as ground truth, so it adapts to whichever JDK is in use. The claim that "practically all modern JVMs" map 0x80 to € does not hold for this JDK.

[greptile-apps]

Static device table singleton — dangling pointer risk on CUDA/RMM re-init

gbk_table_device and gbk_table_init_flag are file-scope statics. std::once_flag is never reset. If CUDA or RMM is torn down and re-initialized within the same process (a common pattern in unit-test suites and JVM-embedded workloads), the sequence is:

1. First run: table is allocated with the stream from the first call.
2. RMM allocator is reset / CUDA context is destroyed (test teardown).
3. gbk_table_device's internal device pointer is now dangling.
4. Next call to decode_gbk: call_once sees the flag already set, skips initialization, and returns the dangling pointer.
5. The GPU kernel reads from freed device memory → silent corruption or crash.

Additionally, at process exit, the static destructor of gbk_table_device issues a cudaFree after CUDA is shut down, which can surface as a CUDA error in test logs.

Consider managing this table through the existing cudf resource lifecycle, or at minimum validating the device pointer on each call:

// Option: use a function-local static that ties lifetime to RMM resource
// Or check before returning:
CUDF_EXPECTS(gbk_table_device != nullptr, "GBK table not initialized");

[thirtiseven]

Acknowledged. This is a known pattern in the project — other static singletons like the bloom filter hash seed table (bloom_filter.cu) follow the same approach. In spark-rapids-jni, the CUDA context and RMM allocator are not torn down and re-initialized within an executor process lifetime.

If this becomes a project-wide concern, it should be addressed holistically rather than in this PR alone.

[greptile-apps]

Raw int charset parameter — no validation before JNI call

decode(ColumnView cv, int charset) accepts any arbitrary integer. In CharsetDecodeJni.cpp, this is directly cast to charset_type via static_cast<spark_rapids_jni::charset_type>(charset). An invalid value (e.g., 99) will reach the switch statement and hit default: CUDF_FAIL(...), but only after an unchecked cast from an integer to an enum.

Consider validating the input before passing to native code, or using an enum on the Java side:

public static ColumnVector decode(ColumnView cv, int charset) {
    if (charset != GBK) {
        throw new IllegalArgumentException("Unsupported charset: " + charset);
    }
    return new ColumnVector(decodeNative(cv.getNativeView(), charset));
}

Alternatively, exposing an enum-typed overload would make misuse a compile-time error.

[thirtiseven]

Fair point. The C++ side already catches invalid values via CUDF_FAIL in the default switch branch, so there is no undefined behavior. The only caller is the GPU plugin which always passes CharsetDecode.GBK. Will add a Java-side validation as defense in depth.

[greptile-apps]

Generator uses REPORT mode; test ground truth uses REPLACE mode

The generator uses CodingErrorAction.REPORT (throws MalformedInputException/UnmappableCharacterException, caught and mapped to 0xFFFD), while CharsetDecodeTest.decodeGbkJava uses CodingErrorAction.REPLACE. For standard GBK pairs these produce identical tables, but the generator does not handle standalone 0x80 at all (it only iterates first from 0x81 to 0xFE). This is fine for the double-byte table itself, but documents a discrepancy with the test's ground truth function worth noting:

• Generator: 0x80 not iterated → not in table.
• Java ground truth (decodeGbkJava): 0x80 decoded with REPLACE mode → returns € on most JVMs.

The generator comment should note that 0x80 → U+20AC needs separate handling in the kernel, since it is outside the generated table range.

[thirtiseven]

Good observation on the mode difference. We investigated this thoroughly — the behavioral difference between REPORT and REPLACE modes is limited to:

1. Second byte 0x7F: REPLACE mode splits (lead, 0x7F) into two characters (FFFD + DEL), while REPORT mode on the isolated 2-byte buffer throws. The kernel excludes 0x7F from valid second bytes (second \!= 0x7F) to match REPLACE behavior.
2. Second byte 0xFF: REPLACE mode consumes (lead, 0xFF) as a pair (advance 2, emit FFFD), while the table range only covers 0x40-0xFE. The kernel handles this with a special case (second > GBK_SECOND_BYTE_MAX → UNICODE_REPLACEMENT, advance 2).

We validated this by running a Java simulation of the kernel logic against Java REPLACE-mode streaming decode on 100k random byte sequences with zero mismatches.

Regarding 0x80: as noted in the reply above, this JDK (OpenJDK 1.8.0) returns FFFD for standalone 0x80, not €. The test dynamically adapts to the JDK behavior.