Byte 0x80 not mapped to U+20AC (€) — mismatch with Java GBK

Java's "GBK" charset in both Oracle JDK and OpenJDK is implemented as CP936/Windows-936, which maps the single byte 0x80 to U+20AC (Euro sign, €). The current kernel treats 0x80 as an invalid lead byte (it falls outside [GBK_FIRST_BYTE_MIN, GBK_FIRST_BYTE_MAX] = [0x81, 0xFE]) and emits U+FFFD instead.

This causes a behavioral divergence from Spark's CPU-side StringDecode("GBK") on any JVM where GBK maps 0x80 → € (i.e., practically all modern JVMs). The testByte0x80 unit test acknowledges this JDK dependency — the test itself will fail on any JVM that maps 0x80 → € because the GPU always returns FFFD.

To match Java's GBK decoder, byte 0x80 should be decoded as U+20AC:

if (byte == 0x80) {
    // CP936/Windows-936 extension: 0x80 → U+20AC (EURO SIGN)
    int utf8_len = codepoint_to_utf8(0x20AC, out_ptr);
    if (out_ptr) { out_ptr += utf8_len; }
    output_size += utf8_len;
    i += 1;
} else if (byte <= 0x7F) {
    // ASCII ...

We verified this on the actual build JDK (OpenJDK 1.8.0_482): new String(new byte[]{(byte)0x80}, "GBK") returns U+FFFD, not U+20AC. We also tried adding a 0x80 → U+20AC special case in the kernel — the integration test (GPU vs CPU comparison with random binary data) failed immediately because the CPU side produces FFFD.

The testByte0x80 test uses decodeGbkJava() which dynamically queries the JDK behavior as ground truth, so it adapts to whichever JDK is in use. The claim that "practically all modern JVMs" map 0x80 to € does not hold for this JDK.

Static device table singleton — dangling pointer risk on CUDA/RMM re-init

gbk_table_device and gbk_table_init_flag are file-scope statics. std::once_flag is never reset. If CUDA or RMM is torn down and re-initialized within the same process (a common pattern in unit-test suites and JVM-embedded workloads), the sequence is:

1. First run: table is allocated with the stream from the first call.
2. RMM allocator is reset / CUDA context is destroyed (test teardown).
3. gbk_table_device's internal device pointer is now dangling.
4. Next call to decode_gbk: call_once sees the flag already set, skips initialization, and returns the dangling pointer.
5. The GPU kernel reads from freed device memory → silent corruption or crash.

Additionally, at process exit, the static destructor of gbk_table_device issues a cudaFree after CUDA is shut down, which can surface as a CUDA error in test logs.

Consider managing this table through the existing cudf resource lifecycle, or at minimum validating the device pointer on each call:

// Option: use a function-local static that ties lifetime to RMM resource
// Or check before returning:
CUDF_EXPECTS(gbk_table_device != nullptr, "GBK table not initialized");

Acknowledged. This is a known pattern in the project — other static singletons like the bloom filter hash seed table (bloom_filter.cu) follow the same approach. In spark-rapids-jni, the CUDA context and RMM allocator are not torn down and re-initialized within an executor process lifetime.

If this becomes a project-wide concern, it should be addressed holistically rather than in this PR alone.

Raw int charset parameter — no validation before JNI call

decode(ColumnView cv, int charset) accepts any arbitrary integer. In CharsetDecodeJni.cpp, this is directly cast to charset_type via static_cast<spark_rapids_jni::charset_type>(charset). An invalid value (e.g., 99) will reach the switch statement and hit default: CUDF_FAIL(...), but only after an unchecked cast from an integer to an enum.

Consider validating the input before passing to native code, or using an enum on the Java side:

public static ColumnVector decode(ColumnView cv, int charset) {
    if (charset != GBK) {
        throw new IllegalArgumentException("Unsupported charset: " + charset);
    }
    return new ColumnVector(decodeNative(cv.getNativeView(), charset));
}

Alternatively, exposing an enum-typed overload would make misuse a compile-time error.

Fair point. The C++ side already catches invalid values via CUDF_FAIL in the default switch branch, so there is no undefined behavior. The only caller is the GPU plugin which always passes CharsetDecode.GBK. Will add a Java-side validation as defense in depth.

Generator uses REPORT mode; test ground truth uses REPLACE mode

The generator uses CodingErrorAction.REPORT (throws MalformedInputException/UnmappableCharacterException, caught and mapped to 0xFFFD), while CharsetDecodeTest.decodeGbkJava uses CodingErrorAction.REPLACE. For standard GBK pairs these produce identical tables, but the generator does not handle standalone 0x80 at all (it only iterates first from 0x81 to 0xFE). This is fine for the double-byte table itself, but documents a discrepancy with the test's ground truth function worth noting:

• Generator: 0x80 not iterated → not in table.
• Java ground truth (decodeGbkJava): 0x80 decoded with REPLACE mode → returns € on most JVMs.

The generator comment should note that 0x80 → U+20AC needs separate handling in the kernel, since it is outside the generated table range.

Good observation on the mode difference. We investigated this thoroughly — the behavioral difference between REPORT and REPLACE modes is limited to:

1. Second byte 0x7F: REPLACE mode splits (lead, 0x7F) into two characters (FFFD + DEL), while REPORT mode on the isolated 2-byte buffer throws. The kernel excludes 0x7F from valid second bytes (second \!= 0x7F) to match REPLACE behavior.
2. Second byte 0xFF: REPLACE mode consumes (lead, 0xFF) as a pair (advance 2, emit FFFD), while the table range only covers 0x40-0xFE. The kernel handles this with a special case (second > GBK_SECOND_BYTE_MAX → UNICODE_REPLACEMENT, advance 2).

We validated this by running a Java simulation of the kernel logic against Java REPLACE-mode streaming decode on 100k random byte sequences with zero mismatches.

Regarding 0x80: as noted in the reply above, this JDK (OpenJDK 1.8.0) returns FFFD for standalone 0x80, not €. The test dynamically adapts to the JDK behavior.