Bypassing JSON batch tuple construction

[iZarrios]

Summary

Scalar LLM functions (llm_complete, llm_filter, etc.) currently materialize each batch of context_columns as a new nlohmann::json object before prompt rendering. This creates a full in-memory copy of the batch, which is then serialized again into a prompt string. The intermediate JSON representation is largely unnecessary and adds avoidable allocation and copying overhead.

Current behavior

Data flow today

1. Chunk → JSON — CastVectorOfStructsToJson builds a JSON representation of context_columns from DuckDB vectors (llm_complete/implementation.cpp, llm_filter/implementation.cpp).
2. Batch slicing → JSON copy — BatchAndComplete calls BuildBatchTuples, which constructs a new JSON array/object per batch by copying metadata and slicing each column's "data" array:

// src/functions/scalar/scalar.cpp
nlohmann::json BuildBatchTuples(const nlohmann::json& tuples, int start_index, int batch_size) {
    auto batch_tuples = nlohmann::json::array();
    for (auto i = 0; i < static_cast<int>(tuples.size()); i++) {
        batch_tuples.push_back(nlohmann::json::object());
        for (const auto& item: tuples[i].items()) {
            if (item.key() != "data") {
                batch_tuples[i][item.key()] = item.value();
                continue;
            }
            for (auto j = 0; j < batch_size && start_index + j < static_cast<int>(item.value().size()); j++) {
                // ... push_back slices into batch_tuples[i]["data"]
            }
        }
    }
    return batch_tuples;
}

3. JSON → string — Complete → PromptManager::Render → ConstructInputTuples* reads the batch JSON and serializes it into XML / Markdown / JSON prompt text.

Cost

For each batch iteration:

• Allocate and populate a full JSON tree for the batch (column metadata + sliced data arrays).
• Walk that JSON again to produce the final tuple string for the LLM prompt.
• Repeat for every batch until all rows are processed (including batch-size retries on token-limit errors).

On large inputs or wide context_columns, this doubles (or more) memory traffic for data that already lives in DuckDB vectors / an existing JSON structure.

Proposed direction

Bypass intermediate batch JSON construction and serialize tuple prompt text directly from the underlying vector / column views.

Possible approach:

1. Introduce a batch-aware API on PromptManager (or a dedicated serializer) that accepts:
   • column metadata (names, types, etc.)
   • a view over row ranges [start_index, start_index + batch_size) per column
   • TupleFormat
2. Refactor BatchAndComplete to pass indices/ranges instead of calling BuildBatchTuples.
3. Keep JSON-based paths only where truly required (e.g. media columns, transcription side paths) rather than as the default for tabular tuple data.

• Related code path: ScalarFunctionBase::BatchAndComplete → Complete → PromptManager::Render.
• Similar batching patterns may exist in aggregate functions; evaluate whether the same optimization applies there in a follow-up issue.