Default empty array() return type to ARRAY<VARCHAR>

[ahkcs]

Description

PPL's array() no-arg form currently returns ARRAY<NULL> (and ARRAY<UNKNOWN> when all operands are typeless nulls). The Calcite engine's in-process executor tolerates either: ArrayImplementor.internalCast only consumes the element type when there are elements to cast, so an empty list flows straight through to ExprCollectionValue regardless of the declared element type.

The analytics-engine route is stricter. When isthmus walks a RexCall like mvjoin(array(), '-'), it eventually feeds the first operand's type to io.substrait.isthmus.TypeConverter.toSubstrait, which throws UnsupportedOperationException: Unable to convert the type UNKNOWN. Substrait has no on-wire encoding for NULL/UNKNOWN element types, so the planner can't serialize the call at all. Two ITs hit this directly:

• CalciteArrayFunctionIT.testMvjoinWithEmptyArray
• CalciteArrayFunctionIT.testMvdedupWithEmptyArray

Substituting VARCHAR when the inferred element type is NULL or UNKNOWN gives the call a substrait-serializable type without affecting any value computation — the result list is empty either way.

Test plan

• :core:test --tests "*ArrayFunction*" — passes (no existing test asserted on the empty-array element type).
• CalciteArrayFunctionIT force-routed through the analytics-engine path via [Analytics Backend / DataFusion] Onboard PPL array constructor and 8 multivalue (mv) functions to analytics-engine route OpenSearch#21554's plugin set — pass-rate 26/60 → 28/60, with testMvjoinWithEmptyArray and testMvdedupWithEmptyArray newly green.

Companion to opensearch-project/OpenSearch#21554.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 22d0cce)

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ No major issues detected

[dai-chen]

Can we add unit test in ArrayFunctionImplTest?

[dai-chen]

np: If possible, can we make the comment more concise. I think it's okay to leave most context/details in PR description.

[ahkcs]

Trimmed in 22d0cce — comment now just points to the PR description.

[penghuo]

The idea is CAST(ARRAY[] to VARCHAR[])? Add an UT to cover this assumption.
Question, Is it limitation of Substrait, becuase it does not supoort UNKNOW?

[penghuo]

Refer PostgreSQL, it does not allow construct empty ARRAY[].

https://www.postgresql.org/docs/current/sql-expressions.html#SQL-SYNTAX-ARRAY-CONSTRUCTORS

You can construct an empty array, but since it's impossible to have an array with no type, you must explicitly cast your empty array to the desired type. For example:

SELECT ARRAY[]::integer[];
 array
-------
 {}
(1 row)

But in PPL, we allow create emtpy array(). So which means we implict covert to varchar[] in future, right?

[ahkcs]

Yes — semantically equivalent to CAST(ARRAY[] AS VARCHAR[]). The fallback path adds an empty-typed-array case Calcite/Postgres handle via explicit cast, but PPL surface allows bare array() so we do it implicitly.

Yes, Substrait limitation — Substrait's Type proto has no encoding for UNKNOWN (or Calcite's NULL SqlTypeName as an element type). When isthmus' TypeConverter.toSubstrait walks an ARRAY<UNKNOWN> it can't produce a valid wire type and throws UnsupportedOperationException: Unable to convert the type UNKNOWN. The Calcite-engine local executor is more lenient because it never serializes the type — it just iterates the empty List<Object> and never reads the element type.

UT added in aa82704 — testReturnTypeForEmptyCallIsVarcharArray and testReturnTypeForAllNullOperandsIsVarcharArray cover both fallback paths (empty operand list and typeless-NULL operand). testReturnTypeForIntegerOperandPreservesType is the regression guard that confirms concrete element types pass through unchanged.

[ahkcs]

Right — that's the trade-off. Postgres requires the explicit cast (SELECT ARRAY[]::integer[]); PPL doesn't have that surface today, so any caller writing bare array() would otherwise hit an analytics-engine-side error. The implicit VARCHAR default keeps the bare form working, and concrete-element calls (array(1, 2), array('a')) keep their original element type — testReturnTypeForIntegerOperandPreservesType in aa82704 is the regression guard for that.

Long-term we could expose a PPL cast(array() as <T>[]) syntax that mirrors Postgres and remove the implicit default, but that's a separate language-surface change.

[github-actions]

Persistent review updated to latest commit aa82704

[ahkcs]

Pushed aa82704 (UTs) and 22d0cce (trim inline comment).

Replies to the open questions:

@dai-chen — "with this change, we actually define VARCHAR[] array(), right?"

Yes, but only for the bare/empty form. Concrete-element calls keep the inferred element type:

• array() → ARRAY<VARCHAR> (was ARRAY<NULL>)
• array(NULL) → ARRAY<VARCHAR> (was ARRAY<UNKNOWN>)
• array(1) → ARRAY<INTEGER> (unchanged)
• array('a', 'b') → ARRAY<VARCHAR> (unchanged)
• array(1, 1.5) → ARRAY<DOUBLE> (unchanged — leastRestrictive widening from SqlLibraryOperators.ARRAY still wins)

testReturnTypeForIntegerOperandPreservesType in aa82704 is the regression guard for the third case.

@penghuo — "CAST(ARRAY[] to VARCHAR[])? Substrait limitation?"

Yes to both, replied inline.

@penghuo — "Postgres requires ARRAY[]::integer[], PPL implicit VARCHAR[]?"

Yes — PPL surface allows bare array() and we silently default to VARCHAR[]. Replied inline with the rationale and a follow-up note about adding explicit cast(array() as <T>[]) syntax later.

[github-actions]

Persistent review updated to latest commit 22d0cce