Remaining CPU hotspots from production query profiling (post issue #19)

[stalep]

Summary

Post-optimization CPU flame graphs from JjqProductionQueryBenchmark (after issue #19 optimizations) reveal five remaining systemic hotspots. These are ordered by impact.

Profiling Context

All flame graphs captured with async-profiler CPU event via JMH -prof async, 3 forks, 5 warmup + 5 measurement iterations. Profiles stored in profiles/production-query-post-opt/.

Hotspot 1: Repeated ensureHashIndex() construction during iteration (CRITICAL)

Where: prod_extractMetric — 71.8% of CPU in JqObject.ensureHashIndex

What happens: The query [.autobench_workload.data[0].pcp_time_series[] | .["mem.util.used"]] iterates 502 PCP time series objects, each with 127 keys. Every .get("mem.util.used") call on a distinct JqObject instance triggers ensureHashIndex(), which builds a HashMap<String, Integer> with 127 entries. The hash index is used for one lookup, then the next object builds its own from scratch. That is 502 HashMap constructions x 127 put() calls each = ~63,754 HashMap insertions per query invocation.

Why it matters: The hash index (issue #19) was designed to amortize across multiple get() calls on the same object. But in iteration patterns, each object is accessed once and discarded. The one-time construction cost exceeds what the linear scan would have cost.

Proposed fix — eager hash index at parse time:

Build the hash index during JqObject.ofArrays() for objects with size > HASH_THRESHOLD. The construction cost shifts from query execution to parsing, which:

• Happens once for pre-parsed documents (h5m pattern: parse once, query many times)
• Is already measured separately in parse benchmarks
• Amortizes the cost if the same object is queried multiple times with different fields

public static JqObject ofArrays(String[] keys, JqValue[] values, int size) {
    if (size == 0) return EMPTY;
    var obj = new JqObject(keys, values, size, null);
    if (size > HASH_THRESHOLD) {
        obj.ensureHashIndex(); // pre-build at construction
    }
    return obj;
}

Expected impact: prod_extractMetric should improve ~3-5x (72% of CPU eliminated, limited by remaining ~15% VM iteration overhead).

Risk: Parse benchmarks will regress slightly for large objects since hash index construction moves there. Need to measure the trade-off. For h5m's use case (parse once, query many times), this is the right trade-off.

Hotspot 2: Evaluator fallback for compilable expressions (HIGH)

Where: prod_deepField — 52% of CPU in evaluator chain; prod_keys — 53% of CPU in evaluator chain

What happens: Expressions like .autobench_workload.data[0].results fall back to the tree-walk Evaluator.eval because chained field access with array indexing (data[0]) isn't compiled to bytecode. The evaluator adds overhead:

Component	CPU share	Why
Evaluator.eval	15-17%	Tree-walk dispatch loop
ArrayList.grow	7-9%	Result list starts at default size 10, resizes repeatedly
Evaluator$$TypeSwitch	5-6%	JDK-generated sealed-class switch on JqExpr subtypes
ArrayList$Itr.checkForComodification	3%	Iterator overhead
Evaluator$$Lambda.accept	3-4%	Lambda captures for continuation passing

prod_keys anomaly: The keys builtin has a BUILTIN_KEYS bytecode opcode, but the flame graph shows Evaluator.eval at 17%. This suggests some keys invocations still fall to the tree-walker — possibly when keys is used in a pipe chain that the compiler doesn't fully compile.

Proposed fix (two parts):

1. Compile chained field + index access: Teach the compiler to handle .field1.field2[0].field3 as a fused sequence of DOT_FIELD + INDEX(const) + DOT_FIELD opcodes, avoiding evaluator fallback.

2. Investigate prod_keys evaluator fallback: Determine why .autobench_workload.data[0].pcp_time_series[0] | keys hits the evaluator. The keys part should use BUILTIN_KEYS — the evaluator is likely triggered by the preceding field chain.

Expected impact: 40-50% throughput improvement on prod_deepField and prod_keys if evaluator fallback is eliminated.

Hotspot 3: VirtualMachine.push bounds check in fused iteration (MEDIUM-HIGH)

Where: 5 of 7 benchmarks, 3-31% of CPU in VirtualMachine.push

What happens: push() at VirtualMachine.java:790 checks if (sp >= stack.length) on every call:

private void push(JqValue val) {
    if (sp >= stack.length) stack = java.util.Arrays.copyOf(stack, stack.length * 2);
    stack[sp++] = val;
}

Inside collectIterateBody, the stack never grows (iteration bodies use 2-3 slots from a 64-slot pre-allocated stack). The branch is always-not-taken but still costs:

• Branch prediction resources
• Prevents method inlining (the copyOf cold path makes the method too large for JIT inlining heuristics)
• Repeated this.stack and this.sp field loads from the heap

Proposed fix — pre-check + direct array access:

private void collectIterateBody(int bodyStart, int bodyLen) {
    // Pre-check: ensure stack has capacity for max body stack depth
    if (sp + MAX_BODY_STACK_DEPTH >= stack.length) {
        stack = java.util.Arrays.copyOf(stack, stack.length * 2);
    }
    final JqValue[] localStack = stack;
    int localSp = sp;
    // ... inner loop uses localStack[localSp++] instead of push()
    sp = localSp;
}

Or simpler: extract sp and stack to local variables at the top of collectIterateBody and write back at the end. This helps the JIT hoist the bounds check.

Expected impact: 10-20% improvement on iteration-heavy benchmarks (prod_iterateExtract, prod_collectConfig).

Hotspot 4: Serialization — escapeJson + itable stub (MEDIUM)

Where: prod_roundTrip_identity — 20.7% escapeJson + 17.4% itable stub = 38% combined

What happens:

• escapeJson scans every character of every string looking for characters that need escaping (", \, control chars). For strings parsed from JSON, the original text was already properly escaped — this scan is entirely redundant.

• itable stub is JVM overhead from calling appendTo(StringBuilder) through the JqValue sealed interface. Each call goes through interface method dispatch table lookup rather than a direct virtual call.

Proposed fix (two parts):

1. Deferred strings (Issue Deferred string value parsing (Jackson-inspired lazy materialization) #13) for zero-copy serialization: Strings holding their original JSON bytes can call sb.append(source, start, end) directly without escaping — the knowledge base already documents this as a planned optimization. This eliminates both escapeJson (20.7%) and much of the itable stub cost.

2. Type-specialize appendTo dispatch in containers: In JqObject.appendTo and JqArray.appendTo, replace:

   values[i].appendTo(sb); // itable dispatch

   with:

   JqValue v = values[i];
   if (v instanceof JqString s) s.appendTo(sb);       // direct call
   else if (v instanceof JqNumber n) n.appendTo(sb);   // direct call
   else v.appendTo(sb);                                 // fallback

   This lets the JIT devirtualize the common cases (93% of production data is strings).

Expected impact: 2-3x improvement on prod_roundTrip_identity if deferred strings eliminate escapeJson. The itable stub fix alone would give ~20% improvement.

Hotspot 5: Builder.put duplicate-key scan on known-unique keys (MEDIUM)

Where: prod_objectConstruct and prod_collectConfig — part of 24% String.equals

What happens: JqObject.Builder.put() scans all existing keys linearly to check for duplicates:

public Builder put(String key, JqValue value) {
    for (int i = 0; i < size; i++) {
        if (key.equals(keys[i])) { // duplicate scan
            values[i] = value;
            return this;
        }
    }
    // ... append new key
}

For {user, uuid, run_id, start_time, end_time} (5 fields), that is 0+1+2+3+4 = 10 string comparisons per object construction. The keys come from the compiled expression's objLayouts array — they are known at compile time to be unique (static layout). The scan is wasted work.

Proposed fix — putUnchecked for known-unique keys:

/** Append without checking for duplicates. Caller guarantees key is new. */
public Builder putUnchecked(String key, JqValue value) {
    if (size >= keys.length) {
        keys = Arrays.copyOf(keys, keys.length * 2);
        values = Arrays.copyOf(values, values.length * 2);
    }
    keys[size] = key;
    values[size] = value;
    size++;
    return this;
}

In VirtualMachine.java BUILD_OBJECT, use putUnchecked when the layout has unique keys (check at compile time: if all layout[i] indices are distinct, keys are guaranteed unique):

case BUILD_OBJECT -> {
    int count = arg1s[curPc];
    int[] layout = objLayouts[arg2s[curPc]];
    JqValue[] vals = new JqValue[count];
    for (int i = count - 1; i >= 0; i--) vals[i] = pop();
    var builder = JqObject.builder(count);
    for (int i = 0; i < count; i++) builder.putUnchecked(names[layout[i]], vals[i]);
    push(builder.build());
}

For expressions with runtime-computed keys (e.g., {(expr): value}), continue using put() with the duplicate scan.

Expected impact: 5-10% improvement on object construction benchmarks. Small change, low risk.

Summary Table

#	Hotspot	Benchmarks	CPU share	Proposed fix	Expected gain
1	ensureHashIndex rebuild per object	extractMetric	72%	Eager hash at parse time	3-5x
2	Evaluator fallback	deepField, keys	52-53%	Compile field+index chains	40-50%
3	push bounds check in hot loop	5 benchmarks	3-31%	Pre-check + local vars	10-20%
4	escapeJson + itable stub	roundTrip_identity	38%	Deferred strings (issue #13)	2-3x
5	Builder duplicate scan	objectConstruct	part of 24%	putUnchecked for unique keys	5-10%

References

• Issue Optimize hot paths identified by production query profiling (issue #18) #19: Hash-indexed get(), Builder-based BUILD_OBJECT, cached sortedKeysAsArray
• Issue Add production-scale jq query benchmarks with async-profiler analysis #18: Production-scale jq query benchmarks
• Issue Deferred string value parsing (Jackson-inspired lazy materialization) #13: Deferred string parsing
• Flame graphs: profiles/production-query-post-opt/ (CPU)
• Results: benchmark-results/production-query-gc-final.json

[stalep]

Review: Issue #22 vs Profiling Findings

Verification against our async-profiler analysis

Hotspot	Identified in our analysis?	Issue #22 accurate?	Assessment
1. ensureHashIndex rebuild per object	Partially -- we noted the +12% shortfall and ramp-up pattern in fork data, mentioned eager parse-time construction as future optimization	Yes, extends with root cause. The 502 objects x 127 HashMap insertions = 63,754 insertions per query is the precise quantification we didn't provide.	High value
2. Evaluator fallback for compilable expressions	Not identified -- we saw Evaluator.eval, evalSubExpr frames but treated them as normal VM overhead	Yes, new and valuable. The insight that .field1.field2[0].field3 falls back to the tree-walker because array indexing in chains isn't compiled is the highest-value finding.	Highest value
3. push() bounds check in hot loop	Partially -- we saw push/pop frames but didn't analyze the inlining impact	Yes, correct mechanism. The Arrays.copyOf cold path preventing inlining is a known JVM pattern.	Medium value
4. escapeJson + itable stub	Verified -- we identified this in our serialization analysis and linked to deferred strings (#13)	Yes, correctly links to #13. One note below on coverage.	Medium value
5. Builder duplicate-key scan	Not identified	Yes, good catch. putUnchecked for compile-time-known unique keys is clean and correct.	Low value

Hotspot 2 deserves emphasis

The evaluator fallback finding is the most impactful discovery in this issue that we missed in our analysis. Our profiling extracted the frame names but we didn't recognize that Evaluator.eval appearing at 52% on prod_deepField was a compiler gap, not normal execution overhead.

The expression .autobench_workload.data[0].results should compile to DOT_FIELD + DOT_FIELD + INDEX(0) + DOT_FIELD bytecode. If the compiler currently can't handle the [0] array index in a chained field access, it falls back to the tree-walker for the entire expression. Fixing this compiler gap would benefit all h5m deep-access patterns.

Hotspot 1: eager hash index trade-off

The proposal to build hash indexes at parse time is correct for h5m's pattern (parse once, query many times). But it shifts cost to parsing:

• Parse regression: 502 objects x 127-key HashMap construction adds ~2-3ms to the 14MB parse. Current parse is ~11ms -- roughly a 20-27% parse regression.
• Query improvement: prod_extractMetric should improve 3-5x.
• Net benefit for h5m: Parse happens once per upload, extraction happens per-node in the DAG (potentially dozens of times). The trade-off is clearly positive.

Alternative: build on second get() call. First call does linear scan and sets a flag. Second call builds the hash. This avoids construction for single-access iteration patterns while still amortizing for multi-access patterns. More complex but avoids the parse-time cost.

Hotspot 4: check deferred string coverage

If escapeJson is still at 20.7% post-#13, some strings are hitting the materialized path instead of zero-copy. Possible causes:

• Strings accessed via stringValue() before serialization (triggers materialization, loses zero-copy benefit)
• Strings from jq operations (concatenation, tostring) that are always eager
• The identity filter . returns the same JqValue tree, but if any string was materialized during a previous operation, it stays materialized

Worth adding a diagnostic counter to track how many strings take the zero-copy path vs the escapeJson path during a round-trip, to understand coverage.

Overall

Issue #22 is well-researched and accurate. The quantified CPU percentages from post-optimization flame graphs provide precise targets. The proposed fixes are all reasonable. The priority ordering is correct, though Hotspot 2 (compiler gap for field+index chains) should be co-priority with Hotspot 1 since it affects more benchmarks and represents a fundamental compiler improvement.