Streaming `Deflate` emits invalid back-references on some inputs (cross-validated against native zlib; `deflateSync` correct on same input)

[PeterCoherent]

fflate version: 0.8.3 (also reproduces on 0.8.2)
Node version: v22.22.0
OS: Linux

The problem

Deflate (the low-level streaming class) and ZipDeflate (which wraps it) produce a deflate stream that off-the-shelf inflaters reject as invalid for some inputs. deflateSync and the callback-style deflate() on the same input are both correct and byte-identical.

The streaming-produced bytes diverge from the sync output starting at offset 0 — the deflate block-type bits themselves differ, so it's a fundamental divergence in the streaming class's encoding decisions, not a chunk-boundary or window edge case. The result emits at least one back-reference whose distance is illegal per RFC 1951.

Discovered while building an XLSB → XLSX converter that streams ZIP output via ZipDeflate. One workbook produced a ZIP whose xl/media/image2.emf entry failed to extract. The trigger was a specific 234-byte prefix of that EMF, bisected from the original 2816 bytes — trimming any further (from either end) makes the bug stop reproducing.

Reproducer (self-contained, no extra deps)

import { Deflate, deflate, deflateSync, inflateSync } from 'fflate';
import zlib from 'node:zlib';

// 234 bytes — prefix of a real Windows Enhanced Metafile (.emf)
// embedded in an XLSB workbook. Bisected from the full 2816-byte
// EMF; trimming either end makes the bug stop triggering. Contains
// only the EMF format magic + standard GDI record opcodes.
const TEST_INPUT = Uint8Array.from(Buffer.from(
  'AQAAAGwAAAAAAAAAAAAAAM0AAAAcAAAAAAAAAAAAAABqHAAA/gMAACBFTUYAAAEAAAsAADIAAAAH' +
  'AAAAAAAAAAAAAAAAAAAAAAUAAAAEAADEAQAAaQEAAAAAAAAAAAAAAAAAAOPjBgAcgwUARgAAAGQC' +
  'AABWAgAAR0RJQwEAAIAAAwAAfQ+OiAAAAAA+AgAAAQAJAAADHwEAAAYAKwAAAAAABAAAAAMBCAAF' +
  'AAAACwIAAAAABQAAAAwCHQDOAAMAAAAeAAcAAAD8AgAAaWlpAAAABAAAAC0BAAAJAAAAHQYhAPAA' +
  'HQABAAAA', 'base64'));

function streamingDeflate(buf, level) {
  const chunks = [];
  const d = new Deflate({ level }, (c) => { if (c?.length) chunks.push(new Uint8Array(c)); });
  d.push(buf, true);
  let total = 0; for (const c of chunks) total += c.length;
  const out = new Uint8Array(total); let off = 0;
  for (const c of chunks) { out.set(c, off); off += c.length; }
  return out;
}

const LEVEL = 3;
const sync = deflateSync(TEST_INPUT, { level: LEVEL });
const oneshot = await new Promise((res, rej) =>
  deflate(TEST_INPUT, { level: LEVEL }, (e, o) => e ? rej(e) : res(o)));
const stream = streamingDeflate(TEST_INPUT, LEVEL);

console.log(`deflateSync: ${sync.length}B  -- fflate.inflateSync: ${tryInflate(inflateSync, sync)}  zlib.inflateRawSync: ${tryInflate(zlib.inflateRawSync, sync)}`);
console.log(`deflate():   ${oneshot.length}B  -- fflate.inflateSync: ${tryInflate(inflateSync, oneshot)}  zlib.inflateRawSync: ${tryInflate(zlib.inflateRawSync, oneshot)}  identical-to-sync: ${eq(oneshot, sync)}`);
console.log(`Deflate():   ${stream.length}B  -- fflate.inflateSync: ${tryInflate(inflateSync, stream)}  zlib.inflateRawSync: ${tryInflate(zlib.inflateRawSync, stream)}  identical-to-sync: ${eq(stream, sync)}`);

function tryInflate(fn, b) { try { fn(b); return 'OK'; } catch (e) { return `FAIL: ${e.message}`; } }
function eq(a, b) { return a.length === b.length && Buffer.compare(Buffer.from(a), Buffer.from(b)) === 0; }

Output (fflate 0.8.3, Node 22.22.0)

deflateSync: 153B  -- fflate.inflateSync: OK  zlib.inflateRawSync: OK
deflate():   153B  -- fflate.inflateSync: OK  zlib.inflateRawSync: OK  identical-to-sync: true
Deflate():   153B  -- fflate.inflateSync: FAIL: invalid distance  zlib.inflateRawSync: FAIL: invalid distance too far back  identical-to-sync: false

The streaming-emitted distance is greater than the actual amount of data fed so far (RFC 1951 §3.2.5 violation), so two independent inflaters reject it. Deflate({level: 0}, ...) (no compression) is fine because the stored encoding skips the back-reference path entirely.

Audit — ruling out "wrong usage"

I went through this carefully before filing. The bug reproduces independently of:

Variable	Variants tried	Same invalid output?
Input construction	Uint8Array.from(Buffer.from(b64)); raw Buffer.from(b64); fresh ArrayBuffer-backed copy; readFileSync round-trip	yes — all 4 produce the same invalid 153 B
Chunk shape	single push(buf, true); 1-byte chunks; 64-byte chunks; 200-byte chunk straddling the input	yes — all produce the same invalid 153 B
Level	1, 3, 6, 9 (all non-zero)	yes — all reproduce; level 0 is fine (stored)
Inflater	fflate's inflateSync; Node's native zlib.inflateRawSync	both reject

And the two non-streaming API paths (deflateSync, callback-style deflate()) produce byte-identical correct output for the same input. So the streaming Deflate class is the only piece that produces wrong bytes.

Characterization — what does and doesn't trigger

Across the same 234-byte input shape, plus a handful of probes:

Input (~2.8 KB sample sizes)	Streaming bytes match deflateSync?	Streaming inflates?
Random binary (crypto.randomBytes)	no — streaming output is slightly larger (~1 % overhead)	yes (valid but less efficient)
All-zeros buffer	yes	yes
Highly repetitive text	yes	yes
Real-world binary like this EMF	yes (same size, different bytes)	no

So the bug isn't size- or level-dependent, and isn't a chunking edge case. It only fires for certain byte distributions.

Suspected area

Without spelunking the deflate source: the symptom looks like the LZ77 match-finder accepting a candidate whose distance is greater than the spec-allowed maximum or greater than the actual amount of data fed so far. deflateSync handles it correctly, so whatever bookkeeping deflateSync does to bound the search window is missing or off-by-something in the streaming Deflate path. The first-byte divergence suggests the block-type choice or initial dictionary setup is different too.

Happy to dig further if it helps.

Workaround used downstream

Routing all "I have the full bytes already" entries through deflateSync (and only using streaming Deflate for genuinely streamed payloads, where the bug hasn't surfaced for our inputs) avoids it. That's what we shipped in our converter while waiting on a fix.

Thanks for the library — fflate is great for the 99 % of cases that don't hit this.

[PeterCoherent]

Tracked this down. Root cause: streaming Deflate lets LZ77 match against the zero-init lookback region on first push.

Deflate's 96 KiB internal buffer reserves the first 32 KiB as an LZ77 lookback window; real input is written at offset 32768. On the first push, that 32 KiB lookback is just buffer-init zeros (from new u8(98304)), not previously-emitted output.

The match-finder in dflt caps match distance with Math.min(32767, i), where i is the buffer position. On first push, i starts at 32768, so the cap is always 32767 (the full lookback). If an input byte happens to equal the zero fill at some offset in the lookback, the encoder accepts the match and emits a back-reference with a distance pointing before the start of the real output stream — violating RFC 1951 §3.2.5.

Empirical confirmation

Instrumented dflt to log every match emission on the 234-byte EMF repro. Last match:

i=32997 (data offset 229)  len=5  dist=230  ← invalid (dist > bytes-emitted)

At input offset 229, only 229 bytes have been emitted, but the streaming encoder picked dist=230 — one byte into the zero-init lookback. deflateSync doesn't reproduce because its dat IS the input itself, so i and "bytes emitted" stay in lockstep, and the cap is correct.

Fix

Opened #261 (4-line patch + comments). Adds an f field to DeflateState holding the first valid source offset in dat:

• Streaming Deflate constructor initializes f = 32768.
• The dictionary path sets f = 32768 - dict.length (dictionary IS valid prior data).
• push() zeroes f after the first buffer-shift (from then on the entire 96 KiB holds real data).
• dflt caps maxd with i - (st.f || 0). deflateSync doesn't set f → undefined || 0 === 0 → behavior unchanged.

Testing done before opening the PR

• This issue's 234 B EMF repro at levels 1, 3, 6, 9: now valid on both fflate.inflateSync and Node zlib.inflateRawSync; at level 3 byte-identical to deflateSync.
• 3-push split of the same input, plus 1-byte and 64-byte chunked pushes — all valid.
• Dictionary path round-trip with fflate.Inflate({dictionary}) — valid.
• 100 KB random + 100 KB all-zeros (forces the buffer-shift / multi-dflt-call path) — valid.
• npm test: 120 / 120 pass.

Performance note

The patch may actually improve performance for large inputs. Median-of-5 on Node 22.22.0:

Input	master	patched	Δ
234 B EMF (trigger)	53.7 µs	48.7 µs	−9.2 %
1 MB random	30.2 ms	25.0 ms	−17.5 %
1 MB zeros	9.8 ms	7.9 ms	−19.4 %
1 MB text	8.0 ms	7.7 ms	−3.7 %
4 KB / 64 KB random	—	—	within noise (±1 %)

Why faster: the LZ77 chain-walk in dflt terminates earlier when the cap stops it chasing matches into the (now-correctly-excluded) zero-init region. Output bytes are identical across every input I tested — the patch only suppresses invalid matches, never valid ones.

[101arrowz]

Thank you for the bug report and for the details! I'll review your PR shortly. I'm surprised this hasn't come up before honestly.