Skip to content

fix: batch and compress UID FETCH for large mailboxes#2

Open
furriest wants to merge 1 commit into
bulwarkmail:mainfrom
furriest:fix/batch-fetch-large-mailboxes
Open

fix: batch and compress UID FETCH for large mailboxes#2
furriest wants to merge 1 commit into
bulwarkmail:mainfrom
furriest:fix/batch-fetch-large-mailboxes

Conversation

@furriest

@furriest furriest commented Jul 4, 2026

Copy link
Copy Markdown

Problem

Mailboxes with a few thousand messages (~2500 in the reported case, against Kerio Connect) appeared empty in JMAP clients while Mailbox/get correctly reported the total message count.

Root cause

fetchSortMetadata() and the hasAttachment post-filter each build a UID FETCH command by joining every matching UID with commas:

const range = uids.join(",");

For a mailbox with thousands of messages this produces a command line long enough to exceed the line-length limits some IMAP servers enforce more strictly than Dovecot/Gmail (observed with MDaemon). The FETCH fails or the connection drops, and the resulting error isn't surfaced as a JMAP-level error — the client just sees an empty result set.

Fix

  • Sort UIDs and chunk them into batches (200 per FETCH).
  • Compress each batch into IMAP sequence-set ranges (compressUidsToRanges), e.g. [100,101,102,110]"100:102,110", instead of a flat comma-separated list.
  • Applied to both fetchSortMetadata (the Email/query sort/fetch path) and the hasAttachment post-filter, which had the same pattern.

Testing

  • npx tsc -p tsconfig.json passes clean.
  • Verified against a live MDaemon mailbox with ~2500 messages: previously returned an empty Email/query result, now returns correctly paginated results.

Happy to add a regression test against the integration compose stack if useful — wasn't sure whether you'd rather fake a large mailbox in compose.test.yml or mock at the ImapFlow level.

fetchSortMetadata() and the hasAttachment post-filter each built a
single UID FETCH command by joining every matching UID with commas
(e.g. UID FETCH 1,2,3,...,2505 (...)). On mailboxes with a few
thousand messages this produces a command line long enough that some
IMAP servers (observed with MDaemon) reject it or
drop the connection outright. The resulting IMAP error wasn't
surfaced as a JMAP error, so JMAP clients saw an empty mailbox instead
of a fetch failure.

Fix: sort UIDs, chunk them into batches of 200, and compress each
batch into IMAP sequence-set ranges (e.g. "100:105,110,204:210")
instead of a flat comma list. Cuts both the number of round-trips and
the command length for large, mostly-contiguous mailboxes.

Applies to both fetchSortMetadata() (Email/query sort/fetch path) and
the hasAttachment post-filter, which had the same pattern.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant