docs: document the crawl.order configuration parameter - #514
Open
marevol wants to merge 3 commits into
Open
Conversation
crawl.order has shipped since the sequential/random switch was added but was never documented in any language. It now names a UrlQueueOrder component, so the page lists the built-in names, records that depth-first is bounded by the polling batch rather than a strict traversal, and notes that the old values still resolve.
Crawling-config Configuration Parameters are dispatched by a "config." prefix that is stripped before use, so the documented key must be config.crawl.order, not crawl.order. An unprefixed crawl.order= line matches no prefix and is silently dropped, leaving the crawl in the default order with no warning. Fix the code samples, prose, and the legacy-values note (config.crawl.order=sequential / config.crawl.order=random) across all seven language trees. While those files were open: - Add a sentence noting that weights are uniform unless a custom UrlQueueWeigher is installed, so weightFirstUrlQueueOrder has no effect by default and sequentialUrlQueueOrder is, in practice, ordered by discovery order. - Add :widths: to the new Crawl Order list-table to match the other tables on the page.
…tch case "has no effect by default" described the symptom without the cause and left the order looking pointless. The two weight-based orders differ only in what happens between entries of equal weight, and the fact that gets missed is that weight is already the default order's primary sort key - installing a UrlQueueWeigher changes the fetch order with no config.crawl.order setting at all. Out of the box every entry ties at the same weight, which is why the default reads as discovery order and why weightFirstUrlQueueOrder has nothing to sort by. It earns its place on a large weighted backlog where only the weight should decide what comes next. The depth-first caveat also gains the case that surprises people: when the whole queue fits in one polling batch, every order proceeds level by level.
marevol
force-pushed
the
docs/crawl-order
branch
from
August 30, 2026 03:46
321bc4b to
6f9c72e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
crawl.orderhas shipped for some time but was never documented in any language. It now names aUrlQueueOrdercomponent, so this documents it in the 15.9 tree for all seven languages.Added to the "Basic Configuration Items" section of
config/crawler-basic.rst, between "Crawl Interval" and "File Size Configuration":config.crawl.order=depthFirstUrlQueueOrderUrlQueueWeigherchanges the fetch order without settingconfig.crawl.orderat all;weightFirstUrlQueueOrderdiffers only between entries of equal weight, and earns its place on a large weighted backlog where only the weight should decide what comes nextsequentialandrandomvalues still resolveThe full parameter name matters:
ParameterUtilstrips theconfig.prefix before building the map, socrawl.order=...is ignored with no diagnostic at all — verified on a live crawl.Only the
15.9tree is touched;versions.jsonmarks it asdevelopment.Verified with standalone docutils rather than a Sphinx build —
conf/conf.py:228contains an invalid\uescape in a non-raw string, so the module raisesSyntaxErroron Python 3 before any RST is parsed. That is a pre-existing problem unrelated to this change and worth a separate fix, since it currently blocks local HTML builds for every language. The parse was run before and after the edit in all seven languages; the message counts are unchanged.