Skip to content

docs: rewrite the JSON connector page for the 15.9 implementation - #516

Merged
marevol merged 1 commit into
masterfrom
docs/ds-json-15.9-current-behaviour
Aug 30, 2026
Merged

docs: rewrite the JSON connector page for the 15.9 implementation#516
marevol merged 1 commit into
masterfrom
docs/ds-json-15.9-current-behaviour

Conversation

@marevol

@marevol marevol commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

The 15.9 JSON connector page still described a much older connector. Seven
parameters were missing entirely, and several statements the page did make are
no longer true. This rewrites it in all seven languages.

The page content was checked against the current fess-ds-json master by
running real data store crawls on a live Fess 15.9.0-SNAPSHOT, rather than by
reading the source alone.

Corrected

The page said Actually
JSON Lines only; an array or a pretty-printed document "cannot be read directly, convert it to JSON Lines" Three shapes are supported — JSON Lines, an array of objects, and a single object — and the shape is detected from the document by default
"If both are given, files takes priority and directories is ignored" They are not exclusive. Both are processed, and a file reachable through both is read once
"The parameter name is the camelCase fileEncoding, not the snake_case file_encoding" Both spellings resolve, for every parameter
Install to app/WEB-INF/lib app/WEB-INF/plugin — where the admin UI installs plugins, and what the plugin README documents. (WEB-INF/lib also happens to be on the crawler child classpath, so it did work; it is just not the managed location.)
Download from Maven Central Maven Central carries 15.7.0 and earlier. 15.8.0 onwards is published to the CodeLibs repository
Script examples in Groovy-only syntax (?:, as Float, collect {}) Fess 15.9 replaced the built-in engine with JavaScript and moved Groovy out to fess-script-groovy, so those no longer run under the default engine

Added

None of this was documented anywhere on the page before.

  • The full parameter table: recursive, max_depth, include_pattern,
    exclude_pattern, file_suffixes, format and root_path in addition to
    the three that were already there.
  • File discovery orderfiles in the order given, directory contents
    oldest first, files before directory contents; the suffix filter applies to
    files too.
  • format — what auto decides, when jsonl is worth setting explicitly,
    and how the choice decides what a malformed record costs: one line in JSON
    Lines, potentially the rest of the source in the other shapes.
  • root_path — array to N records, object to one, no match to none, the
    ~0/~1 escapes, and its precedence over format. A leading / is
    required; omitting it fails the whole data config without naming the
    parameter, so that gets a warning directive.
  • Error routing — which parameter errors abort the crawl with a failure URL
    named after the parameter, and which are logged and defaulted.
  • A note that a data store crawl finishes normally even when it indexed
    nothing, so the index count and the failure URLs are what to judge by.
  • A note that the app.encrypt.property.pattern masking is a case-sensitive
    full match on the parameter name, so access_token is masked but
    accessToken is not.

Notes

  • Only the 15.9 (development) tree is touched; earlier versions are left as
    they are.
  • Every page parses cleanly under docutils.
  • The Japanese page is the source; the other six follow it section for section
    and reuse each language's existing ds-csv.rst terminology.

The ds-json page still described a much older connector. None of
root_path, format, recursive, max_depth, include_pattern,
exclude_pattern or file_suffixes appeared anywhere in it, and several
statements it did make are no longer true.

Corrections:

- "JSON Lines only; a JSON array or a pretty-printed document cannot be
  read directly, convert it to JSON Lines" -- the connector reads three
  shapes (JSON Lines, an array of objects, a single object) and detects
  which one from the document itself by default.
- "If both are given, files takes priority and directories is ignored"
  -- the two are not exclusive. Both are processed, and a file reachable
  through both is read once.
- "The parameter name is the camelCase fileEncoding, not the snake_case
  file_encoding" -- both spellings resolve, for every parameter.
- The install path pointed at app/WEB-INF/lib. Both directories are on
  the crawler child process classpath, so that did work, but
  app/WEB-INF/plugin is where the admin UI installs plugins and is what
  the plugin's own README documents. The download link also pointed at
  Maven Central, which only carries 15.7.0 and earlier; 15.8.0 onwards
  is published to the CodeLibs repository.

New material, all of it previously undocumented:

- The full parameter table, including the seven parameters listed above.
- File discovery order: files in the order given, directory contents
  oldest first, files before directory contents, and the suffix filter
  applying to files as well as to directories.
- format=auto|jsonl|json, when jsonl is worth setting explicitly, and
  how the choice decides what a malformed record costs -- one line in
  JSON Lines, potentially the rest of the source in the other shapes.
- root_path: array to N records, object to one, no match to none, the
  ~0/~1 escapes, and its precedence over format. A leading slash is
  required, and omitting it fails the whole data config without naming
  the parameter, so that carries a warning directive.
- Which parameter errors abort the crawl with a failure URL naming the
  parameter, and which are logged and defaulted.
- That a data store crawl finishes normally even when it indexed
  nothing, so the index count and the failure URLs are what to check.

The script section previously used Groovy-only idioms (?:, as Float,
collect {}). Fess 15.9 replaced the built-in engine with JavaScript and
moved Groovy out to fess-script-groovy, so those examples no longer run
under the default engine. They are replaced with the nested-object and
array-element forms, which behave the same either way, plus a note that
script_type selects the engine and that an omitted value means groovy.

Also documents that the credential masking driven by
app.encrypt.property.pattern is a case-sensitive full match on the
parameter name, so access_token is masked while accessToken is not.

Applied identically to all seven languages. Every page parses cleanly
under docutils.
@marevol
marevol merged commit 23021ff into master Aug 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant