docs: rewrite the JSON connector page for the 15.9 implementation - #516
Merged
Conversation
The ds-json page still described a much older connector. None of
root_path, format, recursive, max_depth, include_pattern,
exclude_pattern or file_suffixes appeared anywhere in it, and several
statements it did make are no longer true.
Corrections:
- "JSON Lines only; a JSON array or a pretty-printed document cannot be
read directly, convert it to JSON Lines" -- the connector reads three
shapes (JSON Lines, an array of objects, a single object) and detects
which one from the document itself by default.
- "If both are given, files takes priority and directories is ignored"
-- the two are not exclusive. Both are processed, and a file reachable
through both is read once.
- "The parameter name is the camelCase fileEncoding, not the snake_case
file_encoding" -- both spellings resolve, for every parameter.
- The install path pointed at app/WEB-INF/lib. Both directories are on
the crawler child process classpath, so that did work, but
app/WEB-INF/plugin is where the admin UI installs plugins and is what
the plugin's own README documents. The download link also pointed at
Maven Central, which only carries 15.7.0 and earlier; 15.8.0 onwards
is published to the CodeLibs repository.
New material, all of it previously undocumented:
- The full parameter table, including the seven parameters listed above.
- File discovery order: files in the order given, directory contents
oldest first, files before directory contents, and the suffix filter
applying to files as well as to directories.
- format=auto|jsonl|json, when jsonl is worth setting explicitly, and
how the choice decides what a malformed record costs -- one line in
JSON Lines, potentially the rest of the source in the other shapes.
- root_path: array to N records, object to one, no match to none, the
~0/~1 escapes, and its precedence over format. A leading slash is
required, and omitting it fails the whole data config without naming
the parameter, so that carries a warning directive.
- Which parameter errors abort the crawl with a failure URL naming the
parameter, and which are logged and defaulted.
- That a data store crawl finishes normally even when it indexed
nothing, so the index count and the failure URLs are what to check.
The script section previously used Groovy-only idioms (?:, as Float,
collect {}). Fess 15.9 replaced the built-in engine with JavaScript and
moved Groovy out to fess-script-groovy, so those examples no longer run
under the default engine. They are replaced with the nested-object and
array-element forms, which behave the same either way, plus a note that
script_type selects the engine and that an omitted value means groovy.
Also documents that the credential masking driven by
app.encrypt.property.pattern is a case-sensitive full match on the
parameter name, so access_token is masked while accessToken is not.
Applied identically to all seven languages. Every page parses cleanly
under docutils.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The 15.9 JSON connector page still described a much older connector. Seven
parameters were missing entirely, and several statements the page did make are
no longer true. This rewrites it in all seven languages.
The page content was checked against the current
fess-ds-jsonmaster byrunning real data store crawls on a live Fess 15.9.0-SNAPSHOT, rather than by
reading the source alone.
Corrected
filestakes priority anddirectoriesis ignored"fileEncoding, not the snake_casefile_encoding"app/WEB-INF/libapp/WEB-INF/plugin— where the admin UI installs plugins, and what the plugin README documents. (WEB-INF/libalso happens to be on the crawler child classpath, so it did work; it is just not the managed location.)?:,as Float,collect {})fess-script-groovy, so those no longer run under the default engineAdded
None of this was documented anywhere on the page before.
recursive,max_depth,include_pattern,exclude_pattern,file_suffixes,formatandroot_pathin addition tothe three that were already there.
filesin the order given, directory contentsoldest first,
filesbefore directory contents; the suffix filter applies tofilestoo.format— whatautodecides, whenjsonlis worth setting explicitly,and how the choice decides what a malformed record costs: one line in JSON
Lines, potentially the rest of the source in the other shapes.
root_path— array to N records, object to one, no match to none, the~0/~1escapes, and its precedence overformat. A leading/isrequired; omitting it fails the whole data config without naming the
parameter, so that gets a
warningdirective.named after the parameter, and which are logged and defaulted.
nothing, so the index count and the failure URLs are what to judge by.
app.encrypt.property.patternmasking is a case-sensitivefull match on the parameter name, so
access_tokenis masked butaccessTokenis not.Notes
they are.
and reuse each language's existing
ds-csv.rstterminology.