Fess Crawler is a Java crawling framework for enterprise-scale content acquisition and text extraction. It is the crawling engine behind Fess, and can be embedded in any JVM application on its own.
- Multi-protocol: HTTP/HTTPS, local/network file systems, FTP/FTPS, SMB/CIFS (SMB1 and SMB2+), MinIO-compatible object storage, Amazon S3, Google Cloud Storage
- Broad text extraction: Office documents, PDF, HTML, XML, JSON, CSV, Markdown, e-mail, archives, PostScript, and media/image metadata — via Apache Tika, POI, PDFBox and format-specific extractors
- Multi-threaded crawling with configurable thread pools, depth limits and access-count limits
- Fault tolerant: retry wrapper (
FaultTolerantClient) and an HTTP client that can fall back between Apache HttpComponents 5.x and 4.x - Polite:
robots.txt(RFC 9309) and sitemap support, plus configurable request intervals - Bounded by design: content-length caps enforced while downloading, archive-bomb and Zip Slip defenses, and bounded recursion in nested extractors
- Pluggable: clients, extractors, transformers, rules and filters are all replaceable via DI
- Pluggable storage: in-memory by default, OpenSearch for persistent/distributed crawls
- Java 21 or later
- Maven 3.8 or later (to build from source)
Add the modules you need. Check the Maven Central badge above for the current release.
<properties>
<fess.crawler.version>15.7.0</fess.crawler.version>
</properties>
<dependencies>
<!-- Core framework -->
<dependency>
<groupId>org.codelibs.fess</groupId>
<artifactId>fess-crawler</artifactId>
<version>${fess.crawler.version}</version>
</dependency>
<!-- Ready-made LastaFlute DI wiring (recommended starting point) -->
<dependency>
<groupId>org.codelibs.fess</groupId>
<artifactId>fess-crawler-lasta</artifactId>
<version>${fess.crawler.version}</version>
</dependency>
<!-- Optional: persist the URL queue and results in OpenSearch -->
<dependency>
<groupId>org.codelibs.fess</groupId>
<artifactId>fess-crawler-opensearch</artifactId>
<version>${fess.crawler.version}</version>
</dependency>
</dependencies>Note
fess-crawler-lastadeclaresjakarta.transaction:jakarta.transaction-apiwithprovidedscope, so it is not pulled in transitively. Lasta DI needs it at runtime — add it to your own dependencies, or container startup fails withClassNotFoundException: jakarta.transaction.Transactional$TxType.You also need an SLF4J/Log4j binding of your choice; none is bundled.
fess-crawler-lasta ships a complete DI configuration (crawler.xml) with every client, extractor,
transformer and rule already wired, so the shortest working crawler is:
import org.codelibs.fess.crawler.Crawler;
import org.codelibs.fess.crawler.entity.AccessResult;
import org.codelibs.fess.crawler.service.DataService;
import org.lastaflute.di.core.factory.SingletonLaContainerFactory;
public class QuickStart {
public static void main(final String[] args) {
SingletonLaContainerFactory.setConfigPath("crawler.xml");
SingletonLaContainerFactory.init();
final Crawler crawler = SingletonLaContainerFactory.getContainer().getComponent("crawler");
final DataService<AccessResult<?>> dataService =
SingletonLaContainerFactory.getContainer().getComponent("dataService");
try (crawler) {
crawler.addUrl("https://example.com/");
crawler.addIncludeFilter("https://example.com/.*");
crawler.setMaxAccessCount(100L);
crawler.setNumOfThread(5);
crawler.setMaxDepth(3);
final String sessionId = crawler.execute(); // blocks until the crawl finishes
System.out.println("crawled=" + dataService.getCount(sessionId));
dataService.iterate(sessionId, result -> System.out.println(result.getUrl()));
dataService.delete(sessionId);
}
}
}crawler is a prototype component: call getComponent("crawler") again for each independent
crawl. execute() returns the session ID, which is the key for everything in DataService and
UrlQueueService.
Swap the URL for file:///path/to/dir, smb://host/share/, ftp://host/dir/, s3://bucket/prefix
… and the same code crawls that source — client selection is driven by the URL scheme.
All of these are Crawler methods; there is no need to reach into CrawlerContext (its field is
protected, so it is not accessible from outside the org.codelibs.fess.crawler package).
crawler.setSessionId("nightly-crawl"); // default: generated timestamp
crawler.setNumOfThread(10); // worker threads
crawler.setMaxDepth(3); // -1 (default) = unlimited
crawler.setMaxAccessCount(10_000L); // 0 (default) = unlimited
crawler.setMaxThreadCheckCount(20); // idle polls before a thread gives up
crawler.setThreadPriority(Thread.NORM_PRIORITY);
crawler.addIncludeFilter("https://example\\.com/.*");
crawler.addExcludeFilter(".*\\.(css|js|png|jpe?g|gif)$");Invalid regular expressions passed to the filters are logged and ignored rather than thrown, so a typo silently widens the crawl — check your logs.
crawler.setBackground(true);
final String sessionId = crawler.execute(); // returns immediately
// CrawlerStatus is INITIALIZING, RUNNING or DONE
crawler.awaitTermination(); // or awaitTermination(millis)
crawler.stop(); // request an early stop
crawler.cleanup(sessionId); // drop queue + filter stateCrawler implements AutoCloseable; close() performs the cleanup for the current session.
final DataService<AccessResult<?>> dataService = container.getComponent("dataService");
System.out.println(dataService.getCount(sessionId));
dataService.iterate(sessionId, result -> {
System.out.println(result.getUrl());
System.out.println(result.getHttpStatusCode());
System.out.println(result.getMimeType());
System.out.println(result.getContentLength());
// Extracted text/binary lives on the child entity, not on AccessResult itself:
System.out.println(result.getAccessResultData().getDataAsString());
});
final AccessResult<?> one = dataService.getAccessResult(sessionId, url);
dataService.delete(sessionId);Clients are selected by URL pattern (see crawler/client.xml in fess-crawler-lasta):
| URL pattern | Client | Notes |
|---|---|---|
http:.*, https:.* |
SwitchableHttpClient |
wraps Hc5HttpClient (default) or Hc4HttpClient |
file:.* |
FileSystemClient |
local and mounted network paths |
smb:.* |
smb.SmbClient |
SMB2+ (SMBJ) |
smb1:.* |
smb1.SmbClient |
legacy SMB1 (JCIFS) |
ftp:.*, ftps:.* |
FtpClient |
shared component for both schemes |
storage:.* |
StorageClient |
MinIO / S3-compatible object storage |
s3:.* |
S3Client |
AWS SDK v2 |
gcs:.* |
GcsClient |
Google Cloud Storage |
Extractors registered by MIME type in crawler/extractor.xml: HTML, XML/XHTML/RDF/FreeMind, PDF,
LHA, e-mail (message/rfc822), JSON, CSV, Markdown, PostScript, and a large Tika-backed catalogue
covering Office (Word/Excel/PowerPoint/Publisher/Visio), OpenDocument, RTF, archives, audio, video
and images (EXIF/IPTC/XMP metadata).
Additional extractors ship in the jar but are not wired into the default DI, so you must register
them yourself if you want them: ZipExtractor, TarExtractor, ApiExtractor, CommandExtractor.
The recommended approach is to include crawler.xml from your own Lasta DI file and redefine only
the components you care about — component names are the contract:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE components PUBLIC "-//DBFLUTE//DTD LastaDi 1.0//EN"
"http://dbflute.org/meta/lastadi10.dtd">
<components namespace="fessCrawler">
<include path="crawler.xml"/>
<component name="internalHttpClient"
class="org.codelibs.fess.crawler.client.http.Hc5HttpClient" instance="prototype">
<property name="userAgent">"MyBot/1.0 (+https://example.com/bot)"</property>
<property name="connectionTimeout">30000</property>
<property name="soTimeout">60000</property>
<property name="maxTotalConnections">200</property>
<property name="maxConnectionsPerRoute">20</property>
</component>
</components>The DI files you can include or override individually live in fess-crawler-lasta's resources:
crawler/ + client.xml, container.xml, contentlength.xml, encoding.xml, extractor.xml,
filter.xml, interval.xml, log.xml, mimetype.xml, robotstxt.xml, rule.xml,
sitemaps.xml, transformer.xml, transformer_basic.xml, urlconverter.xml.
HcHttpClient is an abstract base holding the init-parameter constants — register one of the
concrete classes, Hc5HttpClient (default) or Hc4HttpClient. SwitchableHttpClient extends
FaultTolerantClient and delegates to whichever of the two the system property selects:
-Dfess.crawler.http.client=hc4 # anything else, including unset, selects hc5
The property is read once when SwitchableHttpClient is constructed.
Commonly tuned setters on Hc5HttpClient / Hc4HttpClient — note the names, several differ from
what you might guess:
| Purpose | Setter | Init-parameter key |
|---|---|---|
| User agent | setUserAgent(String) |
userAgent |
| Connect timeout (ms) | setConnectionTimeout(Integer) |
connectionTimeout |
| Read timeout (ms) | setSoTimeout(Integer) |
soTimeout |
| Total pool size | setMaxTotalConnections(Integer) |
maxTotalConnection |
| Per-host pool size | setMaxConnectionsPerRoute(Integer) |
maxConnectionsPerRoute |
| Skip TLS verification | setIgnoreSslCertificate(boolean) |
ignoreSslCertificate |
| Proxy | setProxyHost(String) / setProxyPort(Integer) |
proxyHost / proxyPort |
| Follow redirects | setRedirectsEnabled(boolean) |
redirectsEnabled |
| Honour robots.txt | setUseRobotsTxtDisallows(boolean) / setUseRobotsTxtAllows(boolean) |
— |
Disable TLS verification only against hosts you control.
Every client also inherits generic knobs from AbstractCrawlerClient: accessTimeout,
maxContentLength, maxCachedContentSize. Any of these can be supplied as a map instead of via
setters:
final Map<String, Object> params = new HashMap<>();
params.put(HcHttpClient.USER_AGENT_PROPERTY, "MyBot/1.0");
params.put(HcHttpClient.SO_TIMEOUT_PROPERTY, 60000);
params.put(AbstractCrawlerClient.MAX_CONTENT_LENGTH, 50L * 1024 * 1024);
client.setInitParameterMap(params);S3Client reads accessKey, secretKey, endpoint, region (default us-east-1) and
crossRegionAccessEnabled (default true). When accessKey/secretKey are blank it falls back to
the AWS default credentials provider chain, so IAM roles, instance profiles, IRSA, environment
variables and shared profiles all work without hard-coded keys.
GcsClient reads projectId, credentialsFile and endpoint; without credentialsFile it uses
Application Default Credentials.
DefaultIntervalController exposes four independent delays, all in milliseconds:
<component name="intervalController"
class="org.codelibs.fess.crawler.interval.impl.DefaultIntervalController">
<property name="delayMillisBeforeProcessing">1000</property>
<property name="delayMillisAfterProcessing">0</property>
<property name="delayMillisAtNoUrlInQueue">500</property>
<property name="delayMillisForWaitingNewUrl">1000</property>
</component>Use delayMillisBeforeProcessing for a fixed per-request delay. robots.txt Crawl-delay is
handled separately by RobotsTxtHelper.
Size limits are owned by ContentLengthHelper, not by the transformers. The default cap is 10 MB and
can be overridden per MIME type:
<component name="contentLengthHelper"
class="org.codelibs.fess.crawler.helper.ContentLengthHelper">
<property name="defaultMaxLength">10485760</property>
<postConstruct name="addMaxLength">
<arg>"application/pdf"</arg>
<arg>52428800</arg>
</postConstruct>
</component>HTTP clients enforce this during the download and abort as soon as the limit is exceeded, rather
than buffering the whole body first. Responses larger than maxCachedContentSize spill to a
temporary file, which is deleted when the ResponseData is closed — always use try-with-resources
when handling ResponseData yourself.
HtmlExtractor extracts standard metadata (title, description, OpenGraph, Twitter Card) by default.
JSON-LD extraction is opt-in and bounded in nesting depth, size and item count:
<component name="htmlExtractor" class="org.codelibs.fess.crawler.extractor.impl.HtmlExtractor">
<property name="extractDefaultMetadata">true</property> <!-- default -->
<property name="extractJsonLd">true</property> <!-- default: false -->
</component>ExtractorFactory maps MIME types to extractors. Several extractors may claim the same MIME type; in
that case they are tried in descending weight order until one succeeds (weight defaults to 1,
set via AbstractExtractor.setWeight(int)).
To extract from a stream directly, without running a crawl:
final ExtractData data = extractorFactory.builder(inputStream, params)
.mimeType("application/pdf")
.maxContentLength(10 * 1024 * 1024)
.extract();
System.out.println(data.getContent());When maxContentLength is not set, the builder falls back to ContentLengthHelper.
fess-crawler-opensearch replaces the in-memory queue and result store with OpenSearch indices.
OpenSearchDataService has no no-arg constructor and no host/port setters — it is configured
through OpenSearchCrawlerConfig, as in the module's own crawler_opensearch.xml:
<component name="crawlerConfig"
class="org.codelibs.fess.crawler.util.OpenSearchCrawlerConfig">
<property name="dataIndex">"fess_crawler.data"</property>
<property name="dataShards">5</property>
<property name="dataReplicas">1</property>
</component>
<component name="dataService"
class="org.codelibs.fess.crawler.service.impl.OpenSearchDataService">
<arg>crawlerConfig</arg>
</component>Connectivity comes from an injected fesenClient
(fesen-httpclient); index mappings are created on
first connect.
StandardCrawlerContainer is a small standalone container for applications that do not want Lasta
DI. Be aware of two behaviours before using it:
- Registration eagerly instantiates each component, so a component must be registered after
everything it depends on, and
@Resourceinjection is matched by field name. crawler,crawlerThread,urlQueue,accessResultand the services must be registered asprototype; registeringcrawleras a singleton makes everygetComponent("crawler")return the same instance.
A crawl needs roughly 25 components wired together (crawler + thread + entities, the four services,
eight helpers, a client factory with at least one client, a rule manager with a rule and a response
processor, a transformer, an interval controller, and an extractor factory). Rather than reproduce
all of it here, use
fess-crawler/src/test/java/org/codelibs/fess/crawler/CrawlerTest.java
as the reference wiring — it is exercised by the test suite on every build. For most applications the
Lasta DI route in Quick Start is considerably less work.
fess-crawler-parent/
├── fess-crawler/ # Core framework: clients, extractors, transformers, services
├── fess-crawler-lasta/ # Lasta DI wiring (crawler.xml and friends)
└── fess-crawler-opensearch/ # OpenSearch-backed queue and result store
Crawler ──▶ CrawlerThread ──▶ CrawlerClient ──▶ ResponseData
│
RuleManager (pattern match)
│
ResponseProcessor
│
Transformer ──▶ Extractor ──▶ ExtractData
│
┌───────────────────────────┴───────────────────────────┐
▼ ▼
UrlQueueService (child URLs) DataService (results)
UrlFilter gates URLs on the way into the queue; IntervalController paces the workers;
CrawlingParameterUtil carries the CrawlerContext and UrlQueue in thread-locals.
A custom extractor implements one method:
public class CustomExtractor extends AbstractExtractor {
@Override
public ExtractData getText(final InputStream in, final Map<String, String> params) {
final ExtractData data = new ExtractData();
// ... populate content and metadata
return data;
}
@Override
public int getWeight() {
return 10; // higher wins when several extractors claim the same MIME type
}
}Declare it as a component, then append it to the existing factory after the container starts:
<components namespace="fessCrawler">
<include path="crawler.xml"/>
<component name="customExtractor" class="com.example.CustomExtractor"/>
</components>final ExtractorFactory factory = container.getComponent("extractorFactory");
factory.addExtractor("application/x-custom", container.getComponent("customExtractor"));Warning Do not redeclare
extractorFactoryitself in your DI file to add a mapping. The outer definition replaces the included one wholesale, and every default MIME-type mapping — HTML, PDF, Office, everything — is silently lost, leavinggetExtractor()returningnull.
Custom clients (CrawlerClient), transformers (Transformer) and rules (Rule) follow the same
pattern — implement the interface, register the component, map it in the relevant factory.
git clone https://github.com/codelibs/fess-crawler.git
cd fess-crawler
mvn clean install # build + test all modules
mvn clean install -DskipTests # skip tests
mvn clean install -pl fess-crawlerfess-crawler builds against fess-parent; when working on an unreleased version, build and install
codelibs/fess-parent (branch main) first — this is
exactly what CI does.
Tests use JUnit 5 (Jupiter) on top of UTFlute (PlainTestCase, or LastaDiTestCase for the DI
modules), with Mockito and Testcontainers where a real backend is needed.
mvn test
mvn test -pl fess-crawler
mvn test -Dtest=CrawlerTest
mvn test -Dtest=CrawlerTest#test_execute_webSurefire runs test classes in parallel, so tests must not share mutable static state.
JodExtractorTest is excluded from the default run (it needs a local LibreOffice/JODConverter).
Container-backed tests (S3ClientTest, StorageClientTest, GcsClientTest, SmbClientTest)
require a working Docker daemon.
JaCoCo produces a coverage report during package, but no minimum is enforced by the build.
mvn formatter:format # Eclipse formatter: 4 spaces, no tabs, 140-column lines
mvn license:format # Apache 2.0 headers (license:check runs in the build and fails on gaps)Formatter and license configuration come from fess-parent, which points at the shared CodeLibs
config, so do not add a local formatter file.
.github/workflows/maven.yml runs on pushes and pull requests to master and *.x branches: a
single JDK 21 (Temurin) build that installs fess-parent and then runs mvn -B package.
package is more than tests — it also runs license:check, formatter:format, jacoco:report,
source:jar and javadoc:jar. Javadoc errors therefore fail CI even though mvn test passes
locally. If you touch imports or Javadoc, verify with:
mvn javadoc:jarContainer fails to start with ClassNotFoundException: jakarta.transaction.Transactional$TxType
Add jakarta.transaction:jakarta.transaction-api to your dependencies — fess-crawler-lasta
declares it as provided.
Nothing is crawled beyond the seed URL
Include filters are required for child URLs. crawler.addUrl(url) alone does not authorize the
crawler to follow links; add crawler.addIncludeFilter(url + ".*"). Also check setMaxDepth.
Crawl stops early
Check setMaxAccessCount and setMaxDepth, and whether robots.txt disallows the paths — set
useRobotsTxtDisallows to false on the HTTP client only when you are authorized to ignore it.
Timeouts on slow servers
Raise connectionTimeout and soTimeout on the HTTP client, and accessTimeout on the client base.
Large files are truncated or skipped
Raise ContentLengthHelper.defaultMaxLength (10 MB default) or add a per-MIME-type override, and
raise the client's maxContentLength.
High memory use
Lower setNumOfThread, lower maxCachedContentSize so bodies spill to disk sooner, and keep
ContentLengthHelper limits tight.
<logger name="org.codelibs.fess.crawler" level="DEBUG"/>
<logger name="org.codelibs.fess.crawler.client" level="DEBUG"/>
<logger name="org.codelibs.fess.crawler.extractor" level="DEBUG"/>- Fork the repository and create a feature branch
- Add tests for your change (JUnit 5)
- Run
mvn formatter:format && mvn license:format - Run
mvn package— this catches Javadoc and license failures thatmvn testdoes not - Open a pull request against
master
Bug reports and feature requests: https://github.com/codelibs/fess-crawler/issues
Apache License 2.0 — see LICENSE.