In simple English
This tool reads files from your local computer or a remote server. It opens PDF files, Word documents and spreadsheets. It extracts all the words inside them. Then it saves those words into Elasticsearch (a search engine that lets you query millions of records instantly). You can run it inside Docker (a way to run a program inside its own sealed box, so it cannot break anything else on your server). It costs nothing to download and use under the Apache-2.0 licence.
What it is
FSCrawler is a utility that indexes local or remote file systems into Elasticsearch. It monitors folders for new, updated or deleted files and syncs changes automatically.
What it is good for
Solo builders who want to build a private internal search engine for thousands of documents, invoices or user manuals.
- Crawls local folders or remote servers over SSH and FTP.
- Extracts text from PDF, Word, Excel and OpenOffice files automatically.
- Runs easily inside a pre-built Docker container.
- Requires a running Elasticsearch server to function.
- Configuration uses YAML files which can be tricky to set up.
- Java memory needs can grow high when processing large PDF files.
Writing custom folder walking scripts and file parsing code from scratch.
Elasticsearch
- meilisearch/meilisearch60k stars
A fast search engine that offers simpler out-of-the-box setup than Elasticsearch.
- dbeaver/dbeaver52k stars
A free desktop program that connects to more than a hundred kinds of database and lets you browse and edit them.
- dimitri/pgloader6.5k stars
Loads data into PostgreSQL from other databases and from files, keeping going when individual rows fail.
docker pull dadoonet/fscrawler docker run -it --rm \ --add-host=host.docker.internal:host-gateway \ -v ~/.fscrawler:/root/.fscrawler \ -v $(pwd)/resumes:/tmp/es:ro \ -e FSCRAWLER_ELASTICSEARCH_URLS=http://host.docker.internal:9200 \ dadoonet/fscrawler