1.1k stars · AGPL-3.0 · v1.14.3 (2026-08-20) · Track this in Scout
Drives real Chromium browsers to record a page as a viewer actually saw it, writing a standard WACZ archive plus the page text.
▶Repo detailsthe review · specs · pros & cons · install
What it is
Browsertrix Crawler is one Docker image that drives real Chromium browsers and records everything they load. The result is a WACZ file, a standard archive format you can replay in a viewer, plus a plain text file of everything that was on the pages.What it is good for. Anybody who needs an honest copy of a page rather than a picture of it. Ordinary scrapers read the first thing the server sends, which on a modern site is often an empty shell filled in afterwards by code. This one waits and records what a person would have seen. That matters for competitor pricing pages, for terms and conditions, for regulatory records, and for anything you may need to show somebody in six months. This is the engine behind Webrecorder's own paid service, so it is the real thing rather than a demonstration.
- It handles pages that are built by code in the browser, which is where simple scrapers return nothing useful.
- It runs several browsers at once, with rules for how far to wander from the starting page.
- It writes plain text alongside the archive, so the result is searchable as well as viewable.
- Memory. Budget one to two gigabytes for each browser you run at the same time, so a small server will stop working on anything ambitious.
- The licence is AGPL-3.0, the strictest of the common ones. If you build a service on top of it and let other people use it over the internet, you must publish your changes.
- It gives you an archive and raw text, not neat fields. If you want prices in a spreadsheet, this is not the tool.
- ArchiveBox/ArchiveBox
The same goal of keeping pages forever, with a friendlier screen and many output formats, but a weaker copy of pages that depend heavily on code.
Track this in Scout - apify/crawlee
The same browser-driven crawling, but as a library you write programs against, aimed at pulling out neat data rather than making archives.
Track this in Scout - internetarchive/heritrix3
The same archiving purpose at very large scale, written in Java, but it does not run the page's code, so modern sites come out incomplete.
Track this in Scout
docker pull webrecorder/browsertrix-crawler docker run -v $PWD/crawls:/crawls/ -it webrecorder/browsertrix-crawler \ crawl --url https://example.com --generateWACZ --text --collection test # result: crawls/collections/test/test.wacz
