25.2k stars · Apache-2.0 · v3.18.1 (2026-08-12), read from /releases/latest · Track this in Scout
A crawling framework for JavaScript that keeps the queue, the retries and the rate limits so you only write the part that reads the page.
▶Repo detailsthe review · specs · pros & cons · install
What it is
A library for Node.js, in JavaScript and TypeScript. It gives you a queue of addresses, storage for what you collected, automatic retries with backing off, rotating proxies, and a choice between plain HTTP requests and a real browser driven by Playwright or Puppeteer. Switching between the two is a change of one class name.What it is good for. Anyone who has written a scraping script and watched it die on page four hundred. The problem it removes is all the bookkeeping around the part you actually care about. A short line about ScalpingMate and the channel: collecting a competitor's published pages, or a calendar of releases, is exactly this shape of job.
- The same code can run with plain requests or with a full browser, so a site that needs JavaScript is a small change rather than a rewrite.
- It handles blocking behaviour for you: proxy rotation, realistic browser fingerprints, and sensible rates.
- Apache-2.0 licence, and a Python version exists as well if Node.js is not your language.
- It is a framework, not a command. For ten pages this is more machinery than you need.
- Running real browsers costs memory. Each one wants several hundred megabytes, so a server has a hard limit on how many run at once.
- Scraping a site can breach its terms of service. Read them, and keep the rate low enough to be polite.
- scrapy/scrapy
The long-established Python equivalent, with more extensions and a much older design that handles browsers less naturally.
Track this in Scout - microsoft/playwright
It drives a browser and nothing else, so it is the layer Crawlee sits on rather than a competitor.
Track this in Scout
# Start a project from the template: npx crawlee create my-crawler cd my-crawler npm start # Or add it to something you already have: npm install crawlee playwright npx playwright install chromium
