Data & Research · Web Scraping & Crawling
0 paid against 16 free you can run yourself.
Paid against free › Data & Research › Web Scraping & Crawling › Web Crawlers · Automation and scraping on Repo Radar
Prices and free tiers as checked, dated on each review.
Paid tools for this job are being added. The free alternatives below are ready to use.
Their home is another job; they cover part of this one too.
Free to use. You run it on your own server.
They do this job.
A command-line downloader that also collects every detail a video site publishes about a video or a whole channel.
★ 197kFreeFIfirecrawl/firecrawlA crawler service that turns websites into clean Markdown or structured data.
★ 191kFreeCRunclecode/crawl4aiA crawler that turns web pages into clean text for a language model.
★ 85.7kFreeSCscrapy/scrapyA Python framework for fetching and parsing large numbers of web pages.
★ 64.9kFreeRSDIYgod/RSSHubTurns almost any page into an RSS feed — Reddit searches, YouTube channels, Hacker News — so competitor and keyword monitoring becomes a notification.
★ 46.5kFreeCRapify/crawleeA crawling framework for JavaScript that manages the queue, the retries and the rate limits.
★ 26.2kFreeGAmikf/gallery-dlIt downloads every picture from an album or a profile with one command.
★ 20.1kFreeDEkepano/defuddleIt extracts the main article from a web page and returns it as Markdown.
★ 9.7kFreeRSrss-bridge/rss-bridgeIt makes an RSS feed for a website that removed its own.
★ 9.3kFreeYOjdepoix/youtube-transcript-apiFetches the subtitles of a public YouTube video as text, with no key and no browser.
★ 8.4kFreeHTmgdm/htmlqLike `jq`, but for HTML: it pulls parts out of a page using CSS selectors.
★ 7.6kFreeTRadbar/trafilaturaA Python tool that pulls the article out of a web page and leaves the menus and adverts behind.
★ 7kFreeYTNotJoeMartinez/yt-ftsFull-text search across everything said in a YouTube channel, with a link to the exact second.
★ 1.8kFreeYOegbertbouman/youtube-comment-downloaderA small Python tool that downloads the comments under a video as a data file.
★ 1.3kFreeBRwebrecorder/browsertrix-crawlerSaves a web page in a real browser, so the copy still works later.
★ 1.2kFreeSEjadchaar/sec-edgar-downloaderA small Python package that downloads a United States company's official filings to a local folder.
★ 717FreeA second choice: they cover part of this job or are a building block for it.
Installs and looks after one reader per website so an assistant can read public web content.
★ 95.7kFreeSCD4Vinci/ScraplingA Python toolkit for collecting data from web pages that can find an item again after the page layout changes.
★ 87.4kFreeCHdgtlmoon/changedetection.ioA self-hosted monitor that watches web pages and notifies you when their content changes.
★ 34.9kFreeARArchiveBox/ArchiveBoxIt keeps a full copy of every page saved, so the link going dead does not matter.
★ 28.8kFreeAUalirezamika/autoscraperA Python scraper that learns what to extract from one example you give it.
★ 8.1kFreeSTsteel-dev/steel-browser🔥 Open Source Browser API for AI Agents & Apps. Steel Browser is a batteries-included browser sandbox that lets you automate the web without worrying about infrastructure.
★ 7.8kFreeBOomkarcloud/botasaurusA Python framework for collecting data from web pages that are trying to block automated visitors.
★ 5.8kFreeAIai-robots-txt/ai.robots.txtA frequently updated list of AI scrapers, ready to paste into a web server.
★ 4.2kFreeMOmorphik-org/morphik-coreOpen-source multimodal retrieval engine (Morphik Core)
★ 3.7kFreeCUlexiforest/curl-impersonateA version of curl whose connections look exactly like a real browser's.
★ 2.8kFreeMEmicrolinkhq/metascraperIt reads the title, description and picture out of any web page.
★ 2.7kFreeSEMinishLab/semhashFast removal of near-duplicate text from a dataset.
★ 975FreeWEEdJoPaTo/website-stalkerTrack changes on websites via git
★ 75FreeWEmborsetti/webchangeswebchanges anonymously checks web content (including images) and commands for changes, delivering instant notifications and AI-powered summaries to your favorite platform.
★ 50FreeSPspyweb-app/spywebTiny web monitoring, scraping & browser automation engine with Lua scripting, built-in storage, dedup, scheduler, hot reload, programmable API server, test runner and desktop/webhook alerts. ~8MB self-contained binary with <10MB idle RAM.
★ 45FreeCOLaksh-star/competitive-intelligenceAI-powered competitive intelligence monitor using CocoIndex, Tavily Search, and LLM extraction
★ 23FreeOne job at a time: the paid tools worth paying for, the free ones you can run yourself, and who should pick which.
No tracking pixels. One click to leave. We use your address to send Stack Radar and nothing else. Privacy.