In simple English
This program pulls posts, comments, and user profiles from popular social media sites. It saves the data into files or databases so you can read it later. It uses a real web browser to log in, which helps avoid blocks. You need some technical skill to run a Python script and install software. The project is free to use on your own computer.
What it is
A Python program that controls a browser using Playwright (a tool that lets code drive a web browser automatically) to collect public posts and comments from multiple social media platforms. It saves you from writing custom scrapers for each different website.
What it is good for
Solo builders who need to collect public feedback, research competitors, or gather text data from social media platforms for market research.
- Supports many popular platforms in one tool.
- Reuses your existing browser session to avoid login blocks.
- Saves scraped data directly to CSV, JSON, Excel, or databases.
- Requires Python, Node.js, and command-line setup.
- Social media platforms frequently change their layouts and break scrapers.
- Scraping platforms may violate their terms of service and get your accounts banned.
Writing custom web scraping scripts for every social media site
Playwright (a tool that lets code drive a web browser automatically)
- scrapy/scrapy65k stars
A fast framework for general web crawling and scraping, though it lacks built-in browser login helpers for social media.
- apache/airflow47k stars
Runs chains of scheduled jobs written in Python, and keeps a record of every run.
- cybertec-postgresql/pg_timetable1.4k stars
A job scheduler for PostgreSQL whose jobs and schedules live in the database itself.
cd MediaCrawler uv sync uv run main.py --platform xhs --lt qrcode --type search