Topics / #web-scraper

Web-scraper projects on GitHub

Extracting structured data from websites, often with headless browsers.

As of , the most-starred Web-scraper projects on GitHub are firecrawl (192.5K stars), Agent-Reach (98.8K stars) and puppeteer (95.7K stars). GitHub.Locker tracks 14 Web-scraper repositories with 636.9K stars combined. The fastest-growing on October 16 was Agent-Reach, with +513 stars in a day.

14Projects
636.9KStars in total
+1.5KStars in a day
Browse in the catalog →github.com/topics/web-scraper ↗

Hot right now

Most stars gained in a day · Oct 16

+513

Agent-Reach

Panniantong/Agent-Reach · ★ 98.8K

Agent Reach is a capability layer that gives an AI agent the ability to actually read and search the internet — Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, RSS, web pages and more — instead of hitting paywalls, 403s or login walls.

+334

firecrawl

firecrawl/firecrawl · ★ 192.5K

Firecrawl turns any website into clean, LLM-ready data by crawling, scraping, and searching the web, delivering markdown, structured JSON, or screenshots that AI agents can consume directly.

+317

moli

lexmount/moli · ★ 19.3K

Moli is a production-ready headless browser for AI agents, built in Rust. It helps agents fetch and extract web pages, search the web, and automate browser tasks through the CLI, CDP, WebDriver Classic, or WebDriver BiDi.

+201

Scrapling

D4Vinci/Scrapling · ★ 88.3K

Scrapling is an adaptive web scraping framework for Python that handles everything from a single HTTP request to a full-scale concurrent crawl, with built-in anti-bot bypass.

The all-time top

14 projects · by stars

  1. #1 · firecrawl/firecrawl

    firecrawl

    Firecrawl turns any website into clean, LLM-ready data by crawling, scraping, and searching the web, delivering markdown, structured JSON, or screenshots that AI agents can consume directly.

    192.5K
  2. #2 · Panniantong/Agent-Reach

    Agent-Reach

    Agent Reach is a capability layer that gives an AI agent the ability to actually read and search the internet — Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, RSS, web pages and more — instead of hitting paywalls, 403s or login walls.

    98.8K
  3. #3 · puppeteer/puppeteer

    puppeteer

    Puppeteer is a JavaScript library that gives you a high-level API to control Chrome or Firefox programmatically, either over the DevTools Protocol or WebDriver BiDi.

    95.7K
  1. 4

    Scrapling

    88.3KD4Vinci/Scrapling · Python

    Scrapling is an adaptive web scraping framework for Python that handles everything from a single HTTP request to a full-scale concurrent crawl, with built-in anti-bot bypass.

  2. 5

    scrapy

    64.9Kscrapy/scrapy · Python

    Scrapy is a fast, high-level web scraping and crawling framework for Python that extracts structured data from websites.

  3. 6

    Scrapegraph-ai

    31.9KScrapeGraphAI/Scrapegraph-ai · Python

    ScrapeGraphAI is a Python web scraping library that uses LLMs and direct graph logic to build scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown). You just say which information you want to extract and it does the rest.

  4. 7

    moli

    19.3Klexmount/moli · Rust

    Moli is a production-ready headless browser for AI agents, built in Rust. It helps agents fetch and extract web pages, search the web, and automate browser tasks through the CLI, CDP, WebDriver Classic, or WebDriver BiDi.

  5. 8

    maxun

    17.9Kgetmaxun/maxun · TypeScript

    Maxun is an open-source no-code web data platform that turns any website into structured, reliable data: it handles extraction, crawling, scraping, search and AI-powered data extraction.

  6. 9

    Photon

    13.3Ks0md3v/Photon · Python

    Photon is a fast web crawler built for OSINT (open-source intelligence) gathering, extracting URLs, emails, social media accounts, secret keys, subdomains and more while it crawls a target site.

  7. 10

    google-maps-scraper

    6.5Kgosom/google-maps-scraper · Go

    A free, open-source tool that scrapes Google Maps to extract business leads such as name, address, phone, website, rating, reviews, email, and GPS coordinates.

  8. 11

    skills

    6.2Kbrowser-act/skills · Python

    BrowserAct Skills is a browser automation CLI/skill set that lets AI agents control a real browser, break through anti-bot protections, hand off to a human when stuck, and run parallel, isolated multi-account sessions.

  9. 12

    deepcrawl

    691lumpinif/deepcrawl · TypeScript

    An open-source, agent-oriented web data extraction platform — a self-deployable Firecrawl alternative that turns public pages into clean Markdown, a links tree, and LLM-friendly metadata.

  10. 13

    oc

    542only-cli/oc · JavaScript

    oc is a command-line tool that turns any website into a compact, numbered text view so AI coding agents like Claude Code, Codex, and Antigravity can browse the web in a few hundred tokens instead of tens of thousands.

  11. 14

    tiktok-signature

    398n1tr00-10/tiktok-signature · Python

    This is a Python library that generates the signatures TikTok's Web API requires on requests — X-Gnarly, X-Bogus, and X-Dynosaur — matching SDK version 5.1.2 (build 1.0.0.316).

Often tagged together