Topics / #web-crawler

Web-crawler projects on GitHub

Crawlers and scraping frameworks for extracting data from websites at scale.

As of , the most-starred Web-crawler projects on GitHub are firecrawl (192.5K stars), crawl4ai (85.8K stars) and MediaCrawler (67.1K stars). GitHub.Locker tracks 14 Web-crawler repositories with 506.5K stars combined. The fastest-growing on October 17 was firecrawl, with +311 stars in a day.

14Projects
506.5KStars in total
+759Stars in a day
Browse in the catalog →github.com/topics/web-crawler ↗

Hot right now

Most stars gained in a day · Oct 17

+311

firecrawl

firecrawl/firecrawl · ★ 192.5K

Firecrawl turns any website into clean, LLM-ready data by crawling, scraping, and searching the web, delivering markdown, structured JSON, or screenshots that AI agents can consume directly.

+263

moli

lexmount/moli · ★ 19.3K

Moli is a production-ready headless browser for AI agents, built in Rust. It helps agents fetch and extract web pages, search the web, and automate browser tasks through the CLI, CDP, WebDriver Classic, or WebDriver BiDi.

+63

crawl4ai

unclecode/crawl4ai · ★ 85.8K

Crawl4AI is an open-source web crawler and scraper that turns any website into clean, LLM-ready Markdown for RAG pipelines, AI agents, and data extraction workflows.

+52

MediaCrawler

NanmiCoder/MediaCrawler · ★ 67.1K

A multi-platform crawler for Chinese social media: it collects keyword search results, specific posts, and their comments from Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu.

The all-time top

14 projects · by stars

  1. #1 · firecrawl/firecrawl

    firecrawl

    Firecrawl turns any website into clean, LLM-ready data by crawling, scraping, and searching the web, delivering markdown, structured JSON, or screenshots that AI agents can consume directly.

    192.5K
  2. #2 · unclecode/crawl4ai

    crawl4ai

    Crawl4AI is an open-source web crawler and scraper that turns any website into clean, LLM-ready Markdown for RAG pipelines, AI agents, and data extraction workflows.

    85.8K
  3. #3 · NanmiCoder/MediaCrawler

    MediaCrawler

    A multi-platform crawler for Chinese social media: it collects keyword search results, specific posts, and their comments from Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu.

    67.1K
  1. 4

    Scrapegraph-ai

    31.9KScrapeGraphAI/Scrapegraph-ai · Python

    ScrapeGraphAI is a Python web scraping library that uses LLMs and direct graph logic to build scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown). You just say which information you want to extract and it does the rest.

  2. 5

    crawlee

    26.2Kapify/crawlee · TypeScript

    A web scraping and browser automation library for Node.js: you write crawlers in JavaScript or TypeScript to crawl sites, extract data (e.g. for AI, LLM and RAG pipelines) and save it to disk or cloud.

  3. 6

    moli

    19.3Klexmount/moli · Rust

    Moli is a production-ready headless browser for AI agents, built in Rust. It helps agents fetch and extract web pages, search the web, and automate browser tasks through the CLI, CDP, WebDriver Classic, or WebDriver BiDi.

  4. 7

    katana

    17.7Kprojectdiscovery/katana · Go

    Katana is a next-generation crawling and spidering framework from ProjectDiscovery: point it at URLs and it discovers endpoints, links and forms across a site.

  5. 8

    DrissionPage

    12.5Kg1879/DrissionPage · Python

    DrissionPage is a Python web automation tool that can control a browser, send and receive raw HTTP packets, or combine both in one workflow, giving you browser automation convenience together with requests-level efficiency.

  6. 9

    crawlab

    12.3Kcrawlab-team/crawlab · Go

    Crawlab is an open-source distributed web crawler management platform: you upload spiders written in any language, run them across a cluster of machines, and manage tasks, schedules and results from a web UI.

  7. 10

    reader

    12.2Kjina-ai/reader · TypeScript

    Reader is a service that converts any URL into clean, LLM-friendly markdown by simply prefixing it with https://r.jina.ai/, and it also offers a web-search endpoint at https://s.jina.ai/ that returns readable content for the top search results instead of just titles and snippets.

  8. 11

    webmagic

    11.7Kcode4craft/webmagic · Java

    WebMagic is a scalable web crawler framework for Java. It covers the whole crawler lifecycle — downloading, URL management, content extraction and persistence — to simplify building a specific crawler.

  9. 12

    spider-flow

    11.4Kssssssss-team/spider-flow · Java

    Spider-flow is a next-generation crawler platform that lets you design web scraping workflows as flowcharts instead of writing code.

  10. 13

    wigolo

    5.5KKnockOutEZ/wigolo · TypeScript

    wigolo is a local-first web toolkit for AI coding agents: one surface for search, fetch, crawl, extraction, caching, find-similar, and research, served over MCP, REST, SDKs, and a CLI. No API keys, no cloud, $0 per query.

  11. 14

    jev-seo

    654AgriciDaniel/jev-seo · Python

    jev-seo is a live SEO auditor that crawls a site from one homepage URL, checks it against 52 rules tied to Google Search Central, measures Core Web Vitals, and asks Jev (TypeSafe's System One model) typed questions about every page, producing a PDF, an XLSX action tracker and a Markdown report from the same audit data.

Often tagged together