Topics / #web-crawling

Web-crawling projects on GitHub

Automated crawling of websites to extract and index data

As of , the most-starred Web-crawling projects on GitHub are crawl4ai (85.8K stars), scrapy (64.9K stars) and crawlee (26.2K stars). GitHub.Locker tracks 6 Web-crawling repositories with 218.9K stars combined. The fastest-growing on October 18 was moli, with +229 stars in a day.

6Projects
218.9KStars in total
+327Stars in a day
Browse in the catalog →github.com/topics/web-crawling ↗

Hot right now

Most stars gained in a day · Oct 18

+229

moli

lexmount/moli · ★ 19.5K

Moli is a production-ready headless browser for AI agents, built in Rust. It helps agents fetch and extract web pages, search the web, and automate browser tasks through the CLI, CDP, WebDriver Classic, or WebDriver BiDi.

+68

crawl4ai

unclecode/crawl4ai · ★ 85.8K

Crawl4AI is an open-source web crawler and scraper that turns any website into clean, LLM-ready Markdown for RAG pipelines, AI agents, and data extraction workflows.

+18

scrapy

scrapy/scrapy · ★ 64.9K

Scrapy is a fast, high-level web scraping and crawling framework for Python that extracts structured data from websites.

+11

crawlee

apify/crawlee · ★ 26.2K

A web scraping and browser automation library for Node.js: you write crawlers in JavaScript or TypeScript to crawl sites, extract data (e.g. for AI, LLM and RAG pipelines) and save it to disk or cloud.

The all-time top

6 projects · by stars

  1. #1 · unclecode/crawl4ai

    crawl4ai

    Crawl4AI is an open-source web crawler and scraper that turns any website into clean, LLM-ready Markdown for RAG pipelines, AI agents, and data extraction workflows.

    85.8K
  2. #2 · scrapy/scrapy

    scrapy

    Scrapy is a fast, high-level web scraping and crawling framework for Python that extracts structured data from websites.

    64.9K
  3. #3 · apify/crawlee

    crawlee

    A web scraping and browser automation library for Node.js: you write crawlers in JavaScript or TypeScript to crawl sites, extract data (e.g. for AI, LLM and RAG pipelines) and save it to disk or cloud.

    26.2K
  1. 4

    moli

    19.5Klexmount/moli · Rust

    Moli is a production-ready headless browser for AI agents, built in Rust. It helps agents fetch and extract web pages, search the web, and automate browser tasks through the CLI, CDP, WebDriver Classic, or WebDriver BiDi.

  2. 5

    pyspider

    16.8Kbinux/pyspider · Python

    pyspider is a powerful web spider (crawler) system written in Python: you write crawling scripts in Python and manage your scraping projects through a built-in web interface.

  3. 6

    botasaurus

    5.8Komkarcloud/botasaurus · Python

    Botasaurus is an all-in-one Python web scraping framework built to make scrapers that pass bot-detection checks, including Cloudflare WAF, BrowserScan, Fingerprint, Datadome, and Cloudflare Turnstile CAPTCHA.

Often tagged together