Topics / #crawling

Crawling projects on GitHub

Web crawling frameworks for large-scale site scraping.

As of , the most-starred Crawling projects on GitHub are Scrapling (88.7K stars), scrapy (64.9K stars) and crawlee (26.2K stars). GitHub.Locker tracks 8 Crawling repositories with 258.8K stars combined. The fastest-growing on October 18 was moli, with +229 stars in a day.

8Projects
258.8KStars in total
+492Stars in a day
Browse in the catalog →github.com/topics/crawling ↗

Hot right now

Most stars gained in a day · Oct 18

+229

moli

lexmount/moli · ★ 19.7K

Moli is a production-ready headless browser for AI agents, built in Rust. It helps agents fetch and extract web pages, search the web, and automate browser tasks through the CLI, CDP, WebDriver Classic, or WebDriver BiDi.

+209

Scrapling

D4Vinci/Scrapling · ★ 88.7K

Scrapling is an adaptive web scraping framework for Python that handles everything from a single HTTP request to a full-scale concurrent crawl, with built-in anti-bot bypass.

+19

maxun

getmaxun/maxun · ★ 17.9K

Maxun is an open-source no-code web data platform that turns any website into structured, reliable data: it handles extraction, crawling, scraping, search and AI-powered data extraction.

+18

scrapy

scrapy/scrapy · ★ 64.9K

Scrapy is a fast, high-level web scraping and crawling framework for Python that extracts structured data from websites.

The all-time top

8 projects · by stars

  1. #1 · D4Vinci/Scrapling

    Scrapling

    Scrapling is an adaptive web scraping framework for Python that handles everything from a single HTTP request to a full-scale concurrent crawl, with built-in anti-bot bypass.

    88.7K
  2. #2 · scrapy/scrapy

    scrapy

    Scrapy is a fast, high-level web scraping and crawling framework for Python that extracts structured data from websites.

    64.9K
  3. #3 · apify/crawlee

    crawlee

    A web scraping and browser automation library for Node.js: you write crawlers in JavaScript or TypeScript to crawl sites, extract data (e.g. for AI, LLM and RAG pipelines) and save it to disk or cloud.

    26.2K
  1. 4

    colly

    25.6Kgocolly/colly · Go

    Colly is a scraping and crawling framework for Go: a clean interface for writing any kind of crawler, scraper or spider that extracts structured data from websites, for things like data mining, processing or archiving.

  2. 5

    moli

    19.7Klexmount/moli · Rust

    Moli is a production-ready headless browser for AI agents, built in Rust. It helps agents fetch and extract web pages, search the web, and automate browser tasks through the CLI, CDP, WebDriver Classic, or WebDriver BiDi.

  3. 6

    maxun

    17.9Kgetmaxun/maxun · TypeScript

    Maxun is an open-source no-code web data platform that turns any website into structured, reliable data: it handles extraction, crawling, scraping, search and AI-powered data extraction.

  4. 7

    newspaper

    15.2Kcodelucas/newspaper · Python

    Newspaper3k is a Python 3 library that downloads, parses, and extracts the article text, authors, publish date, top image, and keywords/summary from news articles and news sites.

  5. 8

    deepcrawl

    690lumpinif/deepcrawl · TypeScript

    An open-source, agent-oriented web data extraction platform — a self-deployable Firecrawl alternative that turns public pages into clean Markdown, a links tree, and LLM-friendly metadata.

Often tagged together