Global

web-crawling
6 projects tagged “web-crawling” that gained the most stars yesterday
#204 · +216 · 1d ·GitHub
Moli

Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

Creator
lexmount
Size
123.5 MB
Stars
19.9K
Updated
CommitOct 20Releasev1.1.18Oct 17
Age
2mo
Commits · 90 Days
#527 · +72 · 1d ·GitHub
Crawl4ai

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord

Creator
unclecode
Size
148.1 MB
Stars
86K
Updated
CommitOct 20Releasev0.9.5Oct 14
Age
2y
Commits · 90 Days
#1518 · +20 · 1d ·GitHub
Scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

Creator
scrapy
Size
32.9 MB
Stars
64.9K
Updated
CommitOct 21Release2.19.0Sep 10
Age
16y
Commits · 90 Days
#2462 · +10 · 1d ·GitHub
Crawlee

js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs.

Creator
apify
Size
172.7 MB
Stars
26.2K
Updated
CommitOct 20Releasev3.18.2Sep 29
Age
10y
Commits · 90 Days
#5220 · +2 · 1d ·GitHub
Botasaurus

The All in One Framework to Build Undefeatable Scrapers

Creator
omkarcloud
Size
89.1 MB
Stars
5.8K
Updated
CommitJul 26
Age
3y
Commits · 90 Days
#6992 · +0 · 1d ·GitHub
Pyspider
archived

pyspider [![Build Status]][Travis CI] [![Coverage Status]][Coverage] ========

Creator
binux
Size
4.1 MB
Stars
16.8K
Updated
CommitAug 02Releasev0.3.10Apr 18
Age
12y
Commits · 90 Days