Global

web-scraping
166 projects tagged “web-scraping” that gained the most stars yesterday
#3 · +3.4K · 1d ·GitHub

The job search that runs on your machine. AI job application framework built on Claude Code: evaluate postings, tailor CVs, write cover letters, prep interviews. Fork it and own it.

Size
5 MB
Stars
54.3K
Age
7mo
Commits · 90 Days
#41 · +969 · 1d ·GitHub

Applicant Profiles Directory - manage profiles, linked devices, scan configs, and WhatsApp connections

Size
21.1 MB
Stars
1.7K
Age
4mo
Commits · 90 Days
#101 · +409 · 1d ·GitHub

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.

Size
2.6 MB
Stars
99.7K
Age
7mo
Commits · 90 Days
#123 · +356 · 1d ·GitHub

Stealth browser automation for AI agents over MCP and the Chrome DevTools Protocol: navigation, network hooks, DOM extraction, and pixel-accurate UI cloning.

Creator
vibheksoni
Size
19.6 MB
Stars
3.1K
Age
1y
Commits · 90 Days
#130 · +331 · 1d ·GitHub

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

Creator
firecrawl
Size
181.9 MB
Stars
193.1K
Age
2y
Commits · 90 Days
#145 · +304 · 1d ·GitHub

知らない分野を、原文にたどりつくまで芋づる式に掘る Claude Code のスキル

Creator
MaryCache
Size
0.2 MB
Stars
780
Updated
CommitOct 19
Age
7d
Commits · 90 Days
#184 · +229 · 1d ·GitHub

Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

Creator
lexmount
Size
122 MB
Stars
19.7K
Age
2mo
Commits · 90 Days
#203 · +209 · 1d ·GitHub

🕷️ A Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: and follow here for daily tips and tricks:

Creator
D4Vinci
Size
15.5 MB
Stars
88.7K
Age
2y
Commits · 90 Days
#233 · +167 · 1d ·GitHub

The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.

Creator
apify
Size
9.9 MB
Stars
11.9K
Age
1y
Commits · 90 Days
#358 · +111 · 1d ·GitHub

Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.

Size
36.5 MB
Stars
118.5K
Age
1y
Commits · 90 Days
#375 · +104 · 1d ·GitHub

小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频 | 评论爬虫、微博帖子 | 评论爬虫、百度贴吧帖子 | 百度贴吧评论回复爬虫 | 知乎问答文章|评论爬虫

Creator
NanmiCoder
Size
29.8 MB
Stars
67.2K
Updated
CommitOct 15
Age
3y
Commits · 90 Days
#423 · +92 · 1d ·GitHub

Teach your agent a website once. After that, it calls the site like an API instead of driving a browser.

Size
7 MB
Stars
1.2K
Updated
CommitOct 19
Age
22d
Commits · 90 Days
#440 · +88 · 1d ·GitHub

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.

Creator
microsoft
Size
239.7 MB
Stars
98K
Age
6y
Commits · 90 Days
#442 · +87 · 1d ·GitHub

The 30 Days of Python programming challenge is a step-by-step guide to learn the Python programming language in 30 days. This challenge may take more than 100 days. Follow your own pace.

Creator
Asabeneh
Size
30.8 MB
Stars
76.6K
Updated
CommitSep 30
Age
6y
Commits · 90 Days

These skills help agents find outlier posts, mine comments, summarize transcripts, tear down competitors, analyze ad libraries, and turn public social data into useful business artifacts.

Size
<0.1 MB
Stars
4.2K
Updated
CommitAug 26
Age
4mo
Commits · 90 Days
#543 · +68 · 1d ·GitHub

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.

Creator
unclecode
Size
148 MB
Stars
85.8K
Age
2y
Commits · 90 Days
#580 · +65 · 1d ·GitHub

ScrapeGraphAI is a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.).

Size
16.9 MB
Stars
32K
Age
2y
Commits · 90 Days
#610 · +62 · 1d ·GitHub

Flutter 多端独立短剧 / 影视应用,原名「短剧库 APP」。站源请求、解析、下载和播放均在设备上完成,不依赖自建服务。当前源码版本:0.2.51+57。

Size
1.1 MB
Stars
1.2K
Age
23d
Commits · 90 Days
#646 · +57 · 1d ·GitHub

Browser automation CLI for AI agents. Fast native Rust CLI.

Size
24.4 MB
Stars
44.3K
Age
9mo
Commits · 90 Days
#692 · +52 · 1d ·GitHub

8 安装器支持选择数据位置:程序位于所选目录的 app,数据默认位于同级 data,也可另选文件夹。旧版 AppData 数据可在首次启动确认后复制、逐文件校验并迁移,保留原副本;微信原始数据库仍在原位置。升级与卸载保留用户数据。详见 数据目录与迁移。维护者的账号、密钥、联系人、聊天和运行日志未包含在源码中。

Creator
sourcepro5
Size
3.3 MB
Stars
225
Age
17d
Commits · 90 Days
#705 · +50 · 1d ·GitHub

Lightpanda: the headless browser designed for AI and automation

Size
39.1 MB
Stars
36.7K
Age
3y
Commits · 90 Days
#743 · +47 · 1d ·GitHub

Give your AI coding agent a URL and watch it recreate the website as a clean Next.js app.

Creator
JCodesMore
Size
1.4 MB
Stars
37K
Age
7mo
Commits · 90 Days
#772 · +45 · 1d ·GitHub

Python - 100天从新手到大师

Creator
jackfrued
Size
396.1 MB
Stars
187.3K
Updated
CommitJul 29
Age
8y
Commits · 90 Days
#864 · +38 · 1d ·GitHub

IPcook provides reliable residential proxies for online research, username discovery, and public data collection workflows. 2/GB.

Creator
soxoj
Size
29.8 MB
Stars
39.2K
Age
6y
Commits · 90 Days
#868 · +38 · 1d ·GitHub

Make Any Website into CLI & Use your logged-in browser by AI agent.

Creator
jackwener
Size
18 MB
Stars
30.4K
Age
7mo
Commits · 90 Days
#888 · +37 · 1d ·GitHub

The headless browser for AI agents and web scraping

Size
24.7 MB
Stars
29.1K
Age
6mo
Commits · 90 Days
#893 · +37 · 1d ·GitHub

Extract Google Maps business leads, emails, reviews, phone numbers, websites, ratings, coordinates, and more with a free open-source CLI, Web UI, REST API, and optional self-hosted platform.

Creator
gosom
Size
26.6 MB
Stars
6.6K
Age
3y
Commits · 90 Days
#939 · +35 · 1d ·GitHub

SpiderFoot is an open source intelligence (OSINT) automation tool. It integrates with just about every data source available and utilises a range of methods for data analysis, making that data easy to navigate.

Creator
smicallef
Size
16.1 MB
Stars
23.6K
Age
14y
Commits · 90 Days
#1004 · +32 · 1d ·GitHub

给你的本机 Agent(如 Claude Code、Grok Bot 等)装上完整联网能力的 Skill:三层通道调度 + 浏览器 CDP + 并行分治,自动积累站点经验,不断优化联网效率

Creator
eze-is
Size
0.2 MB
Stars
9.2K
Age
7mo
Commits · 90 Days
#1029 · +31 · 1d ·GitHub

Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.

Creator
jo-inc
Size
15.3 MB
Stars
11.7K
Age
8mo
Commits · 90 Days