Global

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

🕷️ A Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: and follow here for daily tips and tricks:

The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.

小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频 ｜ 评论爬虫、微博帖子 ｜ 评论爬虫、百度贴吧帖子 ｜ 百度贴吧评论回复爬虫 | 知乎问答文章｜评论爬虫

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.

ScrapeGraphAI is a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.).

The open-source headless browser for AI agents and web scraping.

Open Source AI Platform - AI Chat with advanced features that works with every LLM

🚀 自托管的抖音、TikTok 数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok & Douyin scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos.

Turn any website into a structured API. Extract, automate, search and monitor the web.

Scrapy, a fast high-level web crawling & scraping framework for Python.

AI联网搜索、文件解析及网页内容抓取等智能体核心能力提供商： 安思派开放平台(Anspire Open)是面向智能体时代的领先的基础设施提供商。我们为开发者提供构建强大智能体所需的核心能力栈，现已上线AI联网搜索【多版本，极具竞争力的价格】、文件解析【限免】及网页内容抓取【限免】、云端浏览器自动化（Anspire Browser Agent）【内测】、多轮改写等服务，持续为智能体连接并操作复杂的数字世界提供坚实基础。可无缝集成至Dify、Coze、元器等主流智能体平台。通过透明点数计费体系与模块化设计，为企业提供高效、低成本的定制化支持，加速智能化升级…

Native high-speed film and television streaming client supported by CatPaw/TVBOX protocol.

四个平台，四条几乎一样的命令，输出同一种结构的 JSON。笔记、作品、稿件、推文一律叫 item，搜索一律叫 search，点赞一律叫 like。学一次，十个平台都会用。

js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs.

收集整理 GitHub 上高质量、有趣的开源项目。

「阅读」APP 不同于其他小说阅读软件，本质是个空壳浏览器，需要自行编写或导入书源才能使用，其原理为：解析小说网站的搜索页、详情页、目录页、正文页，提取相关信息来显示给你看，该项目只是分享我自用的书源

Hister is a private search engine for the pages you visit and the files you keep. It indexes their full contents so you can find information again from the web interface, terminal, or an AI assistant connected through MCP.

A completely free (including for commercial use and secondary development) visual browser automation test/data collection/crawler software, which can be used to design and execute tasks in a code-free visual way.

🗃 Open source self-hosted web archiving. , saves HTML, JS, PDFs, media, and more...

This GitHub repo is a powerhouse collection of scraping APIs for developers that you can start using immediately to build everything from simple automations to full-scale applications.

🐙 CDP Mode bypasses bot-detection with Chromium-based browsers. 🎭 Stealthy Playwright Mode extends CDP Mode's stealth to Playwright. for the main framework.

Convert any URL to an LLM-friendly input with a simple prefix

Lightning Fast and Elegant Scraping Framework for Gophers

Node.js（app.js）每隔1s读取一次file.txt文件，并将读取结果通过socket.io推送给客户端（index.html）。

26+ to install successfully. If you encounter any installation issues, we recommend trying with the latest available version of Go, as the minimum required version may have changed. Run the command below or download a pre-compiled binary from the release page.

The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.

👾 Fast and simple video download library and CLI tool written in Go

   想自建爬虫？接入 Bright Data MCP Server，让 Claude、Cursor、Windsurf 等 AI 助手直接实时抓取网页——自动破解验证码、绕过地区限制。Scraper Studio 支持 AI 一键生成或 JS 代码定制，全托管基础设施运行，无需自购代理、无需搭服务器，分钟级上线。所有产品底层均由同一套顶级代理网络驱动。

A browser extension for automating your browser by connecting blocks