Topics / #data-collection

Data-collection projects on GitHub

Collecting and ingesting data from sources into pipelines and warehouses.

As of , the most-starred Data-collection projects on GitHub are MediaCrawler (67K stars), EasySpider (44.7K stars) and airbyte (22.2K stars). GitHub.Locker tracks 9 Data-collection repositories with 205.2K stars combined. The fastest-growing on October 16 was MediaCrawler, with +60 stars in a day.

9Projects
205.2KStars in total
+172Stars in a day
Browse in the catalog →github.com/topics/data-collection ↗

Hot right now

Most stars gained in a day · Oct 16

+60

MediaCrawler

NanmiCoder/MediaCrawler · ★ 67K

A multi-platform crawler for Chinese social media: it collects keyword search results, specific posts, and their comments from Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu.

+45

Douyin_TikTok_Download_API

Evil0ctal/Douyin_TikTok_Download_API · ★ 20.7K

A self-hosted data API for Douyin and TikTok: it pulls posts, author profiles, comments, search and playlists, downloads videos and image albums without a watermark, and exposes it all through a REST API, an MCP server, a CLI and a web console.

+30

TikTokDownloader

JoeanAmier/TikTokDownloader · ★ 16.9K

DouK-Downloader (formerly TikTokDownloader) is a tool for downloading videos, images and live streams from Douyin (Chinese TikTok) and TikTok, and for collecting platform data such as account details, comments and search results.

+11

theHarvester

laramies/theHarvester · ★ 18K

An open-source OSINT tool that gathers emails, subdomains and names about a domain or organization from public sources: search engines, certificate transparency logs, DNS datasets, code repositories and threat-intel platforms, normalizing hostnames, emails, IPs, URLs, ASNs, people and breach names.

The all-time top

9 projects · by stars

  1. #1 · NanmiCoder/MediaCrawler

    MediaCrawler

    A multi-platform crawler for Chinese social media: it collects keyword search results, specific posts, and their comments from Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu.

    67K
  2. #2 · NaiboWang/EasySpider

    EasySpider

    EasySpider (易采集) is a visual, no-code web crawler and browser automation tool. You select elements on a webpage with clicks instead of writing code, and it designs and runs the scraping/testing task for you.

    44.7K
  3. #3 · airbytehq/airbyte

    airbyte

    Airbyte is an open-source data movement platform for ELT pipelines and AI agents: it moves data from APIs, databases, and files into warehouses, data lakes, and AI applications, backed by a catalog of 600+ connectors. You can run it self-hosted or as Airbyte Cloud.

    22.2K
  1. 4

    Douyin_TikTok_Download_API

    20.7KEvil0ctal/Douyin_TikTok_Download_API · Python

    A self-hosted data API for Douyin and TikTok: it pulls posts, author profiles, comments, search and playlists, downloads videos and image albums without a watermark, and exposes it all through a REST API, an MCP server, a CLI and a web console.

  2. 5

    theHarvester

    18Klaramies/theHarvester · Python

    An open-source OSINT tool that gathers emails, subdomains and names about a domain or organization from public sources: search engines, certificate transparency logs, DNS datasets, code repositories and threat-intel platforms, normalizing hostnames, emails, IPs, URLs, ASNs, people and breach names.

  3. 6

    TikTokDownloader

    16.9KJoeanAmier/TikTokDownloader · JavaScript

    DouK-Downloader (formerly TikTokDownloader) is a tool for downloading videos, images and live streams from Douyin (Chinese TikTok) and TikTok, and for collecting platform data such as account details, comments and search results.

  4. 7

    API-mega-list

    7.8Kcporter202/API-mega-list · JavaScript

    API Mega List is a massive README directory of ready-to-use APIs for building apps and automations, listing 11,860 core directory APIs plus 318 sponsored partner API listings across 24 categories.

  5. 8

    scraping-apis-for-devs

    3.9Kcporter202/scraping-apis-for-devs · JavaScript

    A giant categorized directory of 2,622 ready-to-use scraping APIs for pulling data from websites, social media, e-commerce sites, job boards, real estate listings and more, so you don't have to build scrapers from scratch.

  6. 9

    prediction-market-analysis

    3.9KJon-Becker/prediction-market-analysis · Python

    It's a framework for collecting and analyzing prediction market data from Polymarket and Kalshi, bundled with the largest publicly available dataset of their market and trade history.

Often tagged together