Topics / #data-extraction

Data-extraction projects on GitHub

Extracting structured data from websites and documents.

As of , the most-starred Data-extraction projects on GitHub are firecrawl (192.8K stars), markitdown (191K stars) and Scrapling (88.5K stars). GitHub.Locker tracks 32 Data-extraction repositories with 1M stars combined. The fastest-growing on October 17 was firecrawl, with +311 stars in a day.

32Projects
1MStars in total
+1KStars in a day
Browse in the catalog →github.com/topics/data-extraction ↗

Hot right now

Most stars gained in a day · Oct 17

+311

firecrawl

firecrawl/firecrawl · ★ 192.8K

Firecrawl turns any website into clean, LLM-ready data by crawling, scraping, and searching the web, delivering markdown, structured JSON, or screenshots that AI agents can consume directly.

+188

Scrapling

D4Vinci/Scrapling · ★ 88.5K

Scrapling is an adaptive web scraping framework for Python that handles everything from a single HTTP request to a full-scale concurrent crawl, with built-in anti-bot bypass.

+160

markitdown

microsoft/markitdown · ★ 191K

MarkItDown is a lightweight Python utility that converts files like PDF, PowerPoint, Word, Excel, images, audio, HTML, CSV/JSON/XML, ZIP archives, EPubs and even YouTube URLs into Markdown, designed for feeding documents into LLMs and text analysis pipelines.

+77

Noria

AhmadHassan-BTed/Noria · ★ 1.2K

Noria is an agentic extractor that receives messages and URLs via WhatsApp, scrapes the linked web pages, evaluates the content against custom scoring matrices (e.g. jobs, scholarships) using an LLM, and sends back a structured summary with a calculated match score to the assigned WhatsApp number.

The all-time top

32 projects · by stars

  1. #1 · firecrawl/firecrawl

    firecrawl

    Firecrawl turns any website into clean, LLM-ready data by crawling, scraping, and searching the web, delivering markdown, structured JSON, or screenshots that AI agents can consume directly.

    192.8K
  2. #2 · microsoft/markitdown

    markitdown

    MarkItDown is a lightweight Python utility that converts files like PDF, PowerPoint, Word, Excel, images, audio, HTML, CSV/JSON/XML, ZIP archives, EPubs and even YouTube URLs into Markdown, designed for feeding documents into LLMs and text analysis pipelines.

    191K
  3. #3 · D4Vinci/Scrapling

    Scrapling

    Scrapling is an adaptive web scraping framework for Python that handles everything from a single HTTP request to a full-scale concurrent crawl, with built-in anti-bot bypass.

    88.5K
  1. 4

    crawl4ai

    85.8Kunclecode/crawl4ai · Python

    Crawl4AI is an open-source web crawler and scraper that turns any website into clean, LLM-ready Markdown for RAG pipelines, AI agents, and data extraction workflows.

  2. 5

    MediaCrawler

    67.2KNanmiCoder/MediaCrawler · Python

    A multi-platform crawler for Chinese social media: it collects keyword search results, specific posts, and their comments from Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu.

  3. 6

    scrapy

    64.9Kscrapy/scrapy · Python

    Scrapy is a fast, high-level web scraping and crawling framework for Python that extracts structured data from websites.

  4. 7

    marker

    40.5Kdatalab-to/marker · Python

    Marker is a document conversion tool that turns PDFs (and other document types) into markdown, JSON, HTML, or chunked output quickly and accurately, extracting tables, equations, forms, and images along the way.

  5. 8

    langextract

    39Kgoogle/langextract · Python

    LangExtract is a Python library that uses LLMs to pull structured information out of unstructured text, mapping each extraction back to its exact location in the source document.

  6. 9

    Scrapegraph-ai

    31.9KScrapeGraphAI/Scrapegraph-ai · Python

    ScrapeGraphAI is a Python web scraping library that uses LLMs and direct graph logic to build scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown). You just say which information you want to extract and it does the rest.

  7. 10

    EasyOCR

    30.1KJaidedAI/EasyOCR · Python

    A ready-to-use, open-source OCR library for Python: it detects and reads text in images across 80+ languages and popular writing scripts — Latin, Chinese, Arabic, Devanagari, Cyrillic and more.

  8. 11

    stagehand

    25.7Kbrowserbase/stagehand · TypeScript

    Stagehand is an SDK for building browser agents — code or AI agents that drive a real browser to extract data and interact with any website, with SDKs for TypeScript, Python, and Go.

  9. 12

    surya

    21.5Kdatalab-to/surya · Python

    Surya is Datalab's open-source document OCR system built around a 650M parameter model. It reads text from images and PDFs, and also does layout analysis, reading order, and table recognition.

  10. 13

    maxun

    17.9Kgetmaxun/maxun · TypeScript

    Maxun is an open-source no-code web data platform that turns any website into structured, reliable data: it handles extraction, crawling, scraping, search and AI-powered data extraction.

  11. 14

    pyspider

    16.8Kbinux/pyspider · Python

    pyspider is a powerful web spider (crawler) system written in Python: you write crawling scripts in Python and manage your scraping projects through a built-in web interface.

  12. 15

    twint

    16.4Ktwintproject/twint · Python

    Twint is a Python tool for scraping Twitter data — tweets, followers, following, likes and more — without using Twitter's official API, so it isn't bound by API limits or sign-up requirements.

  13. 16

    unstructured

    15.6KUnstructured-IO/unstructured · HTML

    Unstructured is an open-source ETL toolkit that converts complex documents like PDFs, Word docs, HTML and images into clean, structured data (JSON) ready for LLMs, RAG pipelines and vector databases.

  14. 17

    gjson

    15.6Ktidwall/gjson · Go

    GJSON is a Go library for pulling values out of JSON documents quickly — one call with a path like 'name.last' returns the value. It also supports iteration and JSON Lines.

  15. 18

    goquery

    15KPuerkitoBio/goquery · Go

    goquery is a Go library that brings jQuery-like syntax and features to HTML parsing and manipulation, built on Go's net/html package and the cascadia CSS selector library.

  16. 19

    HackBrowserData

    14.6KmoonD4rk/HackBrowserData · Go

    HackBrowserData is a command-line tool that decrypts and exports browser data — passwords, cookies, history, bookmarks, credit cards and more — from Chromium-based browsers, Firefox and Safari, built for security research.

  17. 20

    zerox

    12.2Kgetomni-ai/zerox · TypeScript

    Zerox is an open-source OCR and document-extraction tool powered by vision models: give it a PDF, DOCX or image and it returns Markdown — or structured data if you provide a schema.

  18. 21

    spider-flow

    11.4Kssssssss-team/spider-flow · Java

    Spider-flow is a next-generation crawler platform that lets you design web scraping workflows as flowcharts instead of writing code.

  19. 22

    pdfplumber

    10.8Kjsvine/pdfplumber · Python

    pdfplumber is a Python library that inspects PDFs in detail, exposing each character, line, rectangle, and curve, and makes it easy to extract text and tables from them.

  20. 23

    skills

    6.2Kbrowser-act/skills · Python

    BrowserAct Skills is a browser automation CLI/skill set that lets AI agents control a real browser, break through anti-bot protections, hand off to a human when stuck, and run parallel, isolated multi-account sessions.

  21. 24

    scraping-apis-for-devs

    3.9Kcporter202/scraping-apis-for-devs · JavaScript

    A giant categorized directory of 2,622 ready-to-use scraping APIs for pulling data from websites, social media, e-commerce sites, job boards, real estate listings and more, so you don't have to build scrapers from scratch.

  22. 25

    lift

    1.2Kdatalab-to/lift · Python

    lift is an open-source 9B vision-language model and toolkit that extracts structured JSON data from PDFs and images by matching a schema you provide, using schema-constrained decoding to guarantee valid output.

  23. 26

    Noria

    1.2KAhmadHassan-BTed/Noria · JavaScript

    Noria is an agentic extractor that receives messages and URLs via WhatsApp, scrapes the linked web pages, evaluates the content against custom scoring matrices (e.g. jobs, scholarships) using an LLM, and sends back a structured summary with a calculated match score to the assigned WhatsApp number.

  24. 27

    figranium

    769figranium/figranium · TypeScript

    Figranium is an open-source, deterministic browser automation platform: you stack blocks to build scraping and automation workflows visually, then call them as APIs — no usage credits or third-party data hosting.

  25. 28

    EQAI

    742AhmadHassan-BTed/EQAI · Python

    This is a reverse-engineering pipeline that transplants the original EverQuest Online Adventures (EQOA) character geometry into the EverQuest Online Adventures: Frontiers PS2 client, fixing engine-level bugs like VU1 shader errors, Emotion Engine memory overflows, and CDVD hardware faults so the original models load correctly.

  26. 29

    deepcrawl

    690lumpinif/deepcrawl · TypeScript

    An open-source, agent-oriented web data extraction platform — a self-deployable Firecrawl alternative that turns public pages into clean Markdown, a links tree, and LLM-friendly metadata.

  27. 30

    wcdb-key-tool

    654TANGandXUE/wcdb-key-tool · Python

    A single-purpose Python tool that pulls the encryption keys out of WeChat's local WCDB/SQLCipher4 chat databases on Linux, macOS and Windows, so you can decrypt your own chat history.

  28. 31

    wechat-ai-memory

    348ikevss/wechat-ai-memory

    A local-first Windows app that reads your WeChat 4.x chats directly, lets you filter them by conversation, date range and keyword, and exports the result as PDF, Markdown and structured JSON to build a personal, traceable chat memory archive.

  29. 32

    steam2_downloader

    229extremebleem/steam2_downloader · C#

    A desktop browser and downloader for the 'terarelease' Steam2 content leak - a dump of 10,876 depots and 116,339 files totaling 13.3 TB. It shows what the archive holds, works out which files a given depot version actually needs, downloads them, verifies them with sha256, and unpacks them.

Often tagged together