Topics / #text-processing

Text-processing projects on GitHub

Manipulating and transforming text with code and CLI tools.

As of , the most-starred Text-processing projects on GitHub are markitdown (190.7K stars), bat (60.8K stars) and pandoc (46.6K stars). GitHub.Locker tracks 40 Text-processing repositories with 751.2K stars combined. The fastest-growing on October 16 was markitdown, with +216 stars in a day.

40Projects
751.2KStars in total
+539Stars in a day
Browse in the catalog →github.com/topics/text-processing ↗

Hot right now

Most stars gained in a day · Oct 16

+216

markitdown

microsoft/markitdown · ★ 190.7K

MarkItDown is a lightweight Python utility that converts files like PDF, PowerPoint, Word, Excel, images, audio, HTML, CSV/JSON/XML, ZIP archives, EPubs and even YouTube URLs into Markdown, designed for feeding documents into LLMs and text analysis pipelines.

+115

yomiyasu

nanaism/yomiyasu · ★ 2.1K

yomiyasu is an Agent Skill that revises AI-generated Japanese text into more natural, information-dense Japanese, meant to be loaded into AI coding environments such as Codex, Claude Code, and Cursor.

+71

asd-ste100-skill

danyuchn/asd-ste100-skill · ★ 5.1K

A Claude Code skill that rewrites dense, ambiguous English into ASD-STE100 Simplified Technical English (the controlled-language standard aerospace manuals use), but aimed at text an AI agent has to parse — tool descriptions, error messages, and inter-agent messages — instead of human readers.

+20

lieflat-less-ai-tone

larashero3-dotcom/lieflat-less-ai-tone · ★ 2.8K

A research-backed skill that detects and removes 'AI tone' from Chinese writing, based on a statistical study of a 2.83-million-character corpus comparing AI-generated and human-written text.

The all-time top

40 projects · by stars

  1. #1 · microsoft/markitdown

    markitdown

    MarkItDown is a lightweight Python utility that converts files like PDF, PowerPoint, Word, Excel, images, audio, HTML, CSV/JSON/XML, ZIP archives, EPubs and even YouTube URLs into Markdown, designed for feeding documents into LLMs and text analysis pipelines.

    190.7K
  2. #2 · sharkdp/bat

    bat

    bat is a cat(1) clone that prints files in the terminal with syntax highlighting for a large number of programming and markup languages, plus Git modification markers in the sidebar.

    60.8K
  3. #3 · jgm/pandoc

    pandoc

    Pandoc is a Haskell library for converting documents from one markup format to another, plus the command-line tool built on top of it — a universal markup converter.

    46.6K
  1. 4

    tesseract.js

    38.8Knaptha/tesseract.js · JavaScript

    Tesseract.js is a pure JavaScript OCR library that extracts text from images in more than 100 languages, running the Tesseract engine compiled to WebAssembly.

  2. 5

    jq

    35.8Kjqlang/jq · C

    jq is a lightweight command-line JSON processor, akin to sed, awk and grep but for JSON data: it slices, filters, maps and transforms structured data right in your terminal.

  3. 6

    jieba

    35.2Kfxsjy/jieba · Python

    Jieba is a popular open-source Python library for Chinese word segmentation: it cuts Chinese text into words, supporting Simplified and Traditional Chinese. 'Jieba' means 'to stutter' in Chinese.

  4. 7

    spaCy

    34Kexplosion/spaCy · Python

    spaCy is a library for advanced Natural Language Processing in Python and Cython, designed from day one for real products. It ships pretrained pipelines and supports tokenization and training for 70+ languages.

  5. 8

    fmt

    25.9Kfmtlib/fmt · C++

    {fmt} is an open-source C++ formatting library — a fast, safe alternative to C stdio and C++ iostreams, with a format string syntax similar to Python's.

  6. 9

    markdown-it

    22Kmarkdown-it/markdown-it · TypeScript

    markdown-it is a fast Markdown parser for JavaScript with 100% CommonMark support. It also adds syntax extensions and sugar like URL autolinking and a typographer.

  7. 10

    tiktoken

    19.5Kopenai/tiktoken · Python

    tiktoken is a fast byte pair encoding (BPE) tokenizer built for use with OpenAI's models, converting text into tokens and back again.

  8. 11

    newspaper

    15.2Kcodelucas/newspaper · Python

    Newspaper3k is a Python 3 library that downloads, parses, and extracts the article text, authors, publish date, top image, and keywords/summary from news articles and news sites.

  9. 12

    languagetool

    15.1Klanguagetool-org/languagetool · Java

    Open-source proofreading software that checks grammar and style in English, German, French, Spanish, Polish, Dutch and more — over 25 languages — catching many errors a simple spell checker misses.

  10. 13

    parsedown

    15Kerusev/parsedown · PHP

    Parsedown is a Markdown parser library for PHP: you feed it Markdown text and it converts it into HTML.

  11. 14

    nltk

    14.7Knltk/nltk · Python

    NLTK is a suite of open source Python modules, bundled data sets, and tutorials supporting research and development in Natural Language Processing.

  12. 15

    linguist

    13.7Kgithub-linguist/linguist · Ruby

    Linguist is the library GitHub.com uses to detect the programming language of files (blobs), ignore binary or vendored files, suppress generated files in diffs, and build the language breakdown graphs shown on repository pages.

  13. 16

    fg-data-profiling

    13.7KData-Centric-AI-Community/fg-data-profiling · Python

    fg-data-profiling (formerly ydata-profiling / pandas-profiling) is a Python package that generates a one-line exploratory data analysis (EDA) report for pandas or Spark DataFrames, similar to df.describe() but far more detailed.

  14. 17

    compromise

    12.2Kspencermountain/compromise · JavaScript

    Compromise is a lightweight JavaScript natural-language processing library that parses text into structured data: tokenizing words and sentences, tagging part-of-speech, and letting you query phrases like numbers, names, or dates.

  15. 18

    presidio

    11.4Kdata-privacy-stack/presidio · Python

    An open-source framework (SDK) that detects, redacts, masks and anonymizes sensitive data (PII) — names, credit card numbers, SSNs, phone numbers — in text, images and structured data.

  16. 19

    inflector

    11.3Kdoctrine/inflector · PHP

    Doctrine Inflector is a small PHP library that performs string manipulations with regard to uppercase/lowercase and singular/plural forms of words.

  17. 20

    lexer

    11.1Kdoctrine/lexer · PHP

    Doctrine Lexer is a base PHP library for building lexers — the components that turn raw text into tokens — for top-down, recursive descent parsers.

  18. 21

    PyMuPDF

    10.9Kpymupdf/PyMuPDF · Python

    PyMuPDF is a high-performance Python library for extracting, analyzing, converting, rendering, and manipulating PDF and other document formats, built on the MuPDF C engine.

  19. 22

    natural

    10.9KNaturalNode/natural · JavaScript

    Natural is a general-purpose natural language processing library for Node.js that bundles tokenizers, stemmers, classifiers and other NLP utilities into one package.

  20. 23

    pdfplumber

    10.8Kjsvine/pdfplumber · Python

    pdfplumber is a Python library that inspects PDFs in detail, exposing each character, line, rectangle, and curve, and makes it easy to extract text and tables from them.

  21. 24

    word_cloud

    10.5Kamueller/word_cloud · Python

    word_cloud is a little word cloud generator in Python: it turns a body of text into an image where each word is sized by how often it appears. It works as a Python library and also ships a small command-line tool called wordcloudcli.

  22. 25

    nom

    10.5Krust-bakery/nom · Rust

    nom is a parser combinator library for Rust: you build safe parsers by composing small functions, without giving up speed or memory efficiency.

  23. 26

    q

    10.4Kharelba/q · Python

    q is a command-line tool that runs SQL directly on delimited text files (CSV, TSV) and on multi-file SQLite databases, treating files as tables you can query.

  24. 27

    pypdf

    10.3Kpy-pdf/pypdf · Python

    A free and open-source pure-python PDF library: it splits, merges, crops and transforms the pages of PDF files, and can add custom data, viewing options and passwords to them.

  25. 28

    Command-line-text-processing

    10.2Klearnbyexample/Command-line-text-processing · Shell

    A learning repository that teaches command-line text processing tools on Linux: grep, sed, awk, Perl one-liners, Ruby one-liners, sort, diff, and more, organized into chapters with examples and exercises.

  26. 29

    im-not-ai

    6Kepoko77-ai/im-not-ai · Python

    This is a Claude Skill that rewrites Korean text written by AI (ChatGPT, Claude, Gemini) so it reads like it was written by a human, removing translationese, mechanical parallelism and other 'AI tells' without changing the actual content.

  27. 30

    asd-ste100-skill

    5.1Kdanyuchn/asd-ste100-skill · Python

    A Claude Code skill that rewrites dense, ambiguous English into ASD-STE100 Simplified Technical English (the controlled-language standard aerospace manuals use), but aimed at text an AI agent has to parse — tool descriptions, error messages, and inter-agent messages — instead of human readers.

  28. 31

    gigatoken

    4.1Kmarcelroed/gigatoken · Rust

    Gigatoken is an extremely fast tokenizer for language models, built as a near drop-in replacement for HuggingFace Tokenizers and tiktoken, encoding text at multi-GB/s speeds.

  29. 32

    lieflat-less-ai-tone

    2.8Klarashero3-dotcom/lieflat-less-ai-tone · Python

    A research-backed skill that detects and removes 'AI tone' from Chinese writing, based on a statistical study of a 2.83-million-character corpus comparing AI-generated and human-written text.

  30. 33

    yomiyasu

    2.1Knanaism/yomiyasu · Python

    yomiyasu is an Agent Skill that revises AI-generated Japanese text into more natural, information-dense Japanese, meant to be loaded into AI coding environments such as Codex, Claude Code, and Cursor.

  31. 34

    natural-japanese

    1.9Kcoji/natural-japanese · Python

    An Agent Skill for writing and revising clear, natural Japanese for work documents — meeting minutes, reports, internal guides, proposals, blog posts. Its whole point is stripping the telltale 'AI smell' from machine-written prose.

  32. 35

    marisa-trie

    1.1Kpytries/marisa-trie · Cython

    Static, memory-efficient trie-like data structures for Python, built on the marisa-trie C++ library. Strings stored inside can take up to 50x-100x less memory than in a standard Python dict, with comparable lookup speed and fast prefix search.

  33. 36

    Evidence-Bound-Press-Conference-Revision-Skill

    270lensback940701/Evidence-Bound-Press-Conference-Revision-Skill · Python

    It is an evidence-bound revision skill for academic manuscripts in social science research. It strips out defensive, self-protective prose (hedging, work-log narration, anticipatory rebuttals) while keeping every scope condition, evidence-status distinction, and rival explanation that the evidence actually supports.

  34. 37

    iso-24495-skill

    154danyuchn/iso-24495-skill

    A Claude Code skill that rewrites your text into plain language following ISO 24495-1:2023 — the standard that defines plain language by reader outcome: readers can find what they need, understand it on first reading, and use it. It works on English and Traditional Chinese.

  35. 38

    claude-watermark-cleaner

    151mikiane/claude-watermark-cleaner · Python

    An experimental Python CLI that tries to strip Claude's statistical text watermark: it removes suspicious invisible Unicode characters, then has a non-Claude model rewrite the text. It doubles as a critical demo arguing such watermarks are easy to circumvent.

  36. 39

    paper-docx

    122paper-instruments/paper-docx · Python

    paper-docx is an agent-safe fork of python-docx for creating and editing Microsoft Word (.docx) documents, built to refuse unsupported operations instead of silently corrupting files.

  37. 40

    roster-match

    104charlenemercury/roster-match · JavaScript

    A zero-dependency JavaScript library that matches a Chinese name heard through speech recognition (or any garbled transcript) to the correct entry in a roster, comparing by pinyin syllables and surfacing candidates with reasons instead of forcing a decision when it is unsure.

Often tagged together