Topics / #batch-processing

Batch-processing projects on GitHub

Processing large datasets in scheduled batches rather than streaming.

As of , the most-starred Batch-processing projects on GitHub are pathway (62.1K stars), Umi-OCR (48K stars) and EasySpider (44.7K stars). GitHub.Locker tracks 15 Batch-processing repositories with 335.4K stars combined. The fastest-growing on October 16 was Filee, with +297 stars in a day.

15Projects
335.4KStars in total
+386Stars in a day
Browse in the catalog →github.com/topics/batch-processing ↗

Hot right now

Most stars gained in a day · Oct 16

+297

Filee

KnifeLemon/Filee · ★ 1.3K

Filee is a free, open-source file converter for Windows, macOS and Linux that handles 180+ formats — images, PDF, Word, Excel, PowerPoint, HWP/HWPX, video, audio, e-books, CAD and archives — by letting you drag files onto a floating format menu, all processed locally with no upload.

+22

Umi-OCR

hiroi-sora/Umi-OCR · ★ 48K

Umi-OCR is a free, open-source, offline OCR tool that extracts text from screenshots, batches of images, and PDF documents, and can also scan or generate QR codes.

+19

marker

datalab-to/marker · ★ 40.5K

Marker is a document conversion tool that turns PDFs (and other document types) into markdown, JSON, HTML, or chunked output quickly and accurately, extracting tables, equations, forms, and images along the way.

+19

pyvideotrans

jianchang512/pyvideotrans · ★ 19.4K

An open-source video translation and dubbing tool: it converts a video from one language to another through a one-click pipeline of speech recognition, subtitle translation, TTS dubbing, and video synthesis.

The all-time top

15 projects · by stars

  1. #1 · pathwaycom/pathway

    pathway

    Pathway is a Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG, with the same code usable for batch jobs, CI/CD tests, and live data streams.

    62.1K
  2. #2 · hiroi-sora/Umi-OCR

    Umi-OCR

    Umi-OCR is a free, open-source, offline OCR tool that extracts text from screenshots, batches of images, and PDF documents, and can also scan or generate QR codes.

    48K
  3. #3 · NaiboWang/EasySpider

    EasySpider

    EasySpider (易采集) is a visual, no-code web crawler and browser automation tool. You select elements on a webpage with clicks instead of writing code, and it designs and runs the scraping/testing task for you.

    44.7K
  1. 4

    marker

    40.5Kdatalab-to/marker · Python

    Marker is a document conversion tool that turns PDFs (and other document types) into markdown, JSON, HTML, or chunked output quickly and accurately, extracting tables, equations, forms, and images along the way.

  2. 5

    flink

    26.4Kapache/flink · Java

    Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities, running your programs on its own distributed runtime.

  3. 6

    ddia

    23.8KVonng/ddia · Python

    A free Chinese translation of Martin Kleppmann's 'Designing Data-Intensive Applications' (DDIA), covering both the 1st and the complete 2nd edition; 210 contributors have polished it since 2018.

  4. 7

    pyvideotrans

    19.4Kjianchang512/pyvideotrans · Python

    An open-source video translation and dubbing tool: it converts a video from one language to another through a one-click pipeline of speech recognition, subtitle translation, TTS dubbing, and video synthesis.

  5. 8

    luigi

    18.8Kspotify/luigi · Python

    Luigi is a Python package for building complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, failure handling and command line integration.

  6. 9

    ImageMagick

    17.7KImageMagick/ImageMagick · C

    ImageMagick is a free, open-source suite for editing and manipulating digital images: it creates, edits, composes, and converts bitmap images. It is widely used in web development, graphic design, video editing, scientific research, and medical imaging.

  7. 10

    argo-workflows

    17Kargoproj/argo-workflows · Go

    An open source, container-native workflow engine for Kubernetes: it orchestrates parallel jobs where every step of your workflow runs as a container. It is a CNCF graduated project.

  8. 11

    manga-image-translator

    10.5Kzyddnys/manga-image-translator · Python

    An end-to-end manga and image text translator: it detects text, reads it with OCR, machine-translates it, erases the original lettering and renders the translation back into the picture.

  9. 12

    moyin-creator

    4.7KMemeCalculate/moyin-creator · TypeScript

    Moyin Creator (魔因漫创) is a production-grade AI tool for film and video creators that automates the whole pipeline from script to finished video: script parsing, character design, scene generation, storyboarding, and Seedance 2.0-powered image/video batch generation.

  10. 13

    Filee

    1.3KKnifeLemon/Filee · C#

    Filee is a free, open-source file converter for Windows, macOS and Linux that handles 180+ formats — images, PDF, Word, Excel, PowerPoint, HWP/HWPX, video, audio, e-books, CAD and archives — by letting you drag files onto a floating format menu, all processed locally with no upload.

  11. 14

    palette-from-image-ai

    300egotoadcitadel/palette-from-image-ai · Python

    It's a standalone Windows desktop app that pulls a clean, role-tagged color palette out of any image using an on-device vision model and perceptual color clustering, with no cloud upload or subscription involved.

  12. 15

    dsh-lowtide

    174KelaoHu/dsh-lowtide · TypeScript

    lowtide is a plugin for DeepSeek Harness (dsh) that lets you queue agent tasks during the day and have them run automatically, unattended, during DeepSeek's off-peak pricing hours, so you review a finished report later instead of waiting on the agent in real time.

Often tagged together