Topics / #multimodal-ai

Multimodal AI projects on GitHub

AI models that process multiple input types like text, images, and audio.

As of , the most-starred Multimodal AI projects on GitHub are Open-AutoGLM (26.4K stars), Awesome-Multimodal-Large-Language-Models (18.1K stars) and pipecat (16.6K stars). GitHub.Locker tracks 9 Multimodal AI repositories with 83.9K stars combined. The fastest-growing on October 16 was pipecat, with +25 stars in a day.

9Projects
83.9KStars in total
+85Stars in a day
Browse in the catalog →github.com/topics/multimodal-ai ↗

Hot right now

Most stars gained in a day · Oct 16

+25

pipecat

pipecat-ai/pipecat · ★ 16.6K

Pipecat is an open-source Python framework for building real-time voice and multimodal conversational agents — from a single voice bot to a distributed multi-agent system — maintained by Daily and the community.

+22

Duix-Avatar

duixcom/Duix-Avatar · ★ 16K

Duix.Avatar is a free, open-source AI digital human toolkit that clones your face and voice to generate fully offline avatar videos driven by text or speech.

+16

Valen

Liuziyu77/Valen · ★ 861

Valen (万澜) is a multimodal decision model that adds visual perception to System One-style decision-making: it scores candidate actions against text, images and video using a task instruction, instead of generating free-form answers.

+9

Open-AutoGLM

zai-org/Open-AutoGLM · ★ 26.4K

Open-AutoGLM (Phone Agent) is an open phone-use agent framework built on the AutoGLM model that understands your phone screen visually and automates tasks via natural-language commands like 'open Xiaohongshu and search for food'.

The all-time top

9 projects · by stars

  1. #1 · zai-org/Open-AutoGLM

    Open-AutoGLM

    Open-AutoGLM (Phone Agent) is an open phone-use agent framework built on the AutoGLM model that understands your phone screen visually and automates tasks via natural-language commands like 'open Xiaohongshu and search for food'.

    26.4K
  2. #2 · BradyFU/Awesome-Multimodal-Large-Language-Models

    Awesome-Multimodal-Large-Language-Models

    A curated, continuously updated list of papers, models, datasets, and benchmarks tracking the latest advances in Multimodal Large Language Models (MLLMs) — models like GPT-4o, Gemini, and VITA that understand images, video, and audio alongside text.

    18.1K
  3. #3 · pipecat-ai/pipecat

    pipecat

    Pipecat is an open-source Python framework for building real-time voice and multimodal conversational agents — from a single voice bot to a distributed multi-agent system — maintained by Daily and the community.

    16.6K
  1. 4

    Duix-Avatar

    16Kduixcom/Duix-Avatar · C

    Duix.Avatar is a free, open-source AI digital human toolkit that clones your face and voice to generate fully offline avatar videos driven by text or speech.

  2. 5

    AgnesAI-Models

    5.2KAgnesAI-Labs/AgnesAI-Models

    Agnes AI is the official gateway and model catalog for Agnes AI, giving developers OpenAI-compatible access to multimodal models for text, image, video, and agent workflows through one unified API.

  3. 6

    Valen

    861Liuziyu77/Valen · Jupyter Notebook

    Valen (万澜) is a multimodal decision model that adds visual perception to System One-style decision-making: it scores candidate actions against text, images and video using a task instruction, instead of generating free-form answers.

  4. 7

    LongCat-Next

    477meituan-longcat/LongCat-Next · Python

    LongCat-Next is a native multimodal foundation model from Meituan that processes text, vision, and audio under a single autoregressive objective using discrete tokens, built on the LongCat-Flash-Lite MoE backbone (A3B).

  5. 8

    damo-radar

    206alibaba-damo-academy/damo-radar · Python

    RADAR is a generalist vision-language AI model for abdominal CT diagnosis, trained on over 400,000 contrast-enhanced abdominal CT exams with 15 million anatomy-aware image-text pairs derived directly from clinical reports.

  6. 9

    S-Space

    174MirroS-Lab/S-Space · Python

    S-Space is a research project that finds a three-dimensional linear subspace inside multimodal model activations which encodes object positions along horizontal, vertical, and distance axes, and shows it can be read out and even causally edited to change the model's spatial judgments.

Often tagged together