Multimodal projects on GitHub

1. 1
   #1 · huggingface/transformers

   transformers

   Transformers is Hugging Face's model-definition framework for state-of-the-art machine learning models across text, vision, audio, video, and multimodal tasks, usable for both inference and training.

   167.8K
2. 2
   #2 · vllm-project/vllm

   vllm

   vLLM is a library for fast LLM inference and serving, built around PagedAttention for efficient memory management of attention keys and values.

   94.1K
3. 3
   #3 · infiniflow/ragflow

   ragflow

   RAGFlow is an open-source Retrieval-Augmented Generation (RAG) engine that combines RAG with Agent capabilities to build a context layer for LLMs, turning complex enterprise data into production-ready AI systems.

   92.4K

4. 4

   PaddleOCR

   91.4KPaddlePaddle/PaddleOCR · Python

   PaddleOCR is an OCR and document AI toolkit that converts PDFs and images into structured, LLM-ready data (JSON or Markdown), supporting over 100 languages.

   • pdf-parser
   • document-parsing
   • table
   • vlm
5. 5

   unsloth

   78.2Kunslothai/unsloth · Python

   Unsloth is a local-first app for running and training LLMs, diffusion, TTS and embedding models on your own machine, including models like Qwen3.8, DeepSeek-V4, MiniMax-H3, Gemma 4 and FLUX.

   • dpo
   • qlora
   • intel
   • kaggle
6. 6

   anything-llm

   67.3KMintplex-Labs/anything-llm · JavaScript

   AnythingLLM is an all-in-one, self-hosted AI application that lets you chat with your documents, run AI agents, and build a private ChatGPT-like experience using local or cloud LLMs.

   • localai
   • self-hosted-ai
   • agentic-workflow
   • hermes-agent
7. 7

   dive-into-llms

   56.9KLordog/dive-into-llms · Jupyter Notebook

   It's a free Chinese-language tutorial series called 'Dive into LLMs' (《动手学大模型》) that teaches large language model concepts through hands-on coding exercises, built from Shanghai Jiao Tong University's NLP and AI-security course materials.

   • rlhf
   • jailbreak
   • ai-security
   • reasoning
8. 8

   ai-agent-book

   55Kbojieli/ai-agent-book · Python

   An open-source book by Li Bojie taking AI agents from design principles to engineering practice, built on the formula Agent = LLM + context + tools; text, figures and labs all live in this repo.

   • experiments
   • epub
   • computer-use
   • llm-agents
9. 9

   claude-cookbooks

   53.6Kanthropics/claude-cookbooks · Jupyter Notebook

   Anthropic's collection of notebooks and recipes showing fun and effective ways of using Claude, with copy-able code snippets you can drop straight into your own projects.

   • cookbook
   • claude-api
   • notebooks
   • snippets
10. 10

    BettaFish

    42.5K666ghj/BettaFish · Python

    BettaFish (Chinese name 'Weiyu'/微舆) is a from-scratch multi-agent public opinion analysis system that monitors social media and large volumes of public comments, then produces reports to help break information bubbles and support decision-making.

    • gpt-2
    • weibo
    • bert
    • sentiment-analysis
11. 11

    LightRAG

    40.2KHKUDS/LightRAG · Python

    LightRAG is an open-source, graph-based RAG framework (EMNLP 2025) by HKUDS. It indexes your documents into a knowledge graph plus vector embeddings so you can query them — a lighter, faster alternative to Microsoft GraphRAG.

    • question-answering
    • graphrag
    • genai
    • graph-database
12. 12

    UI-TARS-desktop

    39.3Kbytedance/UI-TARS-desktop · TypeScript

    TARS is an open-source multimodal AI agent stack from ByteDance that ships two projects: Agent TARS, a CLI/Web UI agent for terminal, browser and tool-use tasks, and UI-TARS Desktop, a native app that lets a vision-language model see your screen and control mouse/keyboard.

    • gui-agent
    • browser-use
    • vision
    • vlm
13. 13

    Langchain-Chatchat

    38.7Kchatchat-space/Langchain-Chatchat · Python

    Langchain-Chatchat is an open-source RAG and Agent application that lets you build a local-knowledge-base Q&A chatbot using LLMs like ChatGLM, Qwen or Llama through the LangChain framework.

    • milvus
    • chatglm
    • faiss
    • embedding
14. 14

    CLIP

    34.5Kopenai/CLIP · Jupyter Notebook

    CLIP is a neural network trained on many (image, text) pairs: given an image, it predicts the most relevant text snippet in natural language, zero-shot, without directly optimizing for the task.

    • vision-language
    • vision-language-model
    • image-classification
    • pretrained-models
15. 15

    paper-reading

    34Kmli/paper-reading

    It's a curated reading list for deep learning: each entry is a recorded lecture that walks through a paper paragraph by paragraph, from classics like CLIP and GPT-3 to recent ones like Llama 3.1 and Sora.

    • llama
    • whisper
    • paper
    • gpt
16. 16

    TaskMatrix

    33.9Kchenfei-wu/TaskMatrix · Python

    TaskMatrix (also known as Visual ChatGPT) connects ChatGPT to a set of Visual Foundation Models so you can send and receive images while chatting, letting the LLM draw, edit, and analyze pictures.

    • vision-language
    • controlnet
    • inpainting
    • segmentation
17. 17

    fish-speech

    33.3Kfishaudio/fish-speech · Python

    Fish Speech (Fish Audio S2) is a state-of-the-art open-source text-to-speech system that generates natural, emotionally rich speech in 80+ languages, led by the flagship S2-Pro model that combines a Dual-AR architecture with reinforcement-learning alignment.

    • vits
    • speech-synthesis
    • voice-cloning
    • llama
18. 18

    self-llm

    32.5Kdatawhalechina/self-llm · Jupyter Notebook

    Self-llm is a Chinese-language tutorial collection that teaches beginners how to configure environments, deploy, and fine-tune (full-parameter or LoRA) 50+ open-source large language models and multimodal models on Linux.

    • deepseek-r1
    • lmstudio
    • chatglm
    • llama3
19. 19

    openai-python

    31.9Kopenai/openai-python · Python

    This is the official Python library for the OpenAI API, giving convenient programmatic access to OpenAI models like GPT for text, vision, audio and more.

    • async
    • gpt
    • oauth
    • websocket
20. 20

    smolagents

    29.9Khuggingface/smolagents · Python

    smolagents is a lightweight Python library for building AI agents that write their actions as Python code instead of JSON tool calls, letting you run powerful agents in just a few lines of code.

    • llm-agents
    • langchain
    • transformers
    • autonomous-agents
21. 21

    Hands-On-Large-Language-Models

    29.8KHandsOnLLM/Hands-On-Large-Language-Models · Jupyter Notebook

    The official code repo for the O'Reilly book 'Hands-On Large Language Models', known as the Illustrated LLM Book. It contains a Jupyter notebook for each of the book's 12 chapters.

    • text-classification
    • classification
    • llms
    • semantic-search
22. 22

    semantic-kernel

    28.7Kmicrosoft/semantic-kernel · C#

    Semantic Kernel is a model-agnostic SDK from Microsoft for building, orchestrating, and deploying AI agents and multi-agent systems, from a simple chatbot to complex multi-agent workflows.

    • multi-agent-systems
    • microsoft
    • vector-database
    • agent-framework
23. 23

    Strata

    27.7KNiko1221/Strata · C++

    Strata runs Qwen3.8-Flash-Next, a 125-billion-parameter AI model, on a normal gaming PC with one NVIDIA GPU (12-24GB VRAM) and 64GB of RAM, with a one-click installer for Windows and Linux.

    • llama-cpp
    • speculative-decoding
    • inference-engine
    • mixture-of-experts
24. 24

    MiniCPM-V

    26.9KOpenBMB/MiniCPM-V · Python

    A family of pocket-sized multimodal AI models from OpenBMB (MiniCPM-V and MiniCPM-o) built to run on local devices, even phones — understanding images, video and text, and in the o-series also speech.

    • harmonyos
    • video-understanding
    • llama-cpp
    • on-device
25. 25

    Open-AutoGLM

    26.4Kzai-org/Open-AutoGLM · Python

    Open-AutoGLM (Phone Agent) is an open phone-use agent framework built on the AutoGLM model that understands your phone screen visually and automates tasks via natural-language commands like 'open Xiaohongshu and search for food'.

    • multimodal-ai
    • mobile-automation
    • adb
    • vision-language-model
26. 26

    MiniGPT-4

    25.6KVision-CAIR/MiniGPT-4 · Python

    This is the open-sourced research code for MiniGPT-4 and MiniGPT-v2, two vision-language models that use a large language model as the interface for understanding and chatting about images.

    • vision-language
    • demo
    • model
    • vlm
27. 27

    OmniParser

    25.5Kmicrosoft/OmniParser · Jupyter Notebook

    OmniParser parses GUI screenshots into structured, labeled elements (icons, text, interactable regions) so a vision model like GPT-4V can accurately decide what to click or type on a screen.

    • gui-agent
    • vision
    • gpt-4
    • vision-language-model
28. 28

    JARVIS

    25.5Kmicrosoft/JARVIS · Python

    JARVIS is an open-source research system that connects an LLM with the ML community: a large language model acts as the controller, picking and invoking expert models from the Hugging Face Hub to solve complex AI tasks.

    • platform
    • text-to-video
    • chatgpt
    • orchestration
29. 29

    LLaVA

    25.1Khaotian-liu/LLaVA · Python

    LLaVA is an open-source large multimodal chatbot: a vision-language model you can show images to and talk with, built by visual instruction tuning toward GPT-4V-level capabilities (NeurIPS 2023 Oral).

    • llama2
    • instruction-tuning
    • gpt-4
    • foundation-models
30. 30

    gallery

    25Kgoogle-ai-edge/gallery · Kotlin

    Google AI Edge Gallery is an app that lets you download and run open-source generative AI models, including the Gemma family, directly on your Android phone, iPhone, or Mac — fully offline.

    • function-calling
    • on-device-ai
    • local-ai
    • speech-to-text
31. 31

    DeepSeek-OCR

    24Kdeepseek-ai/DeepSeek-OCR · Python

    DeepSeek-OCR is a vision-language model built to explore 'contexts optical compression' -- squeezing text-heavy document images into a small number of vision tokens that an LLM can still read back accurately, as a research probe into how much text a vision encoder can represent visually.

    • vision-language-model
    • natural-language-processing
    • transformers
    • vllm
32. 32

    ds4

    24Kantirez/ds4 · C

    DwarfStar (ds4) is a small, self-contained local inference engine specialized for running DeepSeek V4 Flash and PRO, plus GLM 5.2/5.3, on consumer hardware like high-memory Macs, NVIDIA GPUs, and AMD Strix Halo systems.

    • kv-cache
    • llm-serving
    • rocm
    • glm
33. 33

    verl

    23.9Kverl-project/verl · Python

    verl is a flexible, efficient and production-ready reinforcement learning training library for LLMs, initiated by ByteDance's Seed team and maintained by the community. It is the open-source version of the HybridFlow paper (EuroSys 2025).

    • grpo
    • rlhf
    • vlm
    • distributed-training
34. 34

    Awesome-Nano-Banana-images

    23.9KPicoTrex/Awesome-Nano-Banana-images

    A curated gallery of fun, creative images generated with Nano Banana and Nano Banana Pro — Google's Gemini-2.5-flash-image based models — each paired with the prompt that produced it.

    • nano-banana
    • gallery
    • xiaohongshu
    • twitter
35. 35

    RAG-Anything

    23.9KHKUDS/RAG-Anything · Python

    An all-in-one multimodal RAG framework built on LightRAG: it ingests real-world documents that mix text, images, tables and equations, then answers queries across all content types through one pipeline instead of separate specialized tools.

    • rag-pipeline
    • vlm
    • retrieval-augmented-generation
    • vector-search
36. 36

    MaxKB

    23K1Panel-dev/MaxKB · Python

    MaxKB (Max Knowledge Brain) is an open-source platform for building enterprise-grade AI agents, combining RAG pipelines, an agentic workflow engine and MCP tool-use. Typical uses: intelligent customer service, internal company knowledge bases, research and education.

    • deepseek-r1
    • customer-support
    • llama3
    • pgvector
37. 37

    courses

    22.9Kanthropics/courses · Jupyter Notebook

    Anthropic's official set of educational courses on working with Claude: five hands-on courses covering API essentials, prompt engineering, real world prompting, prompt evaluations, and tool use.

    • claude-api
    • anthropic-api
    • function-calling
    • tool-calling
38. 38

    unilm

    22.2Kmicrosoft/unilm · Python

    Microsoft's research hub for large-scale self-supervised pre-training: a family of foundation models spanning NLP, vision, speech, and multimodal tasks across 100+ languages - including BEiT, LayoutLM, Kosmos, MiniLM, TrOCR, and WavLM.

    • self-supervised-learning
    • foundation-models
    • natural-language-processing
    • text-to-speech
39. 39

    datasets

    22.1Khuggingface/datasets · Python

    A lightweight Python library from Hugging Face that gives one-line dataloaders for a huge catalog of ready-to-use public datasets on the HF Hub, plus fast, reproducible preprocessing for Hub and local data.

    • medical-imaging
    • polars
    • jax
    • spark
40. 40

    screenpipe

    22Kscreenpipe/screenpipe · Rust

    screenpipe continuously records your screen and audio locally, turning everything you do into a searchable, AI-powered memory that gives agents like Claude or Codex context about your work. It is positioned as a source-available alternative to Rewind, Microsoft Recall and Granola.

    • meeting-notes
    • agi
    • ai-memory
    • screen-recording
41. 41

    serve

    21.9Kjina-ai/serve · Python

    Jina is an open-source Python framework for building multimodal AI applications on a cloud-native stack — you wrap models into gRPC microservices, chain them into flows, and scale from local to cloud.

    • neural-search
    • ai-framework
    • microservice
    • pipeline
42. 42

    surya

    21.5Kdatalab-to/surya · Python

    Surya is Datalab's open-source document OCR system built around a 650M parameter model. It reads text from images and PDFs, and also does layout analysis, reading order, and table recognition.

    • document-intelligence
    • table-extraction
    • table
    • layout
43. 43

    owl

    20.2Kcamel-ai/owl · Python

    OWL (Optimized Workforce Learning) is an open-source framework for multi-agent collaboration that automates real-world tasks. Built on top of the CAMEL-AI framework, it ranks #1 among open-source frameworks on the GAIA benchmark with a 69.09 average score.

    • task-automation
    • multi-agent-systems
    • webui
    • playwright
44. 44

    Qwen3-VL

    20.1KQwenLM/Qwen3-VL · Jupyter Notebook

    Qwen3-VL is a family of multimodal vision-language models built by the Qwen team at Alibaba Cloud, combining strong text understanding with deep visual perception, reasoning, and agent capabilities.

    • video-understanding
    • alibaba
    • vision-language-model
    • reasoning
45. 45

    olmocr

    19.8Kallenai/olmocr · Python

    olmOCR is a toolkit that converts PDFs, PNGs, and JPEGs into clean Markdown or plain text with natural reading order, preserving equations, tables, and handwriting - built primarily to produce training datasets for LLMs.

    • s3
    • training
    • aws
    • dataset
46. 46

    ml-engineering

    19.6Kstas00/ml-engineering · Python

    An open book on machine learning engineering: methodologies, tools and step-by-step instructions for training, fine-tuning and running inference on large language models and multi-modal (VLM) models, written by the engineer who led the BLOOM-176B training.

    • hpc
    • courses
    • llm-training
    • vlm
47. 47

    deepeval

    19Kconfident-ai/deepeval · Python

    DeepEval is an open-source LLM evaluation framework: a simple-to-use way to evaluate large-language model systems with research-backed metrics like G-Eval, answer relevancy, and hallucination checks.

    • evaluation-framework
    • llmops
    • llm-evaluation
    • unit-testing
48. 48

    Speech

    18.6KNVIDIA-NeMo/Speech · Python

    An open-source NVIDIA framework for building speech AI models — automatic speech recognition (ASR), text-to-speech (TTS), and speech LLMs — with pre-trained checkpoints you can customize and deploy.

    • speaker-recognition
    • diarization
    • machine-translation
    • deeplearning
49. 49

    llama-cookbook

    18.6Kmeta-llama/llama-cookbook · Jupyter Notebook

    The Llama Cookbook — Meta's official collection of recipes for building with the Llama model family: inference, fine-tuning, RAG and end-to-end use cases, mostly written as Jupyter notebooks.

    • finetuning
    • llama2
    • long-context
    • vision
50. 50

    WeClone

    18.3Kxming521/WeClone · Python

    WeClone is an open-source toolkit that turns your chat history into an AI twin: it fine-tunes an LLM on your own messages so a chatbot can reply in your personal style.

    • ai-companion
    • slack
    • lora
    • discord
51. 51

    aie-book

    17.9Kchiphuyen/aie-book · Jupyter Notebook

    The companion repo for Chip Huyen's book 'AI Engineering' (2025) and a collection of resources for AI engineers: chapter summaries, study notes, prompt examples, case studies. It's a work in progress and keeps growing.

    • hallucination-detection
    • study-guide
    • evaluation
    • jupyter notebook
52. 52

    cookbook

    17.9Kgoogle-gemini/cookbook · Jupyter Notebook

    Google's official cookbook for the Gemini API: a structured learning path of hands-on tutorials, practical examples, and links to end-to-end demo apps, published as an open-source notebook collection.

    • gemini-api
    • music-generation
    • video-editing
    • image-editing
53. 53

    generative-ai

    17.8KGoogleCloudPlatform/generative-ai · Jupyter Notebook

    A collection of notebooks, code samples and sample apps for building generative AI on Google Cloud with Gemini and its Agent Platform, the evolution of Vertex AI.

    • vertex-ai
    • samples
    • gemini-api
    • google-cloud
54. 54

    Janus

    17.8Kdeepseek-ai/Janus · Python

    Janus is DeepSeek's family of open multimodal AI models (Janus, Janus-Pro, JanusFlow). One model both understands images and text and generates images from a text prompt.

    • open-weights
    • foundation-models
    • vision-language-model
    • gradio
55. 55

    pipecat

    16.6Kpipecat-ai/pipecat · Python

    Pipecat is an open-source Python framework for building real-time voice and multimodal conversational agents — from a single voice bot to a distributed multi-agent system — maintained by Daily and the community.

    • chatbot-framework
    • chatbots
    • multimodal-ai
    • ai-companion
56. 56

    memvid

    16.6Kmemvid/memvid · Rust

    Memvid is a single-file memory layer for AI agents: it packs your data, embeddings and search index into one portable .mv2 file, so agents get instant retrieval and long-term memory without a RAG pipeline or a vector database server.

    • context
    • faiss
    • ai-memory
    • opencv
57. 57

    ms-swift

    15.9Kmodelscope/ms-swift · Python

    ms-swift is the ModelScope community's fine-tuning and deployment framework for LLMs and multimodal models, covering training, inference, evaluation, quantization and deployment in one pipeline.

    • peft
    • embedding
    • sft
    • grpo
58. 58

    midscene

    15.3Kweb-infra-dev/midscene · TypeScript

    Midscene is a vision-driven GUI Agent combined with an E2E testing kit: it looks at screenshots to act on and verify UI across web, mobile, and desktop apps through one set of Agent APIs.

    • gui-agent
    • computer-use-agent
    • mobile-automation
    • qa
59. 59

    ai-goofish-monitor

    15KUsagi-org/ai-goofish-monitor · Python

    A self-hosted monitoring system for Xianyu (闲鱼, China's largest secondhand marketplace) that scrapes listings with Playwright and uses AI to analyze and filter products in real time or on a schedule, with a full web management UI.

    • webhook
    • ecommerce
    • ai-assistant
    • fastapi
60. 60

    agents

    14.8Klivekit/agents · Python

    LiveKit's open-source Python framework for building realtime voice AI agents: programmable participants that run on servers, can see, hear, and understand, and talk to users over WebRTC.

    • voice-agent
    • voice-ai
    • webrtc
    • voice
61. 61

    open_clip

    14.2Kmlfoundations/open_clip · Python

    OpenCLIP is an open source implementation of OpenAI's CLIP model for training and using contrastive image-text (and now audio-text) models for tasks like zero-shot classification.

    • language-model
    • image
    • pretrained-models
    • fine-tuning
62. 62

    TRELLIS

    14Kmicrosoft/TRELLIS · Python

    Microsoft's large 3D asset generation model: feed it a text or image prompt and it generates high-quality 3D assets as Radiance Fields, 3D Gaussians, or meshes. It is the official code for a CVPR'25 Spotlight paper.

    • text-to-3d
    • gaussian-splatting
    • image-to-3d
    • 3d-generation
63. 63

    skills

    13.7KMiniMax-AI/skills · C#

    MiniMax Skills is a collection of plug-in 'skill' packages that give AI coding agents (Claude Code, Cursor, Codex, OpenCode) structured, production-quality guidance for frontend, fullstack, Android, iOS, Flutter, React Native, shader work, and MiniMax-API-powered media generation (images, video, music, TTS, documents).

    • music-generation
    • pptx
    • shaders
    • voice-cloning
64. 64

    AI-Research-SKILLs

    13.7KOrchestra-Research/AI-Research-SKILLs · TeX

    It's an open-source library of 98 AI research and engineering skills that turn a Claude Code, Codex, or Gemini CLI agent into a full AI research agent, covering everything from idea generation to paper writing.

    • gpt-5
    • paper-writing
    • grpo
    • ai-research
65. 65

    DiffSynth-Studio

    13.3Kmodelscope/DiffSynth-Studio · Python

    DiffSynth-Studio is an open-source Diffusion model engine built and maintained by the ModelScope Community, covering image, video, and audio generation plus image quality metrics, with support for both inference and training.

    • music-generation
    • model-training
    • diffusion
    • comfyui
66. 66

    CogVideo

    13.1Kzai-org/CogVideo · Python

    An open-source family of text-to-video and image-to-video models: CogVideoX (2024) and its CogVideoX1.5 upgrade, following CogVideo (ICLR 2023), the first open-source large Transformer-based text-to-video model.

    • image-to-video
    • text-to-video
    • diffusion-models
    • lora
67. 67

    Agent-S

    12.7Ksimular-ai/Agent-S · Python

    Agent S is an open-source agent framework that controls a real desktop GUI - clicking, typing, and scrolling - to complete natural-language tasks the way a human would, without per-app API integrations.

    • cua
    • planning
    • computer-use-agent
    • retrieval-augmented-generation
68. 68

    axolotl

    12.6Kaxolotl-ai-cloud/axolotl · Python

    Axolotl is a free, open-source framework that streamlines post-training and fine-tuning of large language models — you describe your run in one YAML config and it drives the whole pipeline.

    • distributed-training
    • moe
    • lora
    • training
69. 69

    all-in-rag

    12.3Kdatawhalechina/all-in-rag · Python

    This is a full-stack tutorial on Retrieval-Augmented Generation (RAG) for developers building LLM applications, covering theory and hands-on practice from basic concepts to production-ready RAG systems.

    • llama-index
    • milvus
    • embedding
    • neo4j
70. 70

    Z-Image

    12.1KTongyi-MAI/Z-Image · Python

    Z-Image is a family of efficient image generation models with 6B parameters, built around a single-stream diffusion transformer architecture, developed by Alibaba's Tongyi team.

    • img2img
    • alibaba
    • bilingual
    • foundation-models
71. 71

    cosmos

    12.1KNVIDIA/cosmos · Jupyter Notebook

    NVIDIA Cosmos is an open platform of world models, datasets, and tools for building Physical AI — systems that understand and generate the physical world for robots, autonomous vehicles, and smart infrastructure.

    • autonomous-vehicles
    • world-models
    • datasets
    • diffusion-models
72. 72

    h2ogpt

    11.9Kh2oai/h2ogpt · Python

    h2oGPT is an open-source private chat application that lets you query and summarize your own documents or just chat with local LLMs, with no data leaving your machine.

    • generative
    • llama2
    • llamacpp
    • stable-diffusion
73. 73

    scira

    11.9Kzaidmukaddam/scira · TypeScript

    Scira (formerly MiniPerplx) is an AI-powered search engine that plans a research strategy, retrieves information from the web and many other sources, and answers questions with clickable inline citations.

    • search-engine
    • vercel
    • tailwindcss
    • nextjs
74. 74

    lancedb

    11.7Klancedb/lancedb · Rust

    LanceDB is an open-source vector database for multimodal AI: a central platform where you store, index and search petabytes of vectors and multimodal data, built for fast, production-ready retrieval.

    • image-search
    • recommender-system
    • llamaindex
    • nearest-neighbor-search
75. 75

    UI-TARS

    11.6Kbytedance/UI-TARS · Python

    UI-TARS is an open-source multimodal agent model built on a vision-language model that can operate GUIs directly — clicking, typing, scrolling, dragging — across desktop apps, mobile screens, browsers, and even video games like Minecraft, reasoning about its actions before taking them.

    • gui-agent
    • minecraft
    • vlm
    • computer-use
76. 76

    rerun

    11.6Krerun-io/rerun · Rust

    Rerun is a data layer for physical AI: it logs, queries, visualizes and streams multimodal robotics data (images, point clouds, transforms, time series, video) so you can debug sensor pipelines and feed data into training.

    • time-series
    • robotics
    • training
    • visualization
77. 77

    DALLE2-pytorch

    11.3Klucidrains/DALLE2-pytorch · Python

    It is a PyTorch implementation of DALL-E 2, OpenAI's text-to-image synthesis model, built by a third-party community author (lucidrains).

    • text-to-image
    • diffusion-models
    • embeddings
    • neural-network
78. 78

    LAVIS

    11.3Ksalesforce/LAVIS · Jupyter Notebook

    LAVIS is a Python deep learning library for language-and-vision research, offering a unified interface to 10+ multimodal tasks, 20+ datasets, and 30+ pretrained foundation models like BLIP, BLIP-2, ALBEF, and CLIP.

    • vision-language
    • model
    • jupyter-notebook
    • transformers
79. 79

    LTX-Video

    11.1KLightricks/LTX-Video · Python

    LTX-Video is the official repo for Lightricks' DiT-based video generation model, which generates synchronized audio+video, supports text-to-video and image-to-video, keyframe conditioning, video extension, and video-to-video transformations.

    • foundation-models
    • image-to-video
    • comfyui
    • text-to-video
80. 80

    mistral-inference

    10.8Kmistralai/mistral-inference · Jupyter Notebook

    The official inference code for Mistral AI's open-weight models: minimal Python and CLI tooling to run models like Mixtral, Codestral, Pixtral and Mistral Large locally.

    • mistral
    • coding-assistant
    • function-calling
    • open-weights
81. 81

    autogluon

    10.8Kautogluon/autogluon · Python

    AutoGluon is an open-source Python library that automates machine learning: feed it your data and it delivers strong predictive performance in about three lines of code.

    • structured-data
    • automl
    • hyperparameter-optimization
    • tabular-data
82. 82

    X-AnyLabeling

    10.7KCVHub520/X-AnyLabeling · Python

    A free cross-platform desktop app for AI-assisted annotation of text, image, video, and multimodal data — manual labeling tools plus built-in deep learning models that pre-label or batch-annotate, with multi-format export.

    • paddlepaddle
    • onnxruntime
    • point-cloud
    • instance-segmentation
83. 83

    nlp_course

    10.7Kyandexdataschool/nlp_course · Jupyter Notebook

    This is the Yandex School of Data Analysis (YSDA) Natural Language Processing course: a full set of lecture slides, seminar notebooks, and homework assignments organized by week, covering NLP from word embeddings through modern LLM topics.

    • interpretability
    • university
    • rlhf
    • attention-mechanism
84. 84

    ParlAI

    10.6Kfacebookresearch/ParlAI · Python

    ParlAI is a Python framework from Facebook (Meta) AI Research for sharing, training, and evaluating dialogue AI models, covering everything from open-domain chitchat to task-oriented dialogue and visual question answering, with over 100 datasets available through one unified API.

    • question-answering
    • conversational-ai
    • natural-language-processing
    • dataset
85. 85

    runanywhere-sdks

    10.3KRunanywhereAI/runanywhere-sdks · C++

    A toolkit of eight SDKs over one C++ core that runs AI fully on-device — LLM chat, vision, speech, voice agents, RAG, and image generation — on phones, browsers, and desktops. Everything is private by default and works offline.

    • apple-intelligence
    • edge
    • voice-ai
    • vlm
86. 86

    PixelRAG

    10.3KStarTrail-org/PixelRAG · Python

    An open-source research codebase from Berkeley for pixel-native search: PixelRAG renders web pages, PDFs and images as screenshots and retrieves over the images directly, instead of parsing pages into text.

    • vision
    • vlm
    • vector-search
    • vector-database
87. 87

    self-operating-computer

    10.3KOthersideAI/self-operating-computer · Python

    A framework that lets a multimodal AI model operate your computer: it views the screen and issues mouse and keyboard actions to reach the objective you give it. One of the first full computer-use projects (Nov 2023).

    • gpt-4o
    • gemini-api
    • desktop-automation
    • screen-recording
88. 88

    InternVL

    10.2KOpenGVLab/InternVL · Python

    A family of open-source multimodal chat models that understand images, video and text, positioned as an open alternative to GPT-4o. InternVL2.5-78B was the first open-source model to top 70% on the MMMU benchmark.

    • gpt-4o
    • semantic-segmentation
    • open-weights
    • vlm
89. 89

    MiniMax-H3

    10.1KMiniMax-AI/MiniMax-H3 · Python

    MiniMax H3 is a general-purpose omni-modal generative system: it reads mixed text, image, video and audio input and generates video with native stereo audio, up to 2K resolution and 15 seconds per clip.

    • minimax-h3
    • minimax
    • open-weights
    • vlm
90. 90

    moondream

    10.1Km87-labs/moondream · Python

    Moondream is a tiny, open-source vision language model that understands images — captioning, visual question answering, and object detection — while being small enough to run almost anywhere.

    • model
    • vision-language-model
    • edge-ai
    • object-detection
91. 91

    pyod

    10Kyzhao062/pyod · Python

    PyOD is a Python library for anomaly detection that bundles over 60 detection algorithms into one consistent API, covering tabular, time series, graph, text, image, and audio data.

    • fraud-detection
    • unsupervised-learning
    • anomaly-detection
    • data-mining
92. 92

    Kimi-K3

    8.9KMoonshotAI/Kimi-K3

    Kimi K3 is MoonshotAI's open-weight, native multimodal agentic model — a 2.8T-parameter Mixture-of-Experts system built for frontier-level long-horizon coding, agentic knowledge work, and reasoning.

    • long-horizon-agents
    • deep-research
    • open-weights
    • model
93. 93

    bifrost

    8.9Kmaximhq/bifrost · Go

    An open-source, high-performance AI gateway that unifies 23+ LLM providers behind a single OpenAI-compatible API, with automatic failover, load balancing, semantic caching, and budget controls built in.

    • failover
    • llm-router
    • secret-management
    • llm-observability
94. 94

    seedance-2.0

    7.8KEmily2040/seedance-2.0 · Python

    Seedance 2.0 Skill OS is an agent skill that helps you plan video shots, write detailed directed prompts, bind image/video/audio references, and continue from an already-accepted clip — the actual rendering is done by a separate Seedance 2.0 video provider.

    • image-to-video
    • text-to-video
    • ai-tools
    • video-generation
95. 95

    maths-cs-ai-compendium

    7.7KHenryNdubuaku/maths-cs-ai-compendium · TypeScript

    An open, unconventional online textbook covering maths, computing and AI from the ground up — intuition first, real-world context, for practitioners who want deep understanding rather than exam survival.

    • graph-neural-networks
    • machine-learning-algorithms
    • linear-algebra
    • from-scratch
96. 96

    GLM-OCR

    7.5Kzai-org/GLM-OCR · Python

    GLM-OCR is a multimodal OCR model for complex document understanding, built on a GLM-V encoder-decoder architecture with a lightweight SDK and inference toolchain around it.

    • sglang
    • flask
    • glm
    • vision-language-model
97. 97

    jiuwenswarm

    7.5KopenJiuwen-ai/jiuwenswarm · Python

    An AI agent system built on openJiuwen: you describe a task in natural language, and it orchestrates multi-agent collaboration, Skill self-evolution, and tool invocation to deliver a result end-to-end.

    • feishu
    • whatsapp
    • slack
    • discord
98. 98

    vllm-omni

    7.2Kvllm-project/vllm-omni · Python

    vLLM-Omni is an inference and serving framework that extends vLLM beyond text-only LLMs to omni-modality models, handling text, image, audio, video, and action outputs.

    • world-model
    • world-models
    • diffusion
    • model-serving
99. 99

    turbo-fieldfare

    6.9Kdrumih/turbo-fieldfare · Swift

    TurboFieldfare is a custom Swift and Metal runtime that runs the instruction-tuned Gemma 4 26B-A4B model on Apple Silicon Macs using only about 2 GB of RAM, even on 8 GB machines, by streaming only the experts needed for each token from SSD instead of loading the full 14.3 GB model.

    • gpgpu
    • gemma
    • on-device-ai
    • mixture-of-experts
100. 100

     SenseNova-U1

     6.9KOpenSenseNova/SenseNova-U1 · Python

     Open-source multimodal models that unify image understanding, reasoning and image generation in one architecture. The current release, SenseNova-U1.5-8B-MoT, creates and edits images up to native 4K.

     • open-weights
     • vision-language-model
     • comfyui
     • gguf

Showing the 100 most-starred of 191. See every one in the catalog.

• python2,440
• ai2,001
• llm1,523
• gpu815
• machine-learning931
• open-source5,768
• deep-learning748
• linux4,265
• apache-2.01,457
• research783
• huggingface368
• pytorch422
• computer-vision393
• inference331
• cuda404
• macos3,624
• windows3,493
• generative-ai355
• benchmark447
• fine-tuning189
• nvidia341
• self-hosted1,985
• agent845
• ai-agents1,330