This repository contains a curated list of 120+ LLM libraries category wise.
🚀 LLM Interview Questions and Answers Book
Crack modern LLM and Generative AI interviews with this comprehensive, interview-focused guide designed specifically for ML Engineers, AI Engineers, Data Scientists and Software Engineers.
This book features 100+ carefully curated LLM interview questions, each paired with clear answers and in-depth explanations so you truly understand the concepts interviewers care about. Get the book here.
Use the Coupon Code: LLMQA25 for an exclusive discount (50%) on the book. (Available only for a short period of time).
Stay Updated with Generative AI, LLMs, Agents and RAG.
Join 🚀 AIxFunda free newsletter to get latest updates and interesting tutorials related to Generative AI, LLMs, Agents and RAG.
- ✨ Weekly GenAI updates
- 📄 Weekly LLM, Agents and RAG paper updates
- 📝 1 fresh blog post on an interesting topic every week
Quick links
LLM Training and Fine-Tuning
| Library |
Description |
Link |
| unsloth |
Fine-tune LLMs faster with less memory. |
Link |
| PEFT |
State-of-the-art Parameter-Efficient Fine-Tuning library. |
Link |
| TRL |
Train transformer language models with reinforcement learning. |
Link |
| Transformers |
Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio. |
Link |
| Axolotl |
Tool designed to streamline post-training for various AI models. |
Link |
| LLMBox |
A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation. |
Link |
| LitGPT |
Train and fine-tune LLM lightning fast. |
Link |
| Mergoo |
A library for easily merging multiple LLM experts, and efficiently train the merged LLM. |
Link |
| Llama-Factory |
Easy and efficient LLM fine-tuning. |
Link |
| Ludwig |
Low-code framework for building custom LLMs, neural networks, and other AI models. |
Link |
| Txtinstruct |
A framework for training instruction-tuned models. |
Link |
| Lamini |
An integrated LLM inference and tuning platform. |
Link |
| XTuring |
xTuring provides fast, efficient and simple fine-tuning of open-source LLMs, such as Mistral, LLaMA, GPT-J, and more. |
Link |
| RL4LMs |
A modular RL library to fine-tune language models to human preferences. |
Link |
| DeepSpeed |
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. |
Link |
| torchtune |
A PyTorch-native library specifically designed for fine-tuning LLMs. |
Link |
| PyTorch Lightning |
A library that offers a high-level interface for pretraining and fine-tuning LLMs. |
Link |
| RapidFire AI |
Rapid AI experimentation framework for fine-tuning and post-training LLMs (SFT, DPO, GRPO) with hyperparallelized execution and real-time interactive control. |
Link |
LLM Application Development
Frameworks
| Library |
Description |
Link |
| LangChain |
LangChain is a framework for developing applications powered by large language models (LLMs). |
Link |
| Llama Index |
LlamaIndex is a data framework for your LLM applications. |
Link |
| HayStack |
Haystack is an end-to-end LLM framework that allows you to build applications powered by LLMs, Transformer models, vector search and more. |
Link |
| Prompt flow |
A suite of development tools designed to streamline the end-to-end development cycle of LLM-based AI applications. |
Link |
| Griptape |
A modular Python framework for building AI-powered applications. |
Link |
| Weave |
Weave is a toolkit for developing Generative AI applications. |
Link |
| Llama Stack |
Build Llama Apps. |
Link |
Data Preparation
| Library |
Description |
Link |
| Data Prep Kit |
Data Prep Kit accelerates unstructured data preparation for LLM app developers. Developers can use Data Prep Kit to cleanse, transform, and enrich use case-specific unstructured data to pre-train LLMs, fine-tune LLMs, instruct-tune LLMs, or build RAG applications. |
Link |
Multi API Access
| Library |
Description |
Link |
| LiteLLM |
Library to call 100+ LLM APIs in OpenAI format. |
Link |
| AI Gateway |
A Blazing Fast AI Gateway with integrated Guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast & friendly API. |
Link |
| Bifrost |
High-performance AI gateway with a unified OpenAI-compatible API, multi-provider routing, automatic fallbacks, load balancing, and observability. |
Link |
Routers
| Library |
Description |
Link |
| OpenRoutiQ |
Explainable, policy-controlled Python router for selecting models, providers, deployments, and reasoning levels, with optional outcome learning and an OpenAI-compatible proxy. |
Link |
| RouteLLM |
Framework for serving and evaluating LLM routers - save LLM costs without compromising quality. Drop-in replacement for OpenAI's client to route simpler queries to cheaper models. |
Link |
Memory
| Library |
Description |
Link |
| mem0 |
The Memory layer for your AI apps. |
Link |
| Memoripy |
An AI memory layer with short- and long-term storage, semantic clustering, and optional memory decay for context-aware applications. |
Link |
| Tree Ring Memory |
Local-first memory lifecycle for AI agents with Rust CLI/TUI, SQLite/FTS recall, audit, consolidation, and forgetting. |
Link |
| Letta (MemGPT) |
An open-source framework for building stateful LLM applications with advanced reasoning capabilities and transparent long-term memory |
Link |
| Memobase |
A user profile-based memory system designed to bring long-term user memory to your Generative AI applications. |
Link |
Interface
| Library |
Description |
Link |
| Streamlit |
A faster way to build and share data apps. Streamlit lets you transform Python scripts into interactive web apps in minutes |
Link |
| Gradio |
Build and share delightful machine learning apps, all in Python. |
Link |
| AI SDK UI |
Build chat and generative user interfaces. |
Link |
| AI-Gradio |
Create AI apps powered by various AI providers. |
Link |
| Simpleaichat |
Python package for easily interfacing with chat apps, with robust features and minimal code complexity. |
Link |
| Chainlit |
Build production-ready Conversational AI applications in minutes. |
Link |
Low Code
| Library |
Description |
Link |
| LangFlow |
LangFlow is a low-code app builder for RAG and multi-agent AI applications. It’s Python-based and agnostic to any model, API, or database. |
Link |
Cache
| Library |
Description |
Link |
| GPTCache |
A Library for Creating Semantic Cache for LLM Queries. Slash Your LLM API Costs by 10x 💰, Boost Speed by 100x. Fully integrated with LangChain and LlamaIndex. |
Link |
LLM RAG
| Library |
Description |
Link |
| FastGraph RAG |
Streamlined and promptable Fast GraphRAG framework designed for interpretable, high-precision, agent-driven retrieval workflows. |
Link |
| Chonkie |
RAG chunking library that is lightweight, lightning-fast, and easy to use. |
Link |
| RAGChecker |
A Fine-grained Framework For Diagnosing RAG. |
Link |
| RAG to Riches |
Build, scale, and deploy state-of-the-art Retrieval-Augmented Generation applications. |
Link |
| BeyondLLM |
Beyond LLM offers an all-in-one toolkit for experimentation, evaluation, and deployment of Retrieval-Augmented Generation (RAG) systems. |
Link |
| SQLite-Vec |
A vector search SQLite extension that runs anywhere! |
Link |
| fastRAG |
fastRAG is a research framework for efficient and optimized retrieval-augmented generative pipelines, incorporating state-of-the-art LLMs and Information Retrieval. |
Link |
| FlashRAG |
A Python Toolkit for Efficient RAG Research. |
Link |
| Llmware |
Unified framework for building enterprise RAG pipelines with small, specialized models. |
Link |
| Rerankers |
A lightweight unified API for various reranking models. |
Link |
| Vectara |
Build Agentic RAG applications. |
Link |
| RapidFire AI |
Rapid AI experimentation framework for RAG and context engineering with hyperparallelized, shard-based execution and real-time interactive control. |
Link |
LLM Inference
| Library |
Description |
Link |
| llama.cpp |
LLM inference in C/C++. |
Link |
| Ollama |
Local LLM inference. |
Link |
| vLLM |
High-throughput and memory-efficient inference and serving engine for LLMs. |
Link |
| TensorRT-LLM |
TensorRT-LLM is a library for optimizing Large Language Model (LLM) inference. |
Link |
| WebLLM |
High-performance In-browser LLM Inference Engine. |
Link |
| LLM Compressor |
Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment. |
Link |
| LightLLM |
Python-based LLM inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance. |
Link |
| torchchat |
Run PyTorch LLMs locally on servers, desktop, and mobile. |
Link |
LLM Serving
| Library |
Description |
Link |
| Langcorn |
Serving LangChain LLM apps and agents automagically with FastAPI. |
Link |
| LitServe |
Lightning-fast serving engine for any AI model of any size. It augments FastAPI with features like batching, streaming, and GPU autoscaling. |
Link |
| Library |
Description |
Link |
| Crawl4AI |
Open-source LLM Friendly Web Crawler & Scraper. |
Link |
| ScrapeGraphAI |
A web scraping Python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.). |
Link |
| Docling |
Docling parses documents and exports them to the desired format with ease |
|
A curated catalog of 120+ LLM libraries organized by category, covering training, application development, RAG, inference, serving, data extraction, agents, evaluation, monitoring, prompts, safety and embeddings.