mlc-llm README

The project's README from github.com/mlc-ai/mlc-llm. As one markdown file: README.md.

MLC LLM

Installation License Join Discoard Related Repository: WebLLM

Universal LLM Deployment Engine with ML Compilation

Get Started | Documentation | Blog

About

MLC LLM is a machine learning compiler and high-performance deployment engine for large language models. The mission of this project is to enable everyone to develop, optimize, and deploy AI models natively on everyone's platforms. 

AMD GPU NVIDIA GPU Apple GPU Intel GPU
Linux / Win ✅ Vulkan, ROCm ✅ Vulkan, CUDA N/A ✅ Vulkan
macOS ✅ Metal (dGPU) N/A ✅ Metal ✅ Metal (iGPU)
Web Browser ✅ WebGPU and WASM
iOS / iPadOS ✅ Metal on Apple A-series GPU
Android ✅ OpenCL on Adreno GPU ✅ OpenCL on Mali GPU

MLC LLM compiles and runs code on MLCEngine -- a unified high-performance LLM inference engine across the above platforms. MLCEngine provides OpenAI-compatible API available through REST server, python, javascript, iOS, Android, all backed by the same engine and compiler that we keep improving with the community.

Get Started

Please visit our documentation to get started with MLC LLM.

Citation

Please consider citing our project if you find it useful:

@software{mlc-llm,
    author = {{MLC team}},
    title = {{MLC-LLM}},
    url = {https://github.com/mlc-ai/mlc-llm},
    year = {2023-2025}
}

The underlying techniques of MLC LLM include:

References (Click to expand) 
@inproceedings{tensorir,
    author = {Feng, Siyuan and Hou, Bohan and Jin, Hongyi and Lin, Wuwei and Shao, Junru and Lai, Ruihang and Ye, Zihao and Zheng, Lianmin and Yu, Cody Hao and Yu, Yong and Chen, Tianqi},
    title = {TensorIR: An Abstraction for Automatic Tensorized Program Optimization},
    year = {2023},
    isbn = {9781450399166},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3575693.3576933},
    doi = {10.1145/3575693.3576933},
    booktitle = {Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2},
    pages = {804–817},
    numpages = {14},
    keywords = {Tensor Computation, Machine Learning Compiler, Deep Neural Network},
    location = {Vancouver, BC, Canada},
    series = {ASPLOS 2023}
}

@inproceedings{metaschedule,
    author = {Shao, Junru and Zhou, Xiyou and Feng, Siyuan and Hou, Bohan and Lai, Ruihang and Jin, Hongyi and Lin, Wuwei and Masuda, Masahiro and Yu, Cody Hao and Chen, Tianqi},
    booktitle = {Advances in Neural Information Processing Systems},
    editor = {S. Koyejo and S. Mohamed and A. Agarwal and D. Belgrave and K. Cho and A. Oh},
    pages = {35783--35796},
    publisher = {Curran Associates, Inc.},
    title = {Tensor Program Optimization with Probabilistic Programs},
    url = {https://proceedings.neurips.cc/paper_files/paper/2022/file/e894eafae43e68b4c8dfdacf742bcbf3-Paper-Conference.pdf},
    volume = {35},
    year = {2022}
}

@inproceedings{tvm,
    author = {Tianqi Chen and Thierry Moreau and Ziheng Jiang and Lianmin Zheng and Eddie Yan and Haichen Shen and Meghan Cowan and Leyuan Wang and Yuwei Hu and Luis Ceze and Carlos Guestrin and Arvind Krishnamurthy},
    title = {{TVM}: An Automated {End-to-End} Optimizing Compiler for Deep Learning},
    booktitle = {13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)},
    year = {2018},
    isbn = {978-1-939133-08-3},
    address = {Carlsbad, CA},
    pages = {578--594},
    url = {https://www.usenix.org/conference/osdi18/presentation/chen},
    publisher = {USENIX Association},
    month = oct,
}

mlc-llm (mlc-ai/mlc-llm)

Searching Database…

23.3K ·GitHub
Mlc-llm
What is this?MLC LLM is a machine learning compiler and high-performance deployment engine that lets you run large language models natively on almost any platform, from servers to phones to the browser.
Is this a compiler, a runtime, or both?Both: it compiles models with ML compilation techniques and then runs them through MLCEngine, a unified inference engine shared across all supported platforms.
How do I get it running?There's no single quick command in the README itself; you follow the project's documentation's Installation and Quick start guides to set up MLC LLM for your target platform.
What GPU backend do I need for my hardware?It depends on your device: Vulkan or ROCm for AMD GPUs, Vulkan or CUDA for NVIDIA GPUs, Metal for Apple GPUs, Vulkan for Intel GPUs, and WebGPU/WASM in the browser or OpenCL on mobile GPUs.
Does every GPU vendor work on every platform?No, support is matrix-based and not uniform: for example NVIDIA GPUs are not supported on macOS and AMD GPUs are not available in the web/iOS/Android backends.
What is MLCEngine?MLCEngine is the unified high-performance inference engine that MLC LLM compiles models into; the same engine runs on servers, iOS, Android and the web browser.
Does it expose an OpenAI-compatible API?Yes, MLCEngine exposes an OpenAI-compatible API through a REST server as well as Python, JavaScript, iOS, and Android bindings, all backed by the same engine.
Why compile models instead of just running them with PyTorch?ML compilation lets MLC LLM generate optimized, hardware-specific code (Vulkan, CUDA, Metal, ROCm, OpenCL, WebGPU) so the same model runs efficiently across very different platforms, something a generic framework runtime cannot do out of the box.
What underlying technology powers MLC LLM?It is built on the TVM machine-learning compilation stack, which the project uses to generate and optimize inference code for each target platform.
What are similar projects?tvm, qxotic, rizzo-flow, decider, llama.cpp, web-llm, qwen38-mtp, tilelang
Which topics is it in?opencl, rocm, language-model, metal, llm-inference, compiler, wasm, inference, cuda, rest-api, ios, android
Creator
mlc-ai
Size
36.3 MB
Stars
23.3K
Updated
CommitOct 06
Age
3y
Commits · 90 Days