llama-gpt README

The project's README from github.com/getumbrel/llama-gpt. As one markdown file: README.md.

LlamaGPT

LlamaGPT

A self-hosted, offline, ChatGPT-like chatbot, powered by Llama 2. 100% private, with no data leaving your device.
New: Support for Code Llama models and Nvidia GPUs.

umbrel.com (we're hiring) »

Contents

  1. Demo
  2. Supported Models
  3. How to install
  4. OpenAI-compatible API
  5. Benchmarks
  6. Roadmap and contributing
  7. Acknowledgements

Demo

https://github.com/getumbrel/llama-gpt/assets/10330103/5d1a76b8-ed03-4a51-90bd-12ebfaf1e6cd

Supported models

Currently, LlamaGPT supports the following models. Support for running custom models is on the roadmap.

Model name Model size Model download size Memory required
Nous Hermes Llama 2 7B Chat (GGML q4_0) 7B 3.79GB 6.29GB
Nous Hermes Llama 2 13B Chat (GGML q4_0) 13B 7.32GB 9.82GB
Nous Hermes Llama 2 70B Chat (GGML q4_0) 70B 38.87GB 41.37GB
Code Llama 7B Chat (GGUF Q4_K_M) 7B 4.24GB 6.74GB
Code Llama 13B Chat (GGUF Q4_K_M) 13B 8.06GB 10.56GB
Phind Code Llama 34B Chat (GGUF Q4_K_M) 34B 20.22GB 22.72GB

How to install

Install LlamaGPT on your umbrelOS home server

Running LlamaGPT on an umbrelOS home server is one click. Simply install it from the Umbrel App Store.

LlamaGPT on Umbrel App Store

Install LlamaGPT on M1/M2 Mac

Make sure your have Docker and Xcode installed.

Then, clone this repo and cd into it:

git clone https://github.com/getumbrel/llama-gpt.git
cd llama-gpt

Run LlamaGPT with the following command:

./run-mac.sh --model 7b

You can access LlamaGPT at http://localhost:3000.

To run 13B or 70B chat models, replace 7b with 13b or 70b respectively. To run 7B, 13B or 34B Code Llama models, replace 7b with code-7b, code-13b or code-34b respectively.

To stop LlamaGPT, do Ctrl + C in Terminal.

Install LlamaGPT anywhere else with Docker

You can run LlamaGPT on any x86 or arm64 system. Make sure you have Docker installed.

Then, clone this repo and cd into it:

git clone https://github.com/getumbrel/llama-gpt.git
cd llama-gpt

Run LlamaGPT with the following command:

./run.sh --model 7b

Or if you have an Nvidia GPU, you can run LlamaGPT with CUDA support using the --with-cuda flag, like:

./run.sh --model 7b --with-cuda

You can access LlamaGPT at http://localhost:3000.

To run 13B or 70B chat models, replace 7b with 13b or 70b respectively. To run Code Llama 7B, 13B or 34B models, replace 7b with code-7b, code-13b or code-34b respectively.

To stop LlamaGPT, do Ctrl + C in Terminal.

Note: On the first run, it may take a while for the model to be downloaded to the /models directory. You may also see lots of output like this for a few minutes, which is normal:

llama-gpt-llama-gpt-ui-1       | [INFO  wait] Host [llama-gpt-api-13b:8000] not yet available...

After the model has been automatically downloaded and loaded, and the API server is running, you'll see an output like:

llama-gpt-ui_1   | ready - started server on 0.0.0.0:3000, url: http://localhost:3000

You can then access LlamaGPT at http://localhost:3000.

Install LlamaGPT with Kubernetes

First, make sure you have a running Kubernetes cluster and kubectl is configured to interact with it.

Then, clone this repo and cd into it.

To deploy to Kubernetes first create a namespace:

kubectl create ns llama

Then apply the manifests under the /deploy/kubernetes directory with

kubectl apply -k deploy/kubernetes/. -n llama

Expose your service however you would normally do that.

OpenAI compatible API

Thanks to llama-cpp-python, a drop-in replacement for OpenAI API is available at http://localhost:3001. Open http://localhost:3001/docs to see the API documentation.

Benchmarks

We've tested LlamaGPT models on the following hardware with the default system prompt, and user prompt: "How does the universe expand?" at temperature 0 to guarantee deterministic results. Generation speed is averaged over the first 10 generations.

Feel free to add your own benchmarks to this table by opening a pull request.

Nous Hermes Llama 2 7B Chat (GGML q4_0)

Device Generation speed
M1 Max MacBook Pro (64GB RAM) 54 tokens/sec
GCP c2-standard-16 vCPU (64 GB RAM) 16.7 tokens/sec
Ryzen 5700G 4.4GHz 4c (16 GB RAM) 11.50 tokens/sec
GCP c2-standard-4 vCPU (16 GB RAM) 4.3 tokens/sec
Umbrel Home (16GB RAM) 2.7 tokens/sec
Raspberry Pi 4 (8GB RAM) 0.9 tokens/sec

Nous Hermes Llama 2 13B Chat (GGML q4_0)

Device Generation speed
M1 Max MacBook Pro (64GB RAM) 20 tokens/sec
GCP c2-standard-16 vCPU (64 GB RAM) 8.6 tokens/sec
GCP c2-standard-4 vCPU (16 GB RAM) 2.2 tokens/sec
Umbrel Home (16GB RAM) 1.5 tokens/sec

Nous Hermes Llama 2 70B Chat (GGML q4_0)

Device Generation speed
M1 Max MacBook Pro (64GB RAM) 4.8 tokens/sec
GCP e2-standard-16 vCPU (64 GB RAM) 1.75 tokens/sec
GCP c2-standard-16 vCPU (64 GB RAM) 1.62 tokens/sec

Code Llama 7B Chat (GGUF Q4_K_M)

Device Generation speed
M1 Max MacBook Pro (64GB RAM) 41 tokens/sec

Code Llama 13B Chat (GGUF Q4_K_M)

Device Generation speed
M1 Max MacBook Pro (64GB RAM) 25 tokens/sec

Phind Code Llama 34B Chat (GGUF Q4_K_M)

Device Generation speed
M1 Max MacBook Pro (64GB RAM) 10.26 tokens/sec

Roadmap and contributing

We're looking to add more features to LlamaGPT. You can see the roadmap here. The highest priorities are:

If you're a developer who'd like to help with any of these, please open an issue to discuss the best way to tackle the challenge. If you're looking to help but not sure where to begin, check out these issues that have specifically been marked as being friendly to new contributors.

Acknowledgements

A massive thank you to the following developers and teams for making LlamaGPT possible:

License

umbrel.com

llama-gpt (getumbrel/llama-gpt)

Searching Database…

10.9K ·GitHub
Llama-gpt
这是什么？LlamaGPT 是一个自托管、可离线使用、类似 ChatGPT 的聊天机器人，基于 Llama 2（现在也支持 Code Llama）。你可以使用熟悉的聊天界面，同时无需把任何数据发送到第三方云端。
这是能直接运行的应用，还是只是要我加载到别的程序里的模型？它是一个打包好的应用：基于 Docker 的服务，通过 llama.cpp 运行 Llama 2/Code Llama 模型，并将其封装在类似 ChatGPT 的网页界面（Chatbot UI）后面，同时还提供兼容 OpenAI 的 API。
我该怎样让 LlamaGPT 运行起来？在 umbrelOS 家庭服务器上，可以从 Umbrel 应用商店一键安装；在其他环境（Mac、Linux、x86/arm64）中，则需要克隆仓库并用 Docker 运行，或者将提供的清单应用到 Kubernetes 集群。
运行前需要先安装什么？所有平台都必须安装 Docker；M1/M2 Mac 还需要安装 Xcode；Nvidia GPU 是可选的，启用 --with-cuda 参数后可以加快推理速度。
我能加载自己的自定义模型，而不只用预设的吗？暂时不行。LlamaGPT 目前只内置一组固定的 Llama 2 和 Code Llama 检查点；加载任意自定义模型仍在路线图中。
我的聊天数据会离开我的设备吗？不会。LlamaGPT 的核心在于模型和所有推理都在本地运行，因此不会有任何内容发送到外部服务器。
没有 Nvidia GPU 能正常使用吗？可以。它通过 llama.cpp 开箱即用地支持 CPU 运行（即使是 Raspberry Pi 4 也能跑，只是很慢，7B 模型约 0.9 token/秒）；如果你有 Nvidia GPU，可以传入 --with-cuda 来显著加快生成速度。
LlamaGPT 实际能运行哪些模型？目前支持 7B、13B 和 70B 规格的 Nous Hermes Llama 2 Chat，7B 和 13B 规格的 Code Llama Chat，以及 Phind Code Llama 34B Chat 模型，均为预先量化的 GGML/GGUF 下载版本。
实际生成速度有多快？这取决于硬件：7B 模型在 M1 Max MacBook Pro 上约为 54 token/秒，但在 Raspberry Pi 4 上只有约 0.9 token/秒；即使是高端机器，70B 模型也只能跑到每秒不到 5 个 token。
我能把现有基于 OpenAI 的应用指向 LlamaGPT 吗？可以。它通过 llama-cpp-python 提供兼容 OpenAI 的 API，因此通常可以直接替换 OpenAI API 使用。
不碰 Docker 命令，最简单的试用方式是什么？如果你已经有 Umbrel 家庭服务器，只需在 Umbrel 应用商店中点击一次，即可直接安装 LlamaGPT。
What are similar projects?h2ogpt, jan, textgen, petals, gpt4free, OpenLLM, PS5LM, project-maya
Which topics is it in?localai, llama2, gpt-4, llamacpp, llama-cpp, llama, local-ai, llm-inference, gpt, gpu-acceleration, chatgpt, openai-compatible
Creator
getumbrel
Size
1.7 MB
Stars
10.9K
Updated
CommitOct 09
Age
3y
Commits · 90 Days