What is this?
LlamaGPT is a self-hosted, offline, ChatGPT-like chatbot powered by Llama 2 (and now Code Llama), so you get a familiar chat UI without sending any data to a third-party cloud.
LlamaGPT is a self-hosted, offline, ChatGPT-like chatbot powered by Llama 2 (and now Code Llama), so you get a familiar chat UI without sending any data to a third-party cloud.Is this a ready-to-run app, or just a model I load into something else?It is a packaged application: a Docker-based service that runs the Llama 2/Code Llama model through llama.cpp, exposes it behind a ChatGPT-style web UI (Chatbot UI), and also serves an OpenAI-compatible API.
It is a packaged application: a Docker-based service that runs the Llama 2/Code Llama model through llama.cpp, exposes it behind a ChatGPT-style web UI (Chatbot UI), and also serves an OpenAI-compatible API.How do I get LlamaGPT running?On an umbrelOS home server it's a one-click install from the Umbrel App Store; everywhere else (Mac, Linux, x86/arm64) you clone the repo and run it with Docker, or apply the provided manifests to a Kubernetes cluster.
On an umbrelOS home server it's a one-click install from the Umbrel App Store; everywhere else (Mac, Linux, x86/arm64) you clone the repo and run it with Docker, or apply the provided manifests to a Kubernetes cluster.What do I need installed before running it?Docker is required on every platform; on M1/M2 Macs you also need Xcode; an Nvidia GPU is optional and enabled with the --with-cuda flag for faster inference.
Docker is required on every platform; on M1/M2 Macs you also need Xcode; an Nvidia GPU is optional and enabled with the --with-cuda flag for faster inference.Can I load my own custom model instead of the presets?Not yet — LlamaGPT only ships with a fixed set of Llama 2 and Code Llama checkpoints; loading arbitrary custom models is still on the roadmap.
Not yet — LlamaGPT only ships with a fixed set of Llama 2 and Code Llama checkpoints; loading arbitrary custom models is still on the roadmap.Does any of my chat data leave my device?No — the whole point of LlamaGPT is that the model and all inference run locally, so nothing is sent to an external server.
No — the whole point of LlamaGPT is that the model and all inference run locally, so nothing is sent to an external server.Do I need an Nvidia GPU for this to be usable?No, it runs on CPU out of the box via llama.cpp (even a Raspberry Pi 4 works, just slowly at ~0.9 tokens/sec for the 7B model); if you have an Nvidia GPU you can pass --with-cuda to speed generation up significantly.
No, it runs on CPU out of the box via llama.cpp (even a Raspberry Pi 4 works, just slowly at ~0.9 tokens/sec for the 7B model); if you have an Nvidia GPU you can pass --with-cuda to speed generation up significantly.Which models can I actually run with LlamaGPT?It currently supports Nous Hermes Llama 2 Chat in 7B, 13B and 70B sizes, plus Code Llama Chat in 7B and 13B, and the Phind Code Llama 34B Chat model, all as pre-quantized GGML/GGUF downloads.
It currently supports Nous Hermes Llama 2 Chat in 7B, 13B and 70B sizes, plus Code Llama Chat in 7B and 13B, and the Phind Code Llama 34B Chat model, all as pre-quantized GGML/GGUF downloads.How fast is generation in practice?It varies a lot by hardware: the 7B model hits about 54 tokens/sec on an M1 Max MacBook Pro but only ~0.9 tokens/sec on a Raspberry Pi 4, while the 70B model runs at under 5 tokens/sec even on high-end machines.
It varies a lot by hardware: the 7B model hits about 54 tokens/sec on an M1 Max MacBook Pro but only ~0.9 tokens/sec on a Raspberry Pi 4, while the 70B model runs at under 5 tokens/sec even on high-end machines.Can I point my existing OpenAI-based app at LlamaGPT?Yes, it exposes an OpenAI-compatible API via llama-cpp-python, so you can often swap it in as a drop-in replacement for the OpenAI API.
Yes, it exposes an OpenAI-compatible API via llama-cpp-python, so you can often swap it in as a drop-in replacement for the OpenAI API.What's the simplest way to try it without touching Docker commands?If you already have an Umbrel home server, you can install LlamaGPT with a single click straight from the Umbrel App Store.
If you already have an Umbrel home server, you can install LlamaGPT with a single click straight from the Umbrel App Store.Which topics is it in?localai, llama2, gpt-4, llamacpp, llama-cpp, llama, local-ai, llm-inference, gpt, gpu-acceleration, chatgpt, openai-compatible
localai, llama2, gpt-4, llamacpp, llama-cpp, llama, local-ai, llm-inference, gpt, gpu-acceleration, chatgpt, openai-compatible