What is this?
llama.cpp is a plain C/C++ implementation for running LLM (and VLM) inference with minimal setup and state-of-the-art performance, either locally or in the cloud.
llama.cpp is a plain C/C++ implementation for running LLM (and VLM) inference with minimal setup and state-of-the-art performance, either locally or in the cloud.Is this a library or a ready-to-use tool?Both: it is a C/C++ library (libllama) you can embed in your own code, and it also ships ready-made CLI tools like 'llama-cli' and 'llama-server' built on top of that library.
Both: it is a C/C++ library (libllama) you can embed in your own code, and it also ships ready-made CLI tools like 'llama-cli' and 'llama-server' built on top of that library.How do I get it running?You can download pre-built binaries from the releases page, run it with Docker, or build it from source by cloning the repository and following the build guide.
You can download pre-built binaries from the releases page, run it with Docker, or build it from source by cloning the repository and following the build guide.Is every backend fully finished and ready to use?Most backends (CUDA, Metal, Vulkan, HIP, SYCL, CANN, etc.) are production-ready, but the OpenVINO backend is still marked as in-progress.
Most backends (CUDA, Metal, Vulkan, HIP, SYCL, CANN, etc.) are production-ready, but the OpenVINO backend is still marked as in-progress.What range of hardware does it actually run on?It covers x86 (AVX/AVX2/AVX512/AMX), Apple Silicon and ARM (NEON, Accelerate, Metal), RISC-V, and GPUs from NVIDIA, AMD, Intel, Moore Threads and Qualcomm Adreno.
It covers x86 (AVX/AVX2/AVX512/AMX), Apple Silicon and ARM (NEON, Accelerate, Metal), RISC-V, and GPUs from NVIDIA, AMD, Intel, Moore Threads and Qualcomm Adreno.What is ggml?ggml is the underlying tensor and compute library that llama.cpp is built on top of, providing the low-level math and hardware backends.
ggml is the underlying tensor and compute library that llama.cpp is built on top of, providing the low-level math and hardware backends.Does it support quantized models?Yes, it supports 1.5-bit through 8-bit integer quantization to cut memory use and speed up inference on constrained hardware.
Yes, it supports 1.5-bit through 8-bit integer quantization to cut memory use and speed up inference on constrained hardware.Can I chat with it through a web UI instead of the terminal?Yes, 'llama-server' includes a built-in web UI alongside its REST API, so you can use it like a local ChatGPT-style app.
Yes, 'llama-server' includes a built-in web UI alongside its REST API, so you can use it like a local ChatGPT-style app.Can it handle more than just text models?Yes, newer builds support VLM (vision-language model) sessions directly through the 'llama' CLI.
Yes, newer builds support VLM (vision-language model) sessions directly through the 'llama' CLI.What are similar projects?TensorFold, Strata, rizzo-flow, ggml, project-maya, deltafin, tilelang, BitNet
TensorFold, Strata, rizzo-flow, ggml, project-maya, deltafin, tilelang, BitNetWhich topics is it in?x86, arm, inference-engine, amd, vulkan, webui, llm-inference, high-performance, quantization, local-llm, c, inference
x86, arm, inference-engine, amd, vulkan, webui, llm-inference, high-performance, quantization, local-llm, c, inference