dalai README

The project's README from github.com/cocktailpeanut/dalai. As one markdown file: README.md.

Dalai

Run LLaMA and Alpaca on your computer.

GitHub Twitter Discord

JUST RUN THIS

TO GET

Both alpaca and llama working on your computer!

alpaca.gif

  1. Powered by llama.cpp, llama-dl CDN, and alpaca.cpp
  2. Hackable web app included
  3. Ships with JavaScript API
  4. Ships with Socket.io API

Intro

1. Cross platform

Dalai runs on all of the following operating systems:

  1. Linux
  2. Mac
  3. Windows

2. Memory Requirements

Runs on most modern computers. Unless your computer is very very old, it should work.

According to a llama.cpp discussion thread, here are the memory requirements:

3. Disk Space Requirements

Alpaca

Currently 7B and 13B models are available via alpaca.cpp

7B

Alpaca comes fully quantized (compressed), and the only space you need for the 7B model is 4.21GB:

alpaca_7b.png

13B

Alpaca comes fully quantized (compressed), and the only space you need for the 13B model is 8.14GB:

alpaca_13b.png

LLaMA

You need a lot of space for storing the models. The model name must be one of: 7B, 13B, 30B, and 65B.

You do NOT have to install all models, you can install one by one. Let's take a look at how much space each model takes up:

NOTE

The following numbers assume that you DO NOT touch the original model files and keep BOTH the original model files AND the quantized versions.

You can optimize this if you delete the original models (which are much larger) after installation and keep only the quantized versions.

7B

7b.png

13B

13b.png

30B

30b.png

65B

65b.png

Quickstart

Docker compose

Requires that you have docker installed and running.

docker compose build
docker compose run dalai npx dalai alpaca install 7B # or a different model
docker compose up -d

This will dave the models in the ./models folder

View the site at http://127.0.0.1:3000/

Mac

Step 1. Install node.js >= 18

If your mac doesn't have node.js installed yet, make sure to install node.js >= 18

Install Node.js

Step 2.1. Install models

Currently supported engines are llama and alpaca.

Add alpaca models

To download alpaca models, you can run:

npx dalai alpaca install 7B

Add llama models

To download llama models, you can run:

npx dalai llama install 7B

or to download multiple models:

npx dalai llama install 7B 13B

Now go to step 3.

Step 2.2. Troubleshoot

Normally you don't need this step, but if running the commands above don't do anything and immediately end, it means something went wrong because some of the required modules are not installed on your system.

In that case, try the following steps:

1. Install homebrew

In case homebrew is not installed on your computer, install it by running:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Or you can find the same instruction on the homebrew hompage: https://brew.sh/

2. Install dependencies

Once homebrew is installed, install these dependencies:

brew install cmake
brew install pkg-config

3. Update NPM

Just to make sure we cover every vector, let's update NPM as well:

npm install -g npm@latest

Now go back to step 2.1 and try running the npx dalai commands again.

Step 3. Run Web UI

After everything has been installed, run the following command to launch the web UI server:

npx dalai serve

and open http://localhost:3000 in your browser. Have fun!

Windows

Step 1. Install Visual Studio

On windows, you need to install Visual Studio before installing Dalai.

Press the button below to visit the Visual Studio downloads page and download:

Download Microsoft Visual Studio

IMPORTANT!!!

When installing Visual Studio, make sure to check the 3 options as highlighted below:

  1. Python development
  2. Node.js development
  3. Desktop development with C++

vs.png

Step 2.1. Install models

IMPORTANT

On Windows, make sure to run all commands in cmd.

DO NOT run in powershell. Powershell has unnecessarily strict permissions and makes the script fail silently.

Currently supported engines are llama and alpaca.

Install alpaca

To download alpaca models. Open your cmd application and enter:

npx dalai alpaca install 7B

Add llama models

To download llama models. Open your cmd application and enter:

npx dalai llama install 7B

or to download multiple models:

npx dalai llama install 7B 13B

Step 2.2. Troubleshoot (optional)

In case above steps fail, try installing Node.js and Python separately.

Install Python:

Download Python

Install Node.js >= 18:

Download Node.js

After both have been installed, open powershell and type python to see if the application exists. And also type node to see if the application exists as well.

Once you've checked that they both exist, try again.

Step 3. Run Web UI

After everything has been installed, run the following command to launch the web UI server (Make sure to run in cmd and not powershell!):

npx dalai serve

and open http://localhost:3000 in your browser. Have fun!

Linux

Step 1. Install Dependencies

You need to make sure you have the correct version of Python and Node.js installed.

Step 1.1. Python <= 3.10

Download Python

Make sure the version is 3.10 or lower (not 3.11) Python must be 3.10 or below (pytorch and other libraries are not supported yet on the latest)

Step 1.2. Node.js >= 18

Download node.js

Make sure the version is 18 or higher

Step 2.1. Install models

Currently supported engines are llama and alpaca.

Add alpaca models

To download alpaca models, you can run:

npx dalai alpaca install 7B

Add llama models

To download llama models, you can run:

npx dalai llama install 7B

or to download multiple models:

npx dalai llama install 7B 13B

Step 2.2. Troubleshoot

In case the model install silently fails or hangs forever, try the following command, and try running the npx command again:

On ubuntu/debian/etc.:

sudo apt-get install build-essential python3-venv -y

On fedora/etc.:

dnf install make automake gcc gcc-c++ kernel-devel python3-virtualenv -y

Step 3. Run Web UI

After everything has been installed, run the following command to launch the web UI server:

npx dalai serve

and open http://localhost:3000 in your browser. Have fun!

API

Dalai is also an NPM package:

  1. programmatically install
  2. locally make requests to the model
  3. run a dalai server (powered by socket.io)
  4. programmatically make requests to a remote dalai server (via socket.io)

Dalai is an NPM package. You can install it using:

npm install dalai

1. constructor()

Syntax

const dalai = new Dalai(home)

By default, Dalai automatically stores the entire llama.cpp repository under ~/llama.cpp.

However, often you may already have a llama.cpp repository somewhere else on your machine and want to just use that folder. In this case you can pass in the home attribute.

Examples

Basic

Creates a workspace at ~/llama.cpp

const dalai = new Dalai()

Custom path

Manually set the llama.cpp path:

const dalai = new Dalai("/Documents/llama.cpp")

2. request()

Syntax

dalai.request(req, callback)

Examples

1. Node.js

Using node.js, you just need to initialize a Dalai object with new Dalai() and then use it.

const Dalai = require('dalai')
new Dalai().request({
  model: "7B",
  prompt: "The following is a conversation between a boy and a girl:",
}, (token) => {
  process.stdout.write(token)
})

2. Non node.js (socket.io)

To make use of this in a browser or any other language, you can use thie socket.io API.

Step 1. start a server

First you need to run a Dalai socket server:

// server.js
const Dalai = require('dalai')
new Dalai().serve(3000)     // port 3000
Step 2. connect to the server

Then once the server is running, simply make requests to it by passing the ws://localhost:3000 socket url when initializing the Dalai object:

const Dalai = require("dalai")
new Dalai().request({
  url: "ws://localhost:3000",
  model: "7B",
  prompt: "The following is a conversation between a boy and a girl:",
}, (token) => {
  console.log("token", token)
})

3. serve()

Syntax

Starts a socket.io server at port

dalai.serve(port)

Examples

const Dalai = require("dalai")
new Dalai().serve(3000)

4. http()

Syntax

connect with an existing http instance (The http npm package)

dalai.http(http)

Examples

This is useful when you're trying to plug dalai into an existing node.js web app

const app = require('express')();
const http = require('http').Server(app);
dalai.http(http)
http.listen(3000, () => {
  console.log("server started")
})

5. install()

Syntax

await dalai.install(model_type, model_name1, model_name2, ...)

Examples

Install Llama "7B" and "13B" models:

const Dalai = require("dalai");
const dalai = new Dalai()
await dalai.install("llama", "7B", "13B")

Install alpaca 7B model:

const Dalai = require("dalai");
const dalai = new Dalai()
await dalai.install("alpaca", "7B")

6. installed()

returns the array of installed models

Syntax

const models = await dalai.installed()

Examples

const Dalai = require("dalai");
const dalai = new Dalai()
const models = await dalai.installed()
console.log(models)     // prints ["7B", "13B"]

FAQ

Using a different home folder

By default Dalai uses your home directory to store the entire repository (~/dalai). However sometimes you may want to store the archive elsewhere.

In this case you can call all CLI methods using the --home flag:

1. Installing models to a custom path

npx dalai llama install 7B --home ~/test_dir

2. Serving from the custom path

npx dalai serve --home ~/test_dir

Updating to the latest

To make sure you update to the latest, first find the latest version at https://www.npmjs.com/package/dalai

Let's say the latest version is 0.3.0. To update the dalai version, run:

npx [email protected] setup

Staying up to date

Have questions or feedback? Follow the project through the following outlets:

GitHub Twitter Discord

dalai (cocktailpeanut/dalai)

Searching Database…

12.9K ·GitHub
Dalai
这是什么？Dalai 是一个工具，可以让你直接在自己的电脑上下载并运行 LLaMA 或 Alpaca 语言模型。它提供网页界面、命令行工具（CLI）和 JavaScript/socket.io API，基于 llama.cpp 和 alpaca.cpp 构建。
这是桌面应用，还是我要在终端里运行的程序？两者都是：它是一个可以用 npx 或 node.js 运行的 CLI/NPM 包，同时附带一个可定制的本地网页应用，用来和模型聊天。它在 Linux、Mac 和 Windows 上的使用方式完全相同。
我该怎么把它跑起来？先安装 Node.js >=18，然后运行 npx dalai install 命令，指定 ‘alpaca’ 或 ‘llama’ 以及模型大小（7B、13B、30B、65B）来下载模型；最后运行 serve 命令，在浏览器中打开网页界面。此外还提供 docker-compose 方案作为替代。
除了 Node.js，安装前我还需要装什么？Windows 需要安装 Visual Studio，并勾选 Python 开发、Node.js 开发和使用 C++ 的桌面开发这几个工作负载；Linux 则需要 Python 3.10 或更低版本，因为更新的版本还不被 pytorch 和其他所需库支持。
安装时有哪些坑需要注意？在 Windows 上，所有命令都必须在 cmd 中运行，不能用 PowerShell，因为 PowerShell 更严格的权限会让安装脚本静默失败，不会给出任何报错；在 Linux 上，Python 版本必须是 3.10 或更低。
模型实际需要多少磁盘空间？量化版 Alpaca 模型，7B 约占 4.21GB，13B 约占 8.14GB；量化版 LLaMA 模型则从 7B 的约 4GB 到 65B 的约 40GB 不等，前提是同时保留原始文件和量化文件。
运行更大的模型需要多少内存？内存需求随模型大小增加：7B 约需 4GB 内存，13B 约需 8GB，30B 约需 16GB，65B 模型则需要 32GB。
不用网页界面，我能使用 Dalai 吗？可以。Dalai 以 NPM 包的形式发布，提供一个 Dalai 类，包含 install()、request()、serve() 和 http() 方法，因此你可以用代码的方式运行模型，也可以通过 socket.io 从任何语言连接到正在运行的 Dalai 服务器。
为什么它同时支持 LLaMA 和 Alpaca？Dalai 同时封装了 llama.cpp 和 alpaca.cpp，因此你可以用同一个命令行、网页应用和 API 下载并运行任一模型系列，并根据自己机器的情况选择合适的模型类型和大小。
What are similar projects?unsloth, Bonsai-demo, Strata, qxotic, rizzo-flow, llama.cpp, Second-Me, Janus
Which topics is it in?socket-io, llama, webui, websocket, local-llm, offline, inference, css, c++, local-first, web-app, npm
Creator
cocktailpeanut
Size
12 MB
Stars
12.9K
Updated
CommitMar 22Release0.1.0Mar 13
Age
3y
Commits · 90 Days