What is this?
Petals lets you run big open LLMs at home, BitTorrent-style: you join a public swarm that serves models like Llama 3.1 (405B), Mixtral (8x22B), Falcon (40B+) and BLOOM (176B) for inference and fine-tuning right from your desktop or Google Colab, up to 10x faster than offloading.
Petals lets you run big open LLMs at home, BitTorrent-style: you join a public swarm that serves models like Llama 3.1 (405B), Mixtral (8x22B), Falcon (40B+) and BLOOM (176B) for inference and fine-tuning right from your desktop or Google Colab, up to 10x faster than offloading.Is it just an API for generating text, or can I do more with the model?It gives you the comforts of an API with the flexibility of PyTorch and Transformers: you can use any fine-tuning and sampling methods, execute custom paths through the model, and even inspect its hidden states.
It gives you the comforts of an API with the flexibility of PyTorch and Transformers: you can use any fine-tuning and sampling methods, execute custom paths through the model, and even inspect its hidden states.Will my Windows PC or Mac work, or do I need a Linux box?All of them work for hosting: Linux + Anaconda (NVIDIA or AMD GPUs), Windows through WSL, a Docker image for NVIDIA GPUs, or macOS with Apple M1/M2 GPUs via Homebrew. To just try it with zero setup, use their chatbot web app.
All of them work for hosting: Linux + Anaconda (NVIDIA or AMD GPUs), Windows through WSL, a Docker image for NVIDIA GPUs, or macOS with Apple M1/M2 GPUs via Homebrew. To just try it with zero setup, use their chatbot web app.Can I just load Llama and go, or is there a gate?Llama is gated: you must first request access to its weights, then run huggingface-cli login in the terminal before loading the model.
Llama is gated: you must first request access to its weights, then run huggingface-cli login in the terminal before loading the model.Is generation fast enough for interactive use, or will it crawl?Usable but not instant: single-batch inference runs at up to 6 tokens/sec for Llama 2 (70B) and up to 4 tokens/sec for Falcon (180B) — enough for chatbots and interactive apps.
Usable but not instant: single-batch inference runs at up to 6 tokens/sec for Llama 2 (70B) and up to 4 tokens/sec for Falcon (180B) — enough for chatbots and interactive apps.If I type a prompt into the public swarm, who sees my data?Other people do: in the public swarm your data is processed with the help of other participants. For sensitive data you can set up a private swarm among people you trust.
Other people do: in the public swarm your data is processed with the help of other participants. For sensitive data you can set up a private swarm among people you trust.Can I chip in my own GPU to the network?Yes — Petals is community-run and relies on people sharing GPUs. You can host part of, say, Llama 3.1 (405B) Instruct on an NVIDIA or AMD GPU (Linux), via Docker, or on macOS with Apple M1/M2 GPU.
Yes — Petals is community-run and relies on people sharing GPUs. You can host part of, say, Llama 3.1 (405B) Instruct on an NVIDIA or AMD GPU (Linux), via Docker, or on macOS with Apple M1/M2 GPU.Is it safe to share my GPU with strangers?Yes, per the README: hosting a server does not allow others to run custom code on your computer.
Yes, per the README: hosting a server does not allow others to run custom code on your computer.Can it serve models beyond the ones listed?Yes — besides the available models, you can host a new model from Hugging Face Model Hub, and there is a guide for running a custom model.
Yes — besides the available models, you can host a new model from Hugging Face Model Hub, and there is a guide for running a custom model.Is this a research project or production software?It is backed by peer-reviewed work: an ACL 2023 system demo paper on collaborative inference and fine-tuning, and a NeurIPS 2023 paper on distributed inference and fine-tuning of LLMs over the Internet.
It is backed by peer-reviewed work: an ACL 2023 system demo paper on collaborative inference and fine-tuning, and a NeurIPS 2023 paper on distributed inference and fine-tuning of LLMs over the Internet.What are similar projects?text-generation-inference, airllm, jeff, ColossalAI, jeeves, vllm, StableLM, litgpt
text-generation-inference, airllm, jeff, ColossalAI, jeeves, vllm, StableLM, litgptWhich topics is it in?distributed-computing, colab, community-driven, open-weights, model-serving, amd, neural-networks, llama, decentralized, pretrained-models, p2p, python-library
distributed-computing, colab, community-driven, open-weights, model-serving, amd, neural-networks, llama, decentralized, pretrained-models, p2p, python-library