higgsfield README

The project's README from github.com/higgsfield-ai/higgsfield. As one markdown file: README.md.

higgsfield - multi node training without crying

Higgsfield is an open-source, fault-tolerant, highly scalable GPU orchestration, and a machine learning framework designed for training models with billions to trillions of parameters, such as Large Language Models (LLMs).

PyPI version

architecture

Higgsfield serves as a GPU workload manager and machine learning framework with five primary functions:

  1. Allocating exclusive and non-exclusive access to compute resources (nodes) to users for their training tasks.
  2. Supporting ZeRO-3 deepspeed API and fully sharded data parallel API of PyTorch, enabling efficient sharding for trillion-parameter models.
  3. Offering a framework for initiating, executing, and monitoring the training of large neural networks on allocated nodes.
  4. Managing resource contention by maintaining a queue for running experiments.
  5. Facilitating continuous integration of machine learning development through seamless integration with GitHub and GitHub Actions. Higgsfield streamlines the process of training massive models and empowers developers with a versatile and robust toolset.

Install

$ pip install higgsfield==0.0.3

Train example

That's all you have to do in order to train LLaMa in a distributed setting:

from higgsfield.llama import Llama70b
from higgsfield.loaders import LlamaLoader
from higgsfield.experiment import experiment

import torch.optim as optim
from alpaca import get_alpaca_data

@experiment("alpaca")
def train(params):
    model = Llama70b(zero_stage=3, fast_attn=False, precision="bf16")

    optimizer = optim.AdamW(model.parameters(), lr=1e-5, weight_decay=0.0)

    dataset = get_alpaca_data(split="train")
    train_loader = LlamaLoader(dataset, max_words=2048)

    for batch in train_loader:
        optimizer.zero_grad()
        loss = model(batch)
        loss.backward()
        optimizer.step()

    model.push_to_hub('alpaca-70b')

How it's all done?

  1. We install all the required tools in your server (Docker, your project's deploy keys, higgsfield binary).
  2. Then we generate deploy & run workflows for your experiments.
  3. As soon as it gets into Github, it will automatically deploy your code on your nodes.
  4. Then you access your experiments' run UI through Github, which will launch experiments and save the checkpoints.

Design

We follow the standard pytorch workflow. Thus you can incorporate anything besides what we provide, deepspeed, accelerate, or just implement your custom pytorch sharding from scratch.

Enviroment hell

No more different versions of pytorch, nvidia drivers, data processing libraries. You can easily orchestrate experiments and their environments, document and track the specific versions and configurations of all dependencies to ensure reproducibility.

Config hell

No need to define 600 arguments for your experiment. No more yaml witchcraft. You can use whatever you want, whenever you want. We just introduce a simple interface to define your experiments. We have even taken it further, now you only need to design the way to interact.

Compatibility

We need you to have nodes with:

Clouds we have tested on:

Feel free to open an issue if you have any problems with other clouds.

Getting started

Setup

Here you can find the quick start guide on how to setup your nodes and start training.

Tutorial

API for common tasks in Large Language Models training.

Platform Purpose Estimated Response Time Support Level
Github Issues Bug reports, feature requests, install issues, usage issues, etc. < 1 day Higgsfield Team
Twitter For staying up-to-date on new features. Daily Higgsfield Team
Website Discussion, news. < 2 days Higgsfield Team

higgsfield (higgsfield-ai/higgsfield)

azure
Searching Database…

6.1K ·GitHub
Higgsfield
What is this?Higgsfield is an open-source, fault-tolerant GPU orchestration system and machine learning framework for training huge models, from billions to trillions of parameters, such as LLMs.
Is this a cluster scheduler, a training framework, or both?Both: it is a GPU workload manager that allocates compute to users and queues experiments, plus a PyTorch-based framework for launching, running, and monitoring distributed training jobs.
How do I get a training job running?You install the higgsfield binary which sets up Docker, deploy keys, and tooling on your server, then it generates deploy and run GitHub workflows so pushing to GitHub automatically deploys your code to your nodes.
What do my training nodes need to meet?Each node must run Ubuntu, be reachable over SSH, and have a non-root user with passwordless sudo privileges.
Which cloud providers actually work with it?It has only been tested on Azure, LambdaLabs, and FluidStack; other clouds may work but aren't guaranteed, and you're asked to open an issue if problems arise.
Does it support DeepSpeed or PyTorch FSDP sharding?Yes, Higgsfield supports the ZeRO-3 DeepSpeed API and PyTorch's fully sharded data parallel (FSDP) API for efficient sharding of trillion-parameter models.
What problem is the experiment queue solving?It manages resource contention among users by keeping a queue for running experiments, so exclusive and non-exclusive node access can be allocated fairly.
Why not just write raw PyTorch training scripts?Higgsfield exists to kill 'environment hell' and 'config hell': it standardizes dependency/driver versions across nodes and replaces sprawling YAML configs with a simple experiment interface.
How does GitHub fit into the training workflow?GitHub Actions is used as the deployment backbone: once your experiment code lands in GitHub, workflows automatically deploy it to your nodes and let you watch runs and checkpoints through GitHub's UI.
What are similar projects?metaflow, ray, ludwig, Meshy, skypilot, verl, train-llm-from-scratch, ColossalAI
Which topics is it in?cluster-management, llama2, model-training, distributed, distributed-training, mlops, azure, llama, training, ssh, neural-network, github-actions
Creator
higgsfield-ai
Size
5 MB
Stars
6.1K
Updated
CommitFeb 13Releasev0.0.4-rcMar 23
Age
8y
Commits · 90 Days