LittleBit README

The project's README from github.com/SamsungLabs/LittleBit. As one markdown file: README.md.

The LittleBit Project

Sub-1-Bit LLM Compression via Latent Factorization

Official implementation of LittleBit (NeurIPS 2025) and LittleBit-2 (ICML 2026).

Papers

LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignment (ICML 2026)
Banseok Lee, Youngmin Kim
arXiv ICML

LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025)
Banseok Lee*, Dongkyu Kim*, Youngcheon You, Youngmin Kim
arXiv NeurIPS

Abstract

LittleBit compresses large language models into the sub-1-bit regime by factorizing each dense weight matrix into low-rank latent factors, binarizing those factors, and restoring magnitude information through lightweight learned scales. This enables extreme compression, including the 0.1 bits-per-weight setting, while preserving the original model architecture at inference time.

LittleBit-2 improves this recipe by addressing latent geometry misalignment in the initialization stage. It applies Internal Latent Rotation with Joint Iterative Quantization (Joint-ITQ), aligning the SVD-derived latent factors with the binary hypercube before QAT. LittleBit-2 initialization is available as an opt-in (--use_itq) and produces no additional inference overhead.

Highlights

Supported Models

The codebase currently supports:

Installation

We recommend Python 3.12.

conda create -n littlebit python=3.12
conda activate littlebit

# Install CUDA toolkit. Adjust the CUDA version if needed.
conda install nvidia/label/cuda-12.4.1::cuda-toolkit -c nvidia/label/cuda-12.4.1

# Install PyTorch.
pip install torch==2.8.0+cu124 torchvision==0.23.0+cu124 torchaudio==2.8.0+cu124 --index-url https://download.pytorch.org/whl/cu124

# Install dependencies.
pip install -r requirements.txt

[!IMPORTANT] For reproducing the paper results, use transformers 4.51.x. Newer transformers releases may change model internals or evaluation behavior.

pip install "transformers==4.51.*"

Usage

Training

Train a model with Quantization-Aware Training. By default, LittleBitLinear uses the original SVD-only initialization. To enable LittleBit-2 (Joint-ITQ), pass --use_itq True.

Single GPU

CUDA_VISIBLE_DEVICES=0 python -m main \
    --model_id meta-llama/Llama-2-7b-hf \
    --dataset c4_wiki \
    --save_dir ./outputs/Llama-2-7b-LittleBit-2 \
    --num_train_epochs 5.0 \
    --per_device_train_batch_size 4 \
    --lr 4e-05 \
    --warmup_ratio 0.02 \
    --report wandb \
    --quant_func SmoothSign \
    --quant_mod LittleBitLinear \
    --residual True \
    --eff_bit 1.0 \
    --kv_factor 1.0 \
    --min_split_dim 8 \
    --l2l_loss_scale 10.0

# Opt-in to LittleBit-2 initialization
# --use_itq True

Multi-GPU with DeepSpeed

deepspeed --num_gpus=4 main.py \
    --model_id meta-llama/Llama-2-7b-hf \
    --dataset c4_wiki \
    --save_dir ./outputs/Llama-2-7b-LittleBit-2 \
    --ds_config_path configs/zero3.json \
    --num_train_epochs 5.0 \
    --per_device_train_batch_size 4 \
    --lr 4e-05 \
    --report wandb \
    --quant_func SmoothSign \
    --quant_mod LittleBitLinear \
    --residual True \
    --eff_bit 1.0 \
    --kv_factor 1.0 \
    --min_split_dim 8

Evaluation

Evaluate a local checkpoint or a model hosted on the Hugging Face Hub.

# From a local directory
CUDA_VISIBLE_DEVICES=0 python eval.py \
    --model_id ./outputs/Llama-2-7b-LittleBit-2 \
    --seqlen 2048 \
    --ppl_task wikitext2,c4 \
    --zeroshot_task boolq,piqa,hellaswag,winogrande,arc_easy,arc_challenge,openbookqa

# From the Hugging Face Hub
CUDA_VISIBLE_DEVICES=0 python eval.py \
    --model_id username/littlebit-llama-7b-0.1bpw \
    --seqlen 2048 \
    --ppl_task wikitext2

Legacy Checkpoints

Older checkpoints may not include littlebit_config.json. In that case, pass the quantization arguments explicitly:

CUDA_VISIBLE_DEVICES=0 python eval.py \
    --model_id ./outputs/Legacy-Llama-2-7b \
    --quant_func SmoothSign \
    --quant_mod LittleBitLinear \
    --split_dim 1024

Parameter loading priority:

  1. Explicit CLI arguments
  2. littlebit_config.json in the model directory
  3. config.json fallback for older checkpoints

Citation

If you find this work useful, please cite:

@inproceedings{lee2026littlebit2,
  title={LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignment},
  author={Lee, Banseok and Kim, Youngmin},
  booktitle={Proceedings of the 43rd International Conference on Machine Learning},
  year={2026}
}

@inproceedings{lee2025littlebit,
  title={LittleBit: Ultra Low-Bit Quantization via Latent Factorization},
  author={Lee, Banseok and Kim, Dongkyu and You, Youngcheon and Kim, Youngmin},
  booktitle={Advances in Neural Information Processing Systems},
  year={2025}
}

License

This project is licensed under the CC BY-NC 4.0 license.

LittleBit (SamsungLabs/LittleBit)

Searching Database…

283 ·GitHub
LittleBit
What is this?LittleBit is the official research code for compressing large language models to sub-1-bit precision (down to 0.1 bits per weight) by factorizing each weight matrix into low-rank latent factors and binarizing them, with magnitude restored through learned scales. It also includes LittleBit-2, a follow-up that improves initialization via latent geometry alignment.
Is this a ready-to-use tool or research code I run myself?It's a research codebase (scripts for training and evaluation), not a packaged app or library you pip-install into a product; you clone it and run the training/evaluation scripts directly on supported model families.
How do I train a model with this?You run the quantization-aware training (QAT) script on a single GPU or with DeepSpeed across multiple GPUs; by default it uses SVD-only initialization, and passing --useitq True enables the LittleBit-2 Joint-ITQ initialization instead.
What software version do I need to reproduce the paper results?Python 3.12 is recommended, and for reproducing the paper's exact results you need transformers version 4.51.x specifically, since newer releases can change model internals or evaluation behavior.
Does LittleBit-2 change how the compressed model runs at inference?No — LittleBit-2's Joint-ITQ only affects the initialization stage before QAT; the deployed factorized layer structure and inference-time behavior stay identical to standard LittleBit.
Which model families does it support?OPT, Llama (and Llama 2/3), Phi-4, Qwen2.5, QwQ, Gemma 2 and Gemma 3, and Qwen3.
Can I use this commercially?No — despite GitHub showing no asserted license, the README states the project is licensed under CC BY-NC 4.0, which is non-commercial only.
How do I evaluate an already-trained checkpoint?You can evaluate a local checkpoint or a model hosted on the Hugging Face Hub directly with the evaluation script.
What happens if a checkpoint is missing littlebitconfig.json?For older checkpoints without this file, you must pass the quantization arguments explicitly on the CLI; otherwise the loader falls back to config.json, with CLI args taking highest priority.
What's the difference between LittleBit and LittleBit-2?LittleBit does SVD-only latent factorization and binarization; LittleBit-2 adds an opt-in Joint-ITQ step that aligns the latent factors with the binary hypercube before training, improving spectral energy retention with zero extra inference cost.
Who published this research?LittleBit was published at NeurIPS 2025 and LittleBit-2 at ICML 2026, authored by Banseok Lee, Dongkyu Kim, Youngcheon You, and Youngmin Kim.
What are similar projects?jeeves, Attention-Residuals, train-llm-from-scratch, ZDTaichu5.0-9B, DMI, Megatron-LM, Model-Optimizer, qlora
Which topics is it in?llm-training, model-training, deep-neural-networks, llms, transformers, quantization, academic-research, transformer, large-language-models, inference, huggingface, cuda
Creator
SamsungLabs
Size
<0.1 MB
Stars
283
Updated
CommitOct 08
Age
10mo
Commits · 90 Days