Global

This repository contains the code for developing, pretraining, and finetuning a GPT-like LLM and is the official code repository for the book Build a Large Language Model (From Scratch).

One day, frontier AI research used to be done by meat computers in between eating, sleeping, having other fun, and synchronizing once in a while using sound wave interconnect in the ritual of "group meeting". That era is long gone.

Make sure to download the checkpoint and place the ckpt-0 directory in checkpoints - see Downloading the weights

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

OpenAI Model Craft Challenge: Parameter Golf is a challenge to train the best language model that fits in a 16MB artifact and trains in under 10 minutes on 8xH100s, evaluated by compression on the FineWeb validation set (tokenizer-agnostic, bits per byte).

/one is quick - most sentences take a 10th of a millisecond.

Base library for a lexer that can be used in Top-Down, Recursive Descent Parsers.

A substantial amount of effort has been put into making sure the outputs match exactly with what you would get with HuggingFace Tokenizers in this setting, but this is at a non-negligible cost to performance.

Rust library for generating vector embeddings and reranking locally!

We develop LongCat-Next, a native multimodal model that processes text, vision, and audio under a single autoregressive objective with minimal inductive bias beyond the language paradigm.