What is this?
Grok-1 is the 314B-parameter Mixture-of-Experts language model that xAI opened up; this repository holds the JAX example code for loading and running it.
Grok-1 is the 314B-parameter Mixture-of-Experts language model that xAI opened up; this repository holds the JAX example code for loading and running it.Are the model weights included, or just the code?Just the code — the open weights are a separate download: you fetch them with a torrent client via the magnet link or straight from HuggingFace Hub.
Just the code — the open weights are a separate download: you fetch them with a torrent client via the magnet link or straight from HuggingFace Hub.How do I get it running once I have the weights?Place the downloaded ckpt-0 directory inside checkpoints, then run the test script: it loads the checkpoint and samples from the model on a test input.
Place the downloaded ckpt-0 directory inside checkpoints, then run the test script: it loads the checkpoint and samples from the model on a test input.Is the included code efficient?No — the MoE layer implementation is deliberately inefficient: it avoids custom kernels so the model's correctness could be validated easily.
No — the MoE layer implementation is deliberately inefficient: it avoids custom kernels so the model's correctness could be validated easily.Does the Apache 2.0 license cover both the code and the weights?Yes — the code and the Grok-1 weights are both under Apache 2.0, which applies to this repository's source files and the model weights of Grok-1.
Yes — the code and the Grok-1 weights are both under Apache 2.0, which applies to this repository's source files and the model weights of Grok-1.Will this run on a regular GPU?Not really — at 314B parameters, the README requires a machine with enough GPU memory even just to test the model with the example code.
Not really — at 314B parameters, the README requires a machine with enough GPU memory even just to test the model with the example code.How big is the model, exactly?314B parameters in a Mixture-of-8-Experts architecture, with 2 experts used per token, 64 layers, 48 query heads (8 for keys/values) and a 6,144 embedding size.
314B parameters in a Mixture-of-8-Experts architecture, with 2 experts used per token, 64 layers, 48 query heads (8 for keys/values) and a 6,144 embedding size.How long a context can it handle?Up to 8,192 tokens of context, with text tokenized by a SentencePiece tokenizer whose vocabulary holds 131,072 tokens.
Up to 8,192 tokens of context, with text tokenized by a SentencePiece tokenizer whose vocabulary holds 131,072 tokens.Does it support quantization or sharding?Yes — the model supports activation sharding and 8-bit quantization, and uses rotary embeddings (RoPE) for positional information.
Yes — the model supports activation sharding and 8-bit quantization, and uses rotary embeddings (RoPE) for positional information.What are similar projects?DeepSeek-V3, Kimi-K3, OpenMythos, ktransformers, StartLux-Decision, Hy4-preview, LongCat-Next, Hy-MT2
DeepSeek-V3, Kimi-K3, OpenMythos, ktransformers, StartLux-Decision, Hy4-preview, LongCat-Next, Hy-MT2Which topics is it in?tokenizer, xai, jax, torrent, apache, open-weights, mixture-of-experts, moe, grok, quantization, large-language-models, transformer
tokenizer, xai, jax, torrent, apache, open-weights, mixture-of-experts, moe, grok, quantization, large-language-models, transformer