What is this?
MiniMind is an open-source project that lets you train a tiny large language model (around 64M parameters) completely from scratch, in about 2 hours on a single RTX 3090 for roughly the cost of 3 RMB in GPU rental.
MiniMind is an open-source project that lets you train a tiny large language model (around 64M parameters) completely from scratch, in about 2 hours on a single RTX 3090 for roughly the cost of 3 RMB in GPU rental.Is this a ready-made chatbot or a training framework I run myself?It is a training framework and tutorial: every core algorithm (pretraining, SFT, LoRA, DPO, PPO/GRPO, tool use, distillation) is implemented natively in PyTorch without relying on high-level wrappers from libraries like transformers/trl/peft, so you build and train the model yourself.
It is a training framework and tutorial: every core algorithm (pretraining, SFT, LoRA, DPO, PPO/GRPO, tool use, distillation) is implemented natively in PyTorch without relying on high-level wrappers from libraries like transformers/trl/peft, so you build and train the model yourself.How do I train my own MiniMind model?Install the requirements, download the pretrain_t2t_mini.jsonl and sft_t2t_mini.jsonl datasets into ./dataset, then run the pretrain script followed by the full SFT script to get out/pretrain.pth and out/full_sft.pth.
Install the requirements, download the pretrain_t2t_mini.jsonl and sft_t2t_mini.jsonl datasets into ./dataset, then run the pretrain script followed by the full SFT script to get out/pretrain.pth and out/full_sft.pth.What hardware and software do I need to train it?The author's reference setup is an Intel i9, 128GB RAM, an RTX 3090 (24GB), Ubuntu 20.04, CUDA 12.2 and Python 3.10.16, with dependencies listed in requirements.txt.
The author's reference setup is an Intel i9, 128GB RAM, an RTX 3090 (24GB), Ubuntu 20.04, CUDA 12.2 and Python 3.10.16, with dependencies listed in requirements.txt.Do I need an NVIDIA GPU, or can I run it on CPU or Mac?CUDA is recommended for speed, but training can also fall back to CPU or Apple's MPS backend, though speed and compatibility will differ significantly without CUDA.
CUDA is recommended for speed, but training can also fall back to CPU or Apple's MPS backend, though speed and compatibility will differ significantly without CUDA.Why train from scratch instead of just fine-tuning an existing big model?The author argues that LoRA-style fine-tuning of huge existing models is like teaching Newton to use a smartphone — it feels productive but skips understanding the underlying mechanics. MiniMind lets you understand every line of code by building a model from zero yourself.
The author argues that LoRA-style fine-tuning of huge existing models is like teaching Newton to use a smartphone — it feels productive but skips understanding the underlying mechanics. MiniMind lets you understand every line of code by building a model from zero yourself.What inference engines and tools does MiniMind work with?It is compatible with transformers, trl and peft, as well as popular inference engines like llama.cpp, vllm and ollama, plus training frameworks like Llama-Factory.
It is compatible with transformers, trl and peft, as well as popular inference engines like llama.cpp, vllm and ollama, plus training frameworks like Llama-Factory.Can I chat with a trained MiniMind model through an API or web UI?Yes, there is a minimal server implementing the OpenAI API protocol (so you can plug it into tools like FastGPT or Open-WebUI), plus a Streamlit-based chat web UI with tool-call and reasoning display support.
Yes, there is a minimal server implementing the OpenAI API protocol (so you can plug it into tools like FastGPT or Open-WebUI), plus a Streamlit-based chat web UI with tool-call and reasoning display support.What other model variants does the project include besides the base text LLM?Alongside the main dense/MoE MiniMind, the project also offers a vision model (MiniMind-V), a multimodal model (MiniMind-O), a diffusion language model (MiniMind-dLM) and a linear-attention model (MiniMind-Linear).
Alongside the main dense/MoE MiniMind, the project also offers a vision model (MiniMind-V), a multimodal model (MiniMind-O), a diffusion language model (MiniMind-dLM) and a linear-attention model (MiniMind-Linear).Which topics is it in?large-language-model, ppo, knowledge-distillation, grpo, streamlit, llama-cpp, tool-calling, openai-api, moe, lora, transformers, vllm
large-language-model, ppo, knowledge-distillation, grpo, streamlit, llama-cpp, tool-calling, openai-api, moe, lora, transformers, vllm