What is this?
This is the official code release for the DeepSeek paper 'Conditional Memory via Scalable Lookup', introducing Engram, a module that gives Transformers a native knowledge-lookup primitive by modernizing classic N-gram embeddings for O(1) retrieval, as a complement to Mixture-of-Experts sparsity.
This is the official code release for the DeepSeek paper 'Conditional Memory via Scalable Lookup', introducing Engram, a module that gives Transformers a native knowledge-lookup primitive by modernizing classic N-gram embeddings for O(1) retrieval, as a complement to Mixture-of-Experts sparsity.Is this a full trainable model or just demo code?It is a standalone demonstration implementation meant to show the data flow of the Engram module; it mocks surrounding components like Attention, MoE, and mHC rather than providing a complete trainable model.
It is a standalone demonstration implementation meant to show the data flow of the Engram module; it mocks surrounding components like Attention, MoE, and mHC rather than providing a complete trainable model.How do I try the code?Install Python 3.8+ and PyTorch, then run the standalone script in the repo that demonstrates the Engram module's core logic and data flow.
Install Python 3.8+ and PyTorch, then run the standalone script in the repo that demonstrates the Engram module's core logic and data flow.What do I need installed to run it?Just Python 3.8 or newer and PyTorch; no other setup is described for the demo code.
Just Python 3.8 or newer and PyTorch; no other setup is described for the demo code.Does this include the full Engram-27B model or training pipeline?No, only a simplified standalone demo of the Engram module is released; the paper's Engram-27B pre-training setup, MoE baselines, and real Attention/mHC components are not provided here.
No, only a simplified standalone demo of the Engram module is released; the paper's Engram-27B pre-training setup, MoE baselines, and real Attention/mHC components are not provided here.What exactly is the Engram module doing?It retrieves static N-gram-based memory via deterministic lookup and fuses it with the model's dynamic hidden states, giving the Transformer a conditional-memory mechanism alongside conditional-computation methods like MoE.
It retrieves static N-gram-based memory via deterministic lookup and fuses it with the model's dynamic hidden states, giving the Transformer a conditional-memory mechanism alongside conditional-computation methods like MoE.How does Engram compare to Mixture-of-Experts?Under matched parameter and FLOPs budgets, the Engram-27B model consistently outperforms MoE baselines on knowledge, reasoning, code, and math tasks, and the paper derives a U-shaped scaling law for splitting capacity between MoE compute and Engram memory.
Under matched parameter and FLOPs budgets, the Engram-27B model consistently outperforms MoE baselines on knowledge, reasoning, code, and math tasks, and the paper derives a U-shaped scaling law for splitting capacity between MoE compute and Engram memory.Why is Engram efficient at inference?Because its addressing is deterministic, the large embedding tables it relies on can be offloaded to host memory with minimal inference overhead, and it frees early layers from memorizing static patterns, preserving effective depth for reasoning.
Because its addressing is deterministic, the large embedding tables it relies on can be offloaded to host memory with minimal inference overhead, and it frees early layers from memorizing static patterns, preserving effective depth for reasoning.What are similar projects?Attention-Residuals, bdh, llama3-from-scratch, MSA, DeepGEMM, llm-architecture-from-scratch, minGPT, Chinese-BERT-wwm
Attention-Residuals, bdh, llama3-from-scratch, MSA, DeepGEMM, llm-architecture-from-scratch, minGPT, Chinese-BERT-wwmWhich topics is it in?mixture-of-experts, attention-mechanism, papers, paper, embeddings, neural-network, academic-research, large-language-models, transformer, nlp, artificial-intelligence, deepseek
mixture-of-experts, attention-mechanism, papers, paper, embeddings, neural-network, academic-research, large-language-models, transformer, nlp, artificial-intelligence, deepseek