Efficient Lifelong Memory for LLM Agents — Text & Multimodal
Store, compress, and retrieve long-term memories with semantic lossless compression. Now with multimodal support for text, image, audio & video.
Works with any AI platform that supports MCP (text memory) or Python integration (full multimodal)
🎁 Atlas Cloud is a full-modal, OpenAI-compatible AI inference platform — point SimpleMem's OPENAI_BASE_URL at it to power memory construction, retrieval, and judging with DeepSeek, Qwen, GLM, Kimi, MiniMax and more through a single API, no multi-vendor setup needed. Budget-friendly coding plan available.
🔥 News
- [05/21/2026] 📦 Unified
simplemem package — one import, auto-routing! SimpleMem, Omni-SimpleMem, and EvolveMem now live in a single package. from simplemem import SimpleMem auto-selects the text or multimodal backend from the first method you call, and simplemem.optimize(...) taps EvolveMem's self-evolution loop. Install in one step with pip install -e ..
- [05/14/2026] 🧬 EvolveMem (v3.0) — Self-Evolving Memory via AutoResearch! The retrieval infrastructure itself now self-evolves through LLM-driven closed-loop diagnosis. On LoCoMo, EvolveMem outperforms the strongest baseline by +25.7% relative; on MemBench, by +18.9% relative. The system discovers entirely new retrieval dimensions not present in the original design. View EvolveMem →
- [04/02/2026] 🧠 Omni-SimpleMem (v2.0) — Multimodal Memory is Here! SimpleMem now supports text, image, audio & video memory. Achieving new SOTA on LoCoMo (F1=0.613, +47%) and Mem-Gallery (F1=0.810, +51%) over previous best. View Omni-SimpleMem →
- [02/09/2026] 🚀 Cross-Session Memory — Outperforming Claude-Mem by 64%! View Cross-Session Documentation →
- [01/20/2026] 📦 SimpleMem is now available on PyPI! Install via
pip install simplemem. View Package Usage Guide →
- [01/14/2026] 🎉 SimpleMem MCP Server is LIVE! Cloud-hosted at mcp.simplemem.cloud. View MCP Documentation →
- [01/05/2026] SimpleMem paper was released on arXiv!
📑 Table of Contents
🚀 Quick Start
🧠 Understanding the Basic Workflow
At a high level, SimpleMem works as a long-term memory system for LLM-based agents. The workflow consists of three simple steps:
- Store information – Dialogues or facts are processed and converted into structured, atomic memories.
- Index memory – Stored memories are organized using semantic embeddings and structured metadata.
- Retrieve relevant memory – When a query is made, SimpleMem retrieves the most relevant stored information based on meaning rather than keywords.
This design allows LLM agents to maintain context, recall past information efficiently, and avoid repeatedly processing redundant history.
🎓 Basic Usage
SimpleMem ships as a single simplemem package. The default mode="auto" automatically detects which backend to use based on what you call — no manual configuration needed:
from simplemem import SimpleMem
mem = SimpleMem() # mode="auto" — backend chosen by first call
The first method you call determines the backend:
| First call |
Backend selected |
Why |
add_dialogue() |
Text (SimpleMem) |
Dialogue-based API → text mode |
add_text() / add_image() / add_audio() / add_video() |
Omni (Omni-SimpleMem) |
Multimodal API → omni mode |
|
📝 Auto → Text (pure text input)
from simplemem import SimpleMem
mem = SimpleMem() # auto mode
# add_dialogue() → text backend auto-selected
mem.add_dialogue(
"Alice",
"Bob, let's meet at Starbucks tomorrow at 2pm",
"2025-11-15T14:30:00",
)
mem.add_dialogue(
"Bob",
"Sure, I'll bring the market analysis report",
"2025-11-15T14:31:00",
)
mem.finalize()
answer = mem.ask("When and where will Alice and Bob meet?")
# → "16 November 2025 at 2:00 PM at Starbucks"
|
🧠 Auto → Omni (multimodal input)
from simplemem import SimpleMem
mem = SimpleMem() # auto mode
# add_image() → omni backend auto-selected
mem.add_text(
"User loves hiking in the Rocky Mountains.",
tags=["session_id:D1"],
)
mem.add_image("photo.jpg", tags=["session_id:D1"])
mem.add_audio("voice_note.wav", tags=["session_id:D1"])
result = mem.query("What does the user enjoy?", top_k=5)
for item in result.items:
print(item["summary"])
mem.close()
|
💡 Tip: Auto mode picks the lightest backend that fits your data. You can still use mode="text" or mode="omni" explicitly if you prefer.
🧬 Advanced: Optimize Retrieval Config
Tune retrieval hyperparameters offline on your own dev set, then deploy the resulting Config for inference. This is a thin wrapper around EvolveMem's self-evolution loop:
import simplemem
from simplemem import SimpleMem, load_config
# mem is a finalized SimpleMem instance with memories already built
dev_questions = [
("When is the meeting?", "2pm tomorrow at Starbucks"),
("What should Bob prepare?", "market analysis report"),
]
config = simplemem.optimize(mem, dev_questions, max_rounds=3)
config.save("my_config.json")
# Later, deploy with the optimized config
config = load_config("my_config.json")
mem = SimpleMem(config=config)
EvolveMem runs an LLM-driven Evaluate → Diagnose → Propose → Guard cycle over your dev questions, adjusting global retrieval flags (top_k, fusion mode, answer verification, reflection rounds, ...). For the full standalone version with benchmark adapters and per-category overrides, see EvolveMem/.
🚄 Advanced: Parallel Processing
For large-scale dialogue processing, enable parallel mode:
from simplemem import create
mem = create(
mode="text",
clear_db=True,
enable_parallel_processing=True, # ⚡ Parallel memory building
max_parallel_workers=8,
enable_parallel_retrieval=True, # 🔍 Parallel query execution
max_retrieval_workers=4
)
💡 Pro Tip: Parallel processing significantly reduces latency for batch operations!
🌟 Overview
SimpleMem is a unified memory stack for LLM agents, built on one principle: store semantically lossless memory at high information density, so an agent recalls more while spending far fewer tokens. The package brings together three works that share this principle but attack different parts of the problem.
📝 SimpleMem: the efficiency core (text)
Most memory systems force a bad trade-off. They either passively accumulate raw interaction history (redundant, token-hungry) or run expensive reasoning loops to filter noise (slow, costly). SimpleMem instead compresses interactions through a three-stage pipeline:
| Stage |
What it does |
| 1. Semantic Structured Compression |
Distills unstructured interactions into compact memory units (self-contained facts with resolved coreferences and absolute timestamps), each indexed through multiple complementary views for flexible retrieval. |
| 2. Online Semantic Synthesis |
Merges related context within a session into unified abstract representations, removing redundancy as memory is built rather than at query time. |
| 3. Intent-Aware Retrieval Planning |
Infers the search intent behind a query to decide what to retrieve and assemble a precise, compact context. |
On the LoCoMo benchmark this delivers a 26.4% average F1 gain over prior systems while cutting inference-time token consumption by roughly 30x. Mechanism details (hybrid index layers, compression examples, retrieval planning): SimpleMem text memory →.
🧠 Omni-SimpleMem: multimodal memory (text, image, audio, video)
Omni-SimpleMem extends the compression-first philosophy to four modalities, built on three principles: Selective Ingestion (entropy-driven filtering per modality), Progressive Retrieval (hybrid FAISS + BM25 with pyramid token-budget expansion), and Knowledge Graph Augmentation (multi-hop cross-modal reasoning). Rather than being hand-designed, its architecture was discovered by an autonomous research pipeline that ran around 50 experiments across two benchmarks, diagnosing failure modes, proposing architectural changes, and even repairing data-pipeline bugs with no human in the inner loop. Tellingly, the bug fixes and architectural changes each contributed more than all hyperparameter tuning combined, taking the system from a naive baseline to state-of-the-art on both LoCoMo and Mem-Gallery. Full docs: Omni-SimpleMem →.
🧬 EvolveMem: self-evolving retrieval
EvolveMem closes a blind spot shared by almost every memory system: the stored content evolves, but the retrieval machinery (scoring functions, fusion strategies, answer-generation policies) stays frozen after deployment. EvolveMem runs a closed-loop AutoResearch process (Evaluate → Diagnose → Propose → Guard → Repeat) in which an LLM diagnoses per-question failures and proposes configuration changes, guarded by automatic rollback on regression and exploration incentives during stagnation. It discovers new retrieval dimensions (query decomposition, entity-swap, answer verification) not in the original design, improves LoCoMo by 25.7% relative over the strongest baseline, and its evolved configurations transfer positively across benchmarks. Full docs: EvolveMem →.
How they fit together
from simplemem import SimpleMem gives you the text core with automatic routing to the multimodal backend, and simplemem.optimize(...) taps EvolveMem to tune retrieval for your own data. One package, one mental model: compress losslessly, retrieve by intent, and let the system keep improving itself.
📦 Installation
📝 Notes for First-Time Users
- Ensure you are using Python 3.10+ in your active environment, not just installed globally.
- An OpenAI-compatible API key must be configured before running any memory construction or retrieval, otherwise initialization may fail.
- When using non-OpenAI providers (e.g., Qwen or Azure OpenAI), verify both the model name and
OPENAI_BASE_URL in config.py.
- For large dialogue datasets, enabling parallel processing can significantly reduce memory construction time.
📋 Requirements
- 🐍 Python 3.10+
- 🔑 OpenAI-compatible API (OpenAI, Qwen, Azure OpenAI, etc.)
🛠️ Setup
# 📥 Clone repository
git clone https://github.com/aiming-lab/SimpleMem.git
cd SimpleMem
# 📦 Install dependencies (pinned versions)
pip install -r requirements.txt
# — OR — install as an editable package
pip install -e . # default: text + multimodal + evolver
pip install -e ".[server]" # + MCP / HTTP server (mcp, fastapi, ...)
pip install -e ".[all]" # everything, including dev tools
# ⚙️ Configure API settings
cp config.py.example config.py
# Edit config.py with your API key and preferences
⚙️ Configuration Example
# config.py
OPENAI_API_KEY = "your-api-key"
OPENAI_BASE_URL = None # or custom endpoint for Qwen/Azure
LLM_MODEL = "gpt-4.1-mini"
EMBEDDING_MODEL = "Qwen/Qwen3-Embedding-0.6B" # State-of-the-art retrieval
Because SimpleMem talks to any OpenAI-compatible endpoint, you can point it at a managed provider such as Atlas Cloud by setting OPENAI_BASE_URL:
# config.py — using Atlas Cloud as an OpenAI-compatible backend
OPENAI_API_KEY = "your-atlascloud-api-key"
OPENAI_BASE_URL = "https://api.atlascloud.ai/v1"
LLM_MODEL = "deepseek-ai/deepseek-v4-pro" # a reasoning model — keep max tokens generous
EMBEDDING_MODEL = "Qwen/Qwen3-Embedding-0.6B"
deepseek-ai/deepseek-v4-pro is a reasoning model, so leave enough output budget; if responses come back empty, raise the token limit (>= 512). Any other OpenAI-compatible chat model on Atlas Cloud (e.g. Qwen/Qwen3-Next-80B-A3B-Instruct, zai-org/glm-5, moonshotai/kimi-k2.6) works the same way.
🐳 Run with Docker
The MCP Server can be run in Docker for a consistent, isolated environment. Data (LanceDB and user DB) is persisted in a host volume.
Prerequisites
Quick run
# From the repository root
docker compose up -d
Data is stored in ./data on the host (created automatically).
Custom configuration
- Copy the environment template and edit it:
cp .env.example .env
# Edit .env: set JWT_SECRET_KEY, ENCRYPTION_KEY, LLM_PROVIDER, model URLs, etc.
- Run with the env file:
docker compose --env-file .env up -d
Using Ollama on the host
When LLM_PROVIDER=ollama and Ollama runs on your machine (not in Docker), set in .env:
LLM_PROVIDER=ollama
OLLAMA_BASE_URL=http://host.docker.internal:11434/v1
On Linux, host.docker.internal is enabled automatically via the Compose file.
Using Requesty
Requesty is an OpenAI-compatible router. To use it, set in .env:
LLM_PROVIDER=requesty
REQUESTY_BASE_URL=https://router.requesty.ai/v1
Provide your Requesty API key (from app.requesty.ai/api-keys) when registering. Models use the same provider/model naming as OpenRouter (e.g. openai/gpt-4.1-mini). See the Requesty docs for available models.
Useful commands
docker compose logs -f simplemem # Follow logs
docker compose down # Stop and remove containers
📖 For self-hosting the MCP server (Docker or bare metal), see MCP Documentation.
🔌 MCP Server (text memory)
SimpleMem is available as a cloud-hosted memory service via the Model Context Protocol (MCP), enab
SimpleMem is a lifelong memory stack for LLM agents that stores, compresses, and retrieves long-term memories across text, image, audio, and video, so agents keep context without reprocessing redundant history.