AI Town 🏠💻💌
Live Demo
Join our community Discord: AI Stack Devs
AI Town is a virtual town where AI characters live, chat and socialize.
This project is a deployable starter kit for easily building and customizing your own version of AI
town. Inspired by the research paper
Generative Agents: Interactive Simulacra of Human Behavior.
The primary goal of this project, beyond just being a lot of fun to work on, is to provide a
platform with a strong foundation that is meant to be extended. The back-end natively supports
shared global state, transactions, and a simulation engine and should be suitable from everything
from a simple project to play around with to a scalable, multi-player game. A secondary goal is to
make a JS/TS framework available as most simulators in this space (including the original paper
above) are written in Python.
Overview
Stack
- Game engine, database, and vector search: Convex
- Auth (Optional): Clerk
- Default chat model is
llama3 and embeddings with mxbai-embed-large.
- Local inference: Ollama
- Configurable for other cloud LLMs: Together.ai or anything that speaks the
OpenAI API. PRs welcome to add more cloud provider support.
- Background Music Generation: Replicate using
MusicGen
Other credits:
Installation
The overall steps are:
- Build and deploy
- Connect it to an LLM
Build and Deploy
There are a few ways to run the app on top of Convex (the backend).
- The standard Convex setup, where you develop locally or in the cloud. This requires a Convex
account(free). This is the easiest way to depoy it to the cloud and seriously develop.
- If you want to try it out without an account and you're okay with Docker, the Docker Compose
setup is nice and self-contained.
- There's a community fork of this project offering a one-click install on
Pinokio for
anyone interested in running but not modifying it 😎.
- You can also deploy it to Fly.io. See ./fly for instructions.
Standard Setup
Note, if you're on Windows, see below.
git clone https://github.com/a16z-infra/ai-town.git
cd ai-town
npm install
This will require logging into your Convex account, if you haven't already.
To run it:
npm run dev
You can now visit http://localhost:5173.
If you'd rather run the frontend and backend separately (which syncs your backend functions as
they're saved), you can run these in two terminals:
npm run dev:frontend
npm run dev:backend
See package.json for details.
Using Docker Compose with self-hosted Convex
You can also run the Convex backend with the self-hosted Docker container. Here we'll set it up to
run the frontend, backend, and dashboard all via docker compose.
docker compose up --build -d
The container will keep running in the background if you pass -d. After you've done it once, you
can stop and start services.
To log into the dashboard and deploy from the convex CLI, you will need to generate an admin key.
docker compose exec backend ./generate_admin_key.sh
Add it to your .env.local file. Note: If you run down and up, you'll have to generate the key
again and update the .env.local file.
# in .env.local
CONVEX_SELF_HOSTED_ADMIN_KEY="<admin-key>" # Ensure there are quotes around it
CONVEX_SELF_HOSTED_URL="http://127.0.0.1:3210"
Then set up the Convex backend (one time):
npm run predev
To continuously deploy new code to the backend and print logs:
npm run dev:backend
To see the dashboard, visit http://localhost:6791 and provide the admin key you generated earlier.
Configuring Docker for Ollama
If you'll be using Ollama for local inference, you'll need to configure Docker to connect to it.
npx convex env set OLLAMA_HOST http://host.docker.internal:11434
To test the connection (after you have it running):
docker compose exec backend /bin/bash curl http://host.docker.internal:11434
If it says "Ollama is running", it's good! Otherwise, check out the
Troubleshooting section.
Connect an LLM
Note: If you want to run the backend in the cloud, you can either use a cloud-based LLM API, like
OpenAI or Together.ai or you can proxy the traffic from the cloud to your local Ollama. See
below for instructions.
Ollama (default)
By default, the app tries to use Ollama to run it entirely locally.
- Download and install Ollama.
- Open the app or run
ollama serve in a terminal. ollama serve will warn you if the app is
already running.
- Run
ollama pull llama3 to have it download llama3.
- Test it out with
ollama run llama3.
Ollama model options can be found here.
If you want to customize which model to use, adjust convex/util/llm.ts or set
npx convex env set OLLAMA_MODEL # model. If you want to edit the embedding model:
- Change the
OLLAMA_EMBEDDING_DIMENSION in convex/util/llm.ts and ensure:
export const EMBEDDING_DIMENSION = OLLAMA_EMBEDDING_DIMENSION;
- Set
npx convex env set OLLAMA_EMBEDDING_MODEL # model.
Note: You might want to set NUM_MEMORIES_TO_SEARCH to 1 in constants.ts, to reduce the size of
conversation prompts, if you see slowness.
OpenAI
To use OpenAI, you need to:
// In convex/util/llm.ts change the following line:
export const EMBEDDING_DIMENSION = OPENAI_EMBEDDING_DIMENSION;
Set the OPENAI_API_KEY environment variable. Visit https://platform.openai.com/account/api-keys if
you don't have one.
npx convex env set OPENAI_API_KEY 'your-key'
Optional: choose models with OPENAI_CHAT_MODEL and OPENAI_EMBEDDING_MODEL.
Together.ai
To use Together.ai, you need to:
// In convex/util/llm.ts change the following line:
export const EMBEDDING_DIMENSION = TOGETHER_EMBEDDING_DIMENSION;
Set the TOGETHER_API_KEY environment variable. Visit https://api.together.xyz/settings/api-keys if
you don't have one.
npx convex env set TOGETHER_API_KEY 'your-key'
Optional: choose models via TOGETHER_CHAT_MODEL, TOGETHER_EMBEDDING_MODEL. The embedding model's
dimension must match EMBEDDING_DIMENSION.
Other OpenAI-compatible API
You can use any OpenAI-compatible API, such as Anthropic, Groq, or Azure.
- Change the
EMBEDDING_DIMENSION in convex/util/llm.ts to match the dimension of your embedding
model.
- Edit
getLLMConfig in llm.ts or set environment variables:
npx convex env set LLM_API_URL 'your-url'
npx convex env set LLM_API_KEY 'your-key'
npx convex env set LLM_MODEL 'your-chat-model'
npx convex env set LLM_EMBEDDING_MODEL 'your-embedding-model'
Note: if LLM_API_KEY is not required, don't set it.
Note on changing the LLM provider or embedding model:
If you change the LLM provider or embedding model, you should delete your data and start over. The
embeddings used for memory are based on the embedding model you choose, and the dimension of the
vector database must match the embedding model's dimension. See
below for how to do that.
Customize your own simulation
NOTE: every time you change character data, you should re-run npx convex run testing:wipeAllTables
and then npm run dev to re-upload everything to Convex. This is because character data is sent to
Convex on the initial load. However, beware that npx convex run testing:wipeAllTables WILL wipe
all of your data.
Create your own characters and stories: All characters and stories, as well as their spritesheet
references are stored in characters.ts. You can start by changing
character descriptions.
Updating spritesheets: in data/characters.ts, you will see this code:
export const characters = [
{
name: 'f1',
textureUrl: '/assets/32x32folk.png',
spritesheetData: f1SpritesheetData,
speed: 0.1,
},
...
];
You should find a sprite sheet for your character, and define sprite motion / assets in the
corresponding file (in the above example, f1SpritesheetData was defined in f1.ts)
Update the Background (Environment): The map gets loaded in convex/init.ts from
data/gentle.js. To update the map, follow these steps:
- Use Tiled to export tilemaps as a JSON file (2 layers named
bgtiles and objmap)
- Use the
convertMap.js script to convert the JSON to a format that the engine can use.
node data/convertMap.js <mapDataPath> <assetPath> <tilesetpxw> <tilesetpxh>
<mapDataPath>: Path to the Tiled JSON file.<assetPath>: Path to tileset images.<tilesetpxw>: Tileset width in pixels.<tilesetpxh>: Tileset height in pixels. Generates converted-map.js that you can use like
gentle.js
Adding background music with Replicate (Optional)
For Daily background music generation, create a Replicate account and
create a token in your Profile's API Token page.
npx convex env set REPLICATE_API_TOKEN # token
This only works if you can receive the webhook from Replicate. If it's running in the normal
Convex cloud, it will work by default. If you're self-hosting, you'll need to configure it to hit
your app's url on /http. If you're using Docker Compose, it will be http://localhost:3211,
but you'll need to proxy the traffic to your local machine.
Note: The simulation will pause after 5 minutes if the window is idle. Loading the page will
unpause it. You can also manually freeze & unfreeze the world with a button in the UI. If you
want to run the world without the browser, you can comment-out the "stop inactive worlds" cron in
convex/crons.ts.
- Change the background music by modifying the prompt in
convex/music.ts
- Change how often to generate new music at
convex/crons.ts by modifying the
generate new background music job
Commands to run / test / debug
To stop the back end, in case of too much activity
This will stop running the engine and agents. You can still run queries and run functions to debug.
npx convex run testing:stop
To restart the back end after stopping it
npx convex run testing:resume
To kick the engine in case the game engine or agents aren't running
npx convex run testing:kick
To archive the world
If you'd like to reset the world and start from scratch, you can archive the current world:
npx convex run testing:archive
Then, you can still look at the world's data in the dashboard, but the engine and agents will no
longer run.
You can then create a fresh world with init.
npx convex run init
To pause your backend deployment
You can go to the dashboard to your deployment settings to pause and
un-pause your deployment. This will stop all functions, whether invoked from the client, scheduled,
or as a cron job. See this as a last resort, as there are gentler ways of stopping above.
Windows Installation
Prerequisites
- Windows 10/11 with WSL2 installed
- Internet connection
Steps:
Install WSL2
First, you need to install WSL2. Follow
this guide to set up WSL2 on your Windows
machine. We recommend using Ubuntu as your Linux distribution.
Update Packages
Open your WSL terminal (Ubuntu) and update your packages:
sudo apt update
Install NVM and Node.js
NVM (Node Version Manager) helps manage multiple versions of Node.js. Install NVM and Node.js 18
(the stable version):
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.2/install.sh | bash
export NVM_DIR="$([ -z "${XDG_CONFIG_HOME-}" ] && printf %s "${HOME}/.nvm" || printf %s "${XDG_CONFIG_HOME}/nvm")"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
source ~/.bashrc
nvm install 18
nvm use 18
Install Python and Pip
Python is required for some dependencies. Install Python and Pip:
sudo apt-get install python3 python3-pip sudo ln -s /usr/bin/python3 /usr/bin/python
At this point, you can follow the instructions above.
Deploy the app to production
Deploy Convex functions to prod environment
Before you can run the app, you will need to make sure the Convex functions are deployed to its
production environment. Note: this is assuming you're using the default Convex cloud product.
- Run
npx convex deploy to deploy the convex functions to production
- Run
npx convex run init --prod
To transfer your local data to the cloud, you can run npx convex export and then import it with
npx convex import --prod.
If you have existing data you want to clear, you can run
npx convex run testing:wipeAllTables --prod
Adding Auth (Optional)
You can add clerk auth back in with git revert b44a436. Or just look at that diff for what changed
to remove it.
Make a Clerk account
- Go to https://dashboard.clerk.com/ and click on "Add Application"
- Name your application and select the sign-in providers you would like to offer users
- Create Application
- Add
VITE_CLERK_PUBLISHABLE_KEY and CLERK_SECRET_KEY to .env.local
VITE_CLERK_PUBLISHABLE_KEY=pk_***
CLERK_SECRET_KEY=sk_***
- Go to JWT Templates and create a new Convex Template.
- Copy the JWKS endpoint URL for use below.
npx convex env set CLERK_ISSUER_URL # e.g. https://your-issuer-url.clerk.accounts.dev/
Deploy the frontend to Vercel
- Register an account on Vercel and then install the Vercel CLI.
- If you are using Github Codespaces: You will need to
install the Vercel CLI and authenticate from your codespaces cli by
running
vercel login.
- Deploy the app to Vercel with
vercel --prod.
Using local inference from a cloud deployment
We support using Ollama for conversation generations. To have
it accessible from the web, you can use Tunnelmole or Ngrok or similar so the cloud backend can send
requests to Ollama running on your local machine.
Steps:
- Set up either Tunnelmole or Ngrok.
- Add Ollama endpoint to Convex
npx convex env set OLLAMA_HOST # your tunnelmole/ngrok unique url from the previous step
- Update Ollama domains Ollama has a list of accepted domains. Add the ngrok domain so it won't
reject traffic. see ollama.ai for more details.
Using Tunnelmole
Tunnelmole is an open source tunneling tool.
You can install Tunnelmole using one of the following options:
- NPM:
npm install -g tunnelmole
- Linux:
curl -s https://tunnelmole.com/sh/install-linux.sh | sudo bash
- Mac:
curl -s https://tunnelmole.com/sh/install-mac.sh --output install-mac.sh && sudo bash install-mac.sh
- Windows: Install with NPM, or if you don't have NodeJS installed, download the
exe file for
Windows here and put it somewhere in your PATH.
Once Tunnelmole is installed, run the following command:
tmole 11434
Tunnelmole should output a unique url once you run this command.
Using Ngrok
Ngrok is a popular closed source tunneling tool.
Once ngrok is installed and authenticated, run the following command:
ngrok http http://localhost:11434
Ngrok should output a unique url once you run this command.
Troubleshooting
Wiping the database and starting over
You can wipe the database by running:
npx convex run testing:wipeAllTables
Then reset with:
npx convex run init
Incompatible Node.js versions
If you encounter a node version error on the convex server upon application startup, please use node
version 18, which is the most stable. One way to do this is by
installing nvm and running nvm install 18 and
nvm use 18.
Reaching Ollama
If you're having trouble with the backend communicating with Ollama, it depends on your setup how to
debug:
- If you're running directly on Windows, see
Windows Ollama connection issues.
- If you're using Docker, see
Docker to Ollama connection issues.
- If you're running locally, you can try the following:
npx convex env set OLLAMA_HOST http://localhost:11434
By default, the host is set to http://127.0.0.1:11434. Some systems prefer localhost
¯_(ツ)_/¯.
Windows Ollama connection issues
If the above didn't work after following the windows and regular
installation instructions, you can try the following, assuming you're not using
Docker.
If you're using Docker, see the next section for Docker
troubleshooting.
For running directly on Windows, you can try the following:
Install unzip and socat:
sudo apt install unzip socat
Configure socat to Bridge Ports for Ollama
Run the following command to bridge ports:
socat TCP-LISTEN:11434,fork TCP:$(cat /etc/resolv.conf | grep nameserver | awk '{print $2}'):11434 &
Test if it's working:
curl http://127.0.0.1:11434
If it responds OK, the Ollama API should be accessible.
Docker to Ollama connection issues
If you're having trouble with the backend communicating with Ollama, there's a couple things to
check:
- Is Doc
AI Town 是一个采用 MIT 许可证的入门套件，用来搭建一个虚拟小镇，让 AI 角色在其中生活、聊天，并彼此交往。