Alpha, for self-hosting and building on. Open-ended reasoning, live Google accounts, and CopilotKit Rich Threads require their own configuration. See what is verified and the roadmap.
Demo
On iPhone, ask OpenMuse to find interesting stories on Hacker News and summarize CopilotKit. On desktop, ask it to check the school-trip email, open the message, and research exhibits at Monterey Bay Aquarium. The agent shows email and browser results inline. Take control opens that same browser session when you need it.
The 38-second iPhone and 42-second desktop web demos show the current interface, framed in 16:9. The send arrow becomes a stop square inside the input pill while the agent replies, then switches back. Stopping keeps your draft intact. See the recording notes for the model setup and reproduction steps.
Mobile MP4 · Web MP4 · Recording details and reproduction
What it is
OpenMuse is a personal-agent application with an agent computer, visible work, and rich results. It runs its own server, task worker, and browser worker. You can inspect and change the source under the MIT license.
The computer combines persistent Chromium and an optional Linux workspace. The agent can browse public pages, run commands in its own container, work with files, and move PDFs between the computer and the app. You can open its browser or terminal and continue the work. Graphical desktops and autonomous checkout remain future work.
Features
| Surface |
What runs in this alpha |
| Chat |
CopilotKit headless chat with streamed AG-UI events, mailbox search and reading, send/stop in one input pill, a visible follow-up queue, retained drafts, delegated tasks, and inline email, browser, PDF, plan, and finance cards. |
| Agent computer |
Persistent browser profiles and takeover console; optional isolated Linux terminal, saved command receipts, editable workspace files, and PDF transfer. |
| Activity |
Durable task plans, progress, input requests, pause/resume/cancel/retry, approvals, and saved receipts. SQL leases recover interrupted work. |
| Ideas |
Suggestions with source evidence; edit, accept, or dismiss. Sent replies and completed matching work are excluded. |
| Goals & Tracking |
Goals and milestones; recurring public-page checks for changes, text availability, or USD price thresholds, with deduplicated alerts and failure backoff. |
| Documents |
Email attachment → PDF → requested form values → filled copy → reviewed reply → receipt. Native/web PDF viewing, paging, zoom, supported fields, and sharing. |
| Finance |
Import transaction CSV to create a spending summary with categories, transactions, and a savings-goal action. |
| Gmail & Calendar |
Google OAuth adapters, complete mail threads, drafts/attachments, calendar discovery, and reviewed event creation/update/deletion. Live credentials required. |
| Personal context |
Editable name, tone, avatar, and memories. Background-update preferences and durable in-app notifications. |
| Rich Threads |
CopilotKit Intelligence persistence in every mode, with a stable main conversation, side chats, renaming, archiving, restoring, and replay. A server-only project key is required. |
The feature inventory describes implemented capabilities and planned extensions. Health/bank/social connectors, device push, voice, generated executable tools, and automatic reservations/payments are on the roadmap.
Quick start
Requirements: Node 24 LTS, pnpm 11.19.0, and a CopilotKit Intelligence project key. The local sample app needs no model, Google account, or Docker.
git clone https://github.com/CopilotKit/OpenMuse.git openmuse
cd openmuse
pnpm install --frozen-lockfile
cp .env.example .env
npx copilotkit@latest login
npx copilotkit@latest project select
# Set CPK_INTELLIGENCE_API_KEY in .env to the generated server-only project key.
pnpm dev
In another terminal:
pnpm dev:web
Open localhost:8081. The API runs at localhost:8787/api/health.
Try it
- In Chat, send “Complete the permission slip”. Open the task, supply fictional form values, inspect the saved PDF, and review the prepared reply. This writes only to the local mailbox.
- In Goals → Track, create a built-in availability watch, then change the built-in test page to trigger an alert.
- In Menu → Delegate task → Finance, use Try example transactions to create an interactive spending tracker.
- Start the browser worker and configure a model, then ask “Check out Hacker News for cool stuff” or “Summarize copilotkit.ai”. Follow the browser inline and use Take control to open its session. For a model-free version of this flow, follow the AI Mock demo setup.
For iOS or Android, use pnpm --dir apps/mobile ios or pnpm --dir apps/mobile android. Xcode or Android tooling is required. The PDF reader needs an Expo development build; use native setup.
Copy the commented settings in .env.example into your private .env:
- Set
AGENT_BACKEND=model, MODEL=provider/model-id, and the matching provider key. CopilotKit supports the configured OpenAI, Anthropic or Google provider. Fictional data can still be used with a real model. Provider keys stay on the server.
- Create or select a CopilotKit Intelligence project with
npx copilotkit@latest login and npx copilotkit@latest project select. Keep the generated CPK_INTELLIGENCE_API_KEY on the server.
- For personal mail/calendar, set
WORKSPACE_MODE=live, the generated CPK_INTELLIGENCE_API_KEY, a random OPENMUSE_ACCESS_KEY of at least 24 characters, and TOKEN_ENCRYPTION_KEY containing 32 random bytes encoded as base64. Restart the API.
- Configure a Google OAuth web client with Gmail and Calendar APIs enabled. Set
GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET; register ${PUBLIC_API_URL}/api/google/callback as its redirect URI. Configure consent/test-user access in your Google project.
- Open Apps → Gmail (or Google Calendar), connect read access, and grant write access when needed. Every send or calendar change still requires its own stored review. Changing/disconnecting the account invalidates pending connection-bound work.
Google credentials are encrypted at rest. File URLs and browser consoles use short-lived signatures. This deployment uses one owner protected by a shared access key; it is not a multi-tenant authentication system. Use HTTPS and restricted network access for a remote host. Keep the default local-data mode on loopback.
Browser worker
Set BROWSER_WORKER_URL=http://127.0.0.1:8790 and a random WORKER_TOKEN of at least 32 characters in .env.
pnpm --dir apps/worker exec playwright install chromium
pnpm dev:browser
Or use docker compose --env-file .env -f infra/compose.yaml up --build -d. The same token must reach the API and worker. Sessions have persistent Chromium profiles; the app can open a live screenshot console and import PDF downloads. Agent tools can read public pages and hand interactive work to the person. Worker setup and boundaries.
Persistence and operation
Linux terminal and workspace
Build the computer image, enable it on the API, then open Computer → Terminal → Start computer:
docker build -t openmuse-computer:local apps/computer
COMPUTER_ENABLED=true pnpm dev
The API needs access to the Docker CLI and engine. Commands run in a nonroot container with no host-directory mounts or credentials. A named /workspace volume retains files when stopped. Terminal networking is disabled; public web access uses the browser worker. Commands have a 30-second limit and saved output/exit receipts. Files supports folders, text editing, and PDF transfer to/from Documents. This is a Linux container, not a full operating-system VM. Setup, Colima option, and boundaries.
Application storage
By default, embedded PGlite, documents and the signing key live in .openmuse/; browser profiles live in .openmuse/browser-profiles/. Keep that directory private and back it up. The API hosts the task worker. The host must remain running for background work.
For a separate task worker, configure the same DATABASE_URL, secrets and shared DATA_DIR for both processes, then set TASK_WORKER_ENABLED=false on the API and run pnpm dev:worker. PGlite cannot be opened by separate processes. Production commands are pnpm build:server, pnpm start and pnpm start:worker. Run one API instance; task workers coordinate through SQL leases.
No hidden retry occurs after an uncertain external write. Review its provider outcome before creating a replacement. Pausing/cancelling prevents subsequent task steps; an already approved in-flight provider request may finish.
CopilotKit Rich Threads
Every deployment requires CPK_INTELLIGENCE_API_KEY on the API server for CopilotKit Intelligence conversation persistence and replay. Create or select a project with npx copilotkit@latest login and npx copilotkit@latest project select, set the generated server-only key, and restart the API. The native menu uses useThreads; rich tool results link back to saved tasks, documents, and browser sessions.
Intelligence is a separate service and is not included in this repository's MIT license. No project key is shipped. Configuration and validation boundaries.
Architecture
flowchart TD
Client[Expo / React Native / Web] -->|AG-UI and authenticated API| API[Hono + CopilotKit runtime]
API --> Tasks[Durable task worker]
API --> Threads[CopilotKit Intelligence required in every mode]
API --> Store[(PGlite or PostgreSQL)]
Tasks --> Store
Tasks --> Review[Stored action review]
Review --> Google[Gmail / Calendar adapters]
Tasks --> Browser[Chromium worker + persistent profiles]
API --> Browser
API --> Computer[Optional Docker Linux computer]
Tasks --> Computer
Computer --> Volume[(Persistent workspace volume)]
Tasks --> Files[PDF files + structured artifacts]
API -. future adapter .-> OpenBot[OpenBot]
| Directory |
Purpose |
apps/mobile |
Shared iOS, Android, and web UI with CopilotKit headless hooks. |
apps/server |
API, CopilotKit runtime, identity boundary, task engine, reviews, files, and persistence. |
apps/worker |
Token-protected Playwright browser service with persistent profiles. |
apps/computer |
Nonroot Linux image, bounded filesystem helper, and real container verification. |
packages/domain |
Shared types and request validation. |
packages/integrations |
Google and browser protocol adapters. |
packages/backends |
Optional OpenBot HTTP adapter and its identity boundary. |
tests |
Workflow, runtime, persistence, provider-contract, and authorization tests. |
OpenBot compatibility
OpenMuse's native client and personal-agent workflows are independent of OpenBot. The disabled OpenBot adapter is pinned and contract-tested against upstream interfaces. Live user/session bridging, routine mapping, and computer backend wiring remain future work. OpenBot's Intelligence runtime is not a raw AG-UI endpoint. Integration contract.
Development
pnpm lint
pnpm typecheck
pnpm test
pnpm build:server
pnpm build:web
pnpm build:ios
pnpm build:android
pnpm --dir apps/worker typecheck
pnpm test:browser
pnpm test:computer
Platform build scripts export JavaScript/Hermes bundles; they do not produce signed app binaries. Browser checks require installed Chromium and public fixture access. CI also exercises the browser and Linux computer containers. See contribution guidance and verification results.
Contributing and license
Issues and pull requests are welcome. Start with CONTRIBUTING.md, ROADMAP.md, and the security policy.
MIT licensed. Built by CopilotKit. Its original interface and fictional assets are included. Website, email, and document content supplies evidence, not permission to act.
OpenMuse 是一款个人智能体应用：一个拥有自己的浏览器、终端、文件和可持续后台工作的 AI 智能体。你提出想要的结果，跟着计划走，审核它的操作，然后回来收取成果。