picomq README

The project's README from github.com/PicoMQ/picomq. As one markdown file: README.md.

PicoMQ

PicoMQ is durable, real-time streams over HTTP and Kafka, built on S3-compatible object storage.

Documentation · Discord · Quick start

Install

cargo install --path picomq/pico-cli

This puts the pico binary in ~/.cargo/bin. Or run it in place with cargo run -p picomq-cli -- <args>. If another pico is first on PATH:

export PATH="$HOME/.cargo/bin:$PATH"

Run a node

# single node: SQLite metadata log, local object storage
pico serve \
    --meta-url sqlite:./data/meta.db \
    --storage=-2@file://./objects

Every flag has a PICO_* env equivalent. /health and /ready are on --admin-listen (default 127.0.0.1:9090). Auth is off by default, non-loopback binds need --auth required or --insecure-allow-remote.

Docker

Skips the install, everything runs in compose:

cd harness/aio
cp .env.example .env

docker compose up --build                          # Postgres + RustFS, 1 node
docker compose -f compose.cluster.yml up --build   # same stack, 2 nodes
docker compose -f compose.lite.yml up --build      # SQLite + file://, no deps
docker compose -f compose.lite.yml -f compose.connectors.yml up --build   # + connectors runtime

Pico: http://localhost:4437 (cluster also :4438). Dashboard: :9090. Connectors API: :8081. harness/byo is the same against an existing Postgres and object store, configured through .env.

Use it

pico create /streams/orders --content-type text/plain
seq 1 1000 | pico append /streams/orders --batch 100
pico read /streams/orders
pico tail /streams/orders -f
pico close /streams/orders && pico delete /streams/orders

pico --http2 bench -b 1024 -w 512 --connections 4 --streams 4 -d 60

Test

cargo test --workspace

# Postgres-backed tests, env-gated
PICOMQ_PG_URL=postgres://user:pass@localhost:5432/picomq \
    cargo test -p picomq-sql --test pg_contract --test pg_e2e

picomq (PicoMQ/picomq)

Searching Database…

373 ·GitHub
Picomq
What is this?PicoMQ is a durable, real-time streaming system that stores its data on S3-compatible object storage instead of local disks, while speaking both a native HTTP protocol and the Kafka wire protocol.
Is this a Kafka clone, or a proxy sitting in front of S3?It is a standalone server with its own metadata plane and protocol frontends (HTTP with the native Pico protocol and Durable Streams, plus a Kafka-compatible frontend), backed by a dedicated engine called s3stream that actually writes the log to object storage.
How do I get it running?Install the pico CLI with cargo (it lands the pico binary in ~/.cargo/bin), or run it in place with 'cargo run -p picomq-cli --'; alternatively skip the install entirely and bring up the whole stack with Docker Compose.
Do I need my own Postgres and S3 bucket to use it?The bundled Docker Compose setup provisions Postgres and object storage for you, but PicoMQ can also run 'bring your own' against an existing Postgres database and object store, configured through a .env file.
Is authentication turned on by default?No, auth is off by default. If you bind a node to anything other than loopback, you must explicitly pass --auth required or --insecure-allow-remote.
What ports does the Docker Compose setup expose?It exposes the Pico node (plus :4438 for cluster mode), a web dashboard on :9090, and a Connectors API on :8081.
How do I check if a node is healthy?Every node serves /health and /ready endpoints on its admin listener, which defaults to 127.0.0.1:9090.
Can I configure a node purely with environment variables?Yes, every CLI flag has a matching PICO_ environment variable, so you can configure nodes without passing any flags.
What are similar projects?kafgres, redpanda, automq, celld, nsq, redis, hydradb, emqx
Which topics is it in?message-broker, object-storage, message-queue, kafka, postgres, realtime, s3, distributed-systems, streaming, dashboard, rust, api
Creator
PicoMQ
Size
2.2 MB
Stars
373
Updated
CommitOct 09Releasev0.1.1Sep 06
Age
2mo
Commits · 90 Days