automq README

The project's README from github.com/AutoMQ/automq. As one markdown file: README.md.

Diskless Kafka® on S3, Offering 10x Cost Savings and Scaling in Seconds.

📑  Documentation    🔥  Explore AutoMQ Playground Now   

Linkedin Badge Gurubase DeepWiki

AutoMQ%2Fautomq | Trendshift

automq-solgan
Grab LG U+ Tencent Honda JD
Bambu Lab miHoYo Hungry Studio FunPlus Avia Games

Prerequisites

Before running AutoMQ locally, please ensure:

[!Tip] Deploying a production-ready AutoMQ cluster is challenging. This Quick Start is only for evaluating AutoMQ features and is not suitable for production use. For production deployment best practices, please contact our community for support.

The docker/docker-compose.yaml file provides a simple single-node setup for quick evaluation and development:

curl -O https://raw.githubusercontent.com/AutoMQ/automq/refs/tags/1.5.5/docker/docker-compose.yaml && docker compose -f docker-compose.yaml up -d

This setup features a single AutoMQ node serving as both controller and broker, alongside MinIO for S3 storage. All services operate within a Docker bridge network called automq_net, allowing you to start a Kafka producer in this network to test AutoMQ:

docker run --network automq_net automqinc/automq:latest /bin/bash -c \
"/opt/automq/kafka/bin/kafka-producer-perf-test.sh --topic test-topic --num-records=1024000 --throughput 5120 --record-size 1024 \
--producer-props bootstrap.servers=server1:9092 linger.ms=100 batch.size=524288 buffer.memory=134217728 max.request.size=67108864"

After testing, you can destroy the setup with:

docker compose -f docker-compose.yaml down

The docker/docker-compose-cluster.yaml file offers a more complex setup with three AutoMQ nodes, ideal for testing AutoMQ's cluster features, and can be run in the same way.

There are more deployment options available:

🗞️ Newest Feature - Table Topic

Table Topic is a new feature in AutoMQ that combines stream and table functionalities to unify streaming and data analysis. Currently, it supports Apache Iceberg and integrates with catalog services such as AWS Glue, HMS, and the Rest catalog. Additionally, it natively supports S3 tables, a new AWS product announced at the 2024 re:Invent. Learn more.

image

🔶 Why AutoMQ

AutoMQ is a stateless Kafka alternative that runs on S3 or any S3-compatible storage, such as MinIO. It is designed to address two major issues of Apache Kafka. First, Kafka clusters are difficult to scale out or in due to the stateful nature of its brokers. Data movement is required, and even reassigning partitions between brokers is a complex process. Second, hosting Kafka in the cloud can be prohibitively expensive. You face high costs for EBS storage, cross-AZ traffic, and significant over-provisioning due to Kafka's limited scalability.

Here are some key highlights of AutoMQ that make it an ideal choice to replace your Apache Kafka cluster, whether in the cloud or on-premise, as long as you have S3-compatible storage:

✨Architecture

AutoMQ is a fork of the open-source Apache Kafka. We've introduced a new storage engine based on object storage, transforming the classic shared-nothing architecture into a shared storage architecture.

image

Regarding the architecture of AutoMQ, it is fundamentally different from Kafka. The core difference lies in the storage layer of Apache Kafka and how we leverage object storage to achieve a stateless broker architecture. AutoMQ consists of below key components:

For more on AutoMQ's architecture, visit AutoMQ Architecture or explore the source code directly.

🌟 Stay Ahead

Star AutoMQ on GitHub for instant updates on new releases. star-automq

💬 Community

You can join the following groups or channels to discuss or ask questions about AutoMQ:

👥 How to contribute

If you've found a problem with AutoMQ, please open a GitHub Issues. To contribute to AutoMQ please see Code of Conduct and Contributing Guide. We have a list of good first issues that help you to get started, gain experience, and get familiar with our contribution process.

👍 AutoMQ Enterprise Edition

The enterprise edition of AutoMQ offers a robust, user-friendly control plane for seamless cluster management, with enhanced availability and observability over the open-source version. Additionally, we offer Kafka Linking for zero-downtime migration from any Kafka-compatible cluster to AutoMQ.

Contact us for more information about the AutoMQ enterprise edition, and we'll gladly assist with your free trial.

📜 License

AutoMQ is under the Apache 2.0 license. See the LICENSE file for details.

📝 Trademarks

Apache®, Apache Kafka®, Kafka®, Apache Iceberg®, Iceberg® and associated open source project names are trademarks of the Apache Software Foundation

automq (AutoMQ/automq)

scalability
Searching Database…

10.9K ·GitHub
Automq
What is this?AutoMQ is a diskless, stateless fork of Apache Kafka that stores all log data on S3 or S3-compatible object storage instead of local disks, cutting cloud costs by up to 10x and removing cross-AZ traffic fees.
Is this a whole new message queue or just a tweaked Kafka?It is a fork of open-source Apache Kafka with the storage layer replaced by an S3-based engine, so it stays 100% wire-compatible with existing Kafka clients and tooling.
How do I try AutoMQ locally?Run the bundled docker/docker-compose.yaml for a single-node cluster backed by MinIO as S3 storage, or docker-compose-cluster.yaml for a 3-node setup; both start with a normal docker compose up/down.
What do I need installed before running it?Docker 20.x or later, Docker Compose v2, at least 4GB of RAM allocated to Docker, and free ports 9092 and 9000.
Is the Docker Compose quick start ready for production?No — the maintainers explicitly say this quick start is only for evaluating features, not for production; real deployments need their production best-practice guides or community/enterprise support.
What is Table Topic?Table Topic lets AutoMQ topics also act as Apache Iceberg tables, unifying streaming and analytics with support for AWS Glue, HMS, and REST catalogs, plus native S3 tables.
How does it avoid AWS cross-AZ traffic fees?A rack-aware router hands clients in each availability zone specific partition metadata so they exchange data through object storage instead of replicating across AZs, which is normally where most Kafka cloud costs come from.
What exactly changed compared to stock Apache Kafka?AutoMQ swaps local-disk log storage for an S3-based engine (S3Stream, WAL, LogCache/BlockCache) and adds a built-in Auto Balancer, turning Kafka's stateful shared-nothing brokers into stateless, S3-backed shared storage.
Can I deploy it on Kubernetes or just Docker?Besides the Docker Compose quick start, the README links to guides for deploying a multi-node cluster on plain Linux, on Kubernetes, or via AWS/Alibaba Cloud Marketplace trials.
What are similar projects?emqx, kestra, thanos, full-stack-data-engineering, doris, picomq, VictoriaMetrics, centrifugo
Which topics is it in?cost-optimization, object-storage, high-availability, kafka, scalability, prometheus, s3, serverless, aws, enterprise, distributed-systems, streaming
Creator
AutoMQ
Size
170.6 MB
Stars
10.9K
Updated
CommitOct 15Release1.7.5-rc1Sep 24
Age
3y
Commits · 90 Days