If you appreciate the content 📖, support projects visibility, give 👍| ⭐| 👏
A curated list of awesome System Designing articles, videos and resources for distributed computing, AKA Big Data.
Whether you're preparing for an interview or you want to design a distributed/microservice oriented application, this list will definitely help you achieve that.
Attention: Stars on GitHub does not reflect usage or popularity for every item here listed.
Inspired By Awesome-BigData
Started By Gabriel Leon de Mattos
Contents
Articles
Books
Videos
Tools
Bonus
Articles
Introduction / Interviews
Advanced
Books
Designing Distributed Systems: Patterns and Paradigms for Scalable, Reliable Services - [Paid 💵] - Book that talks about disitributed systems as well as lightly demonstrating some code of what it looks like.
Designing Data Intensive Applications - [Paid 💵] - Goes in depth to explain various resources we use when working with distributed systems, as well as how it came to be and what problems it aims to solve.
The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
Building Microservices - [Free 👍] - Awesome book that talks about designing sytem architecture with microservices in depth, includes most relevant topics in this regard.
Monolith to Microservices - [Free 👍] - Written by the same author as the one above, this book will cover the migration from Monolith to Microservices, it's recommended you start by the previous book.
Distributed Systems (3rd Edition) - [Free 👍] - Great overview of and in-depth introduction to distributed systems. Recommended for intermediate level readers.
Safe by Design - [Free 👍] - Great book on enforcing code correctness by design.
Videos
A collection of videos based on distributed systems.
Introduction / Interviews
Advanced
- A collection of most commonly used tools for distributed systems
Relational Database Management System
MariaDB - MariaDB is a fork of MySQL server.
MySQL - Widely used relational database.
PostgresSQL - Relational database that has been gaining popularity.
SQLite - Another widely used database that is built into all mobile phones and most computers.
Sql Server - Widely used relational database.
NoSQL
Cache (Key-Value)
Apache Ignite - [3.3k ⭐] - In memory caching with ACID properties.
Couchbase - Inspired by memcached, adding features such as replication and persistance.
Oracle Coherence - [126 ⭐] - High scaling, low latency in-memory caching.
Memcached - [10.2k ⭐] - One of the first in-memory caching database, high performing and multi-threaded.
Redis - [44k ⭐] - Widely used in-memory caching database with many added features such as persistent storage and supporting strings, lists, sets, hashses, streams, bitmaps, etc.
Store (Key-Value)
Apple FoundationDB - [10k ⭐] - Multi-model (many data types in a single database), ACID key-value store. Easily scalable and fault tolerant.
Cosmos DB - Microsoft's globally distributed, multi-model database service. Eastically and independently scale throughput and storage. SQL, MongoDB, Cassandra, Tables, Gremlin, and Spark APIs.
Document Store
CouchDB - [4.6k ⭐] - ACID compliant NoSQL document-store DB, provides a RESTful HTTP API for reading and updating database documents.
MongoDB - One of the most popular 'NoSQL' database for general purpose.
RethinkDB - [23.8k ⭐] - Document-store DB.
ElasticSearch - [49.9k ⭐] - Widely popular 'NoSQL' database for fast and scalable search engines.
Cosmos DB - Microsoft's globally distributed, multi-model database service. Eastically and independently scale throughput and storage. SQL, MongoDB, Cassandra, Tables, Gremlin, and Spark APIs.
Wide Column Store
Amazon DynamoDB - Key-Value and Document database, highly performant, scalable and secure.
Google Bigtable - Scalable and performant 'NoSQL' database for large analytical and operational workload.
Cassandra - Facebook-born project very fast, easily scalable, with option to include consistency with each operation.
Scylla - [4.9k ⭐] - 'NoSQL' data store using seastar framework, compatible with Cassandra.
HBase - [3.6k ⭐] - Modeled after Google's Bigtable and written in Java. Developed as a part of Apache Hadoop project and runs on top of HDFS or Alluxio. (See Hadoop Related)
Cosmos DB - Microsoft's globally distributed, multi-model database service. Eastically and independently scale throughput and storage. SQL, MongoDB, Cassandra, Tables, Gremlin, and Spark APIs.
Graph
Amazon Neptune - Fast, reliable and fully managed graph database service.
ArangoDB - [10k ⭐] - Flexible database for documents, key-value, graphs. Uses its own query language, AQL.
Neo4j - [7.9k ⭐] - Good support for a graph db, ACID compliant and flexible.
Cosmos DB - Microsoft's globally distributed, multi-model database service. Eastically and independently scale throughput and storage. SQL, MongoDB, Cassandra, Tables, Gremlin, and Spark APIs.
Distributed File Systems
HDFS - Hadoop File System is a a widely popular choice among its big data competitors, providing high throughput access.
Lustre - File system for computer clusters.
CephFS - Unified, distributed storage system.
GlusterFS - Scale-out NAS file system.
MooseFS - POSIX-compliant distributed file system.
XtreemFS - Fault tolerant file system.
Resource Management
- Kubernetes - Highly popular way to deploy, manage and automatically scale a cluster of containers on bare-metal or virtual servers.
Stream Processing
Apache Samza - Build stateful applications that process data in real time from multiple sources, including Kafka. Easy and inexpensive multi-subscriber model, can eliminate backpressure and has reliable persistency with low latency.
Apache Flink - Based on the concept of streams and transofrmations. Uses maven, handles batch tasks as data streams with finite boundaries. Low latency, high throughput.
Amazon Kinesis Streams - Durable, scalable, real-tme service. Collects gigabytes of data per second from hundreds of thousands of sources, including database event streams, website clickstreams, financial transactions, etc.
Azure Stream Analytics - Real-time analytics service that is designed for mission-critical workloads.
Message Broker
Amazon MQ - Open source message broker from Amazon.
Apache ActiveMQ - It's a multi-protocol, java based messaging server.
Apache Kafka - Widely popular message broker with low latency for data streaming.
RabbitMQ - Widely popular lightweight
message broker written in erlang that also supports multiple messaging protocols.
IronMQ - Very fast and highly scalable messaging broker. (not open source)
Apache Pulsar - Created by yahoo, also highly scalable, low latency, geo-replication and multi-tenacy.
Kestrel - Written in Scala and speaks the memcached protocol. It works much like Kafka.
Azure Service Bus - A fully managed enterprise integration message broker.
Load Balancers
Open Source Software
SeeSaw - [5.1k ⭐] - Used by Google, developed in Go, linux-based virtual load balancer server.
HAProxy - Widely popular option, provides high-availability, proxy, TCP/HTTP load balancing. Used by Reddit, Imgur, MaxCDN, GitHub, AirBNB.
Zevenet - Supports L3, L4 and L7. Easy install with a docker repo. Supports advanced health-check monitorining.
Neutrino - Used by eBay, built with Scala and Netty. Supports round-robin and least-connection algorithms.
Nginx - Wait, isn't Nginx a web server? Yes, the open source does support basic level of content switching and request routing. Plus edition supports load balancing, WAF, monitoring, etc.
Openresty - Nginx + Lua, perfect combination.
Hardware
F5 - Robust hardware load balancer option, supporting multiple protocols (IP, TCP, FTP, UDP, HTTP).
TP-Link - Cheaper alternative that works as a load balancer.
Barracuda - One of the top choices for load balancing when it comes to in-house servers. Top security measures built in, comprehensive reports and monitoring outbound traffic for data loss prevention.
Cloud
Amazon Elastic Load Balancing - Popular choice for amazon customers, supports lambda functions, highly scalable.
Google Load Balancing - Popular choice for google customers, comes with auto-scaling feature, very fast, has intergrated CDN.
Cloudflare Load Balancing - Scalable load balancing by Cloudflare, feature fast failover and a dashboard.
DigitalOcean Load Balancing - If you're a digitalocean customer, this is a good option, very cheap, regional availability, scalable, easy to deploy among your other droplets.
Azure Load Balancing - Popular choice for Microsoft's Azure customers. Supports internal and external traffics, ipv6, monitorining and the standard load balancing set of features.
Hadoop Ecosystem
Dashboard
- Ambari - Dashboard that integrates most of hadoop related technologies for easy management and executions.
Data Ingestion
Sqoop - Efficiently transfer data between Hadoop and structured datastores such as relational databases.
Flume - Distributed, highly available and efficient in collecting, aggregating and moving large amounts of log data.
Apache Kafka - Widely popular message broker with low latency for data streaming.
Workflow Scheduler
- Oozie - Create workflows in xml to execute jobs (from other hadoop-ecosystem applications) in steps, allows for parallel execution as well.
Query
- Hive - Query hadoop stored data in SQL.
- Pig - Scriping language that looks like SQL to query hadoop data.
Processing
这是一个精选列表，收录了学习系统设计（分布式系统）的文章、书籍、视频和工具，既适合面试准备，也适合设计真实的分布式或微服务应用。