Topics / #hadoop

Hadoop projects on GitHub

Distributed storage and batch processing built around the Hadoop ecosystem.

As of , the most-starred Hadoop projects on GitHub are spark (44.2K stars), data-science-ipython-notebooks (29.4K stars) and cat (18.9K stars). GitHub.Locker tracks 14 Hadoop repositories with 260.8K stars combined. The fastest-growing on October 16 was spark, with +10 stars in a day.

14Projects
260.8KStars in total
+25Stars in a day
Browse in the catalog →github.com/topics/hadoop ↗

Hot right now

Most stars gained in a day · Oct 16

+10

spark

apache/spark · ★ 44.2K

Apache Spark is a unified analytics engine for large-scale data processing, offering high-level APIs in Scala, Java, Python, and R on top of an optimized execution engine.

+5

trino

trinodb/trino · ★ 13.3K

Trino is a fast distributed SQL query engine for big data analytics — you run it as a server (or cluster) and query large datasets with standard SQL. It was formerly known as PrestoSQL.

+2

DataX

alibaba/DataX · ★ 17.4K

DataX is an offline data synchronization tool/platform from Alibaba Cloud, open-sourced from the DataWorks Data Integration product, used widely inside Alibaba to move data between heterogeneous data sources.

+2

BigData-Notes

heibaiying/BigData-Notes · ★ 17K

This is a comprehensive Chinese-language study guide covering the big data ecosystem: Hadoop, Hive, Spark, Storm, Flink, HBase, Kafka, Zookeeper, Flume, Sqoop, Azkaban, and Scala.

The all-time top

14 projects · by stars

  1. #1 · apache/spark

    spark

    Apache Spark is a unified analytics engine for large-scale data processing, offering high-level APIs in Scala, Java, Python, and R on top of an optimized execution engine.

    44.2K
  2. #2 · donnemartin/data-science-ipython-notebooks

    data-science-ipython-notebooks

    A large curated collection of IPython/Jupyter notebooks covering the Python data science stack: deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, statistics with SciPy, pandas, matplotlib, NumPy, Kaggle competition analyses, big data (Spark, Hadoop MapReduce, HDFS), AWS, and command-line essentials.

    29.4K
  3. #3 · dianping/cat

    cat

    CAT is an open-source real-time application monitoring platform (APM) written in Java at Meituan-Dianping. It collects performance metrics, health status and alerts from your services and turns them into live reports.

    18.9K
  1. 4

    luigi

    18.8Kspotify/luigi · Python

    Luigi is a Python package for building complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, failure handling and command line integration.

  2. 5

    APIJSON

    18.4KAPIJSON/APIJSON · Java

    APIJSON is a JSON-based communication protocol and ORM library that auto-provides CRUD APIs and documentation with no backend coding, while clients define exactly what JSON data and structure they get back.

  3. 6

    DataX

    17.4Kalibaba/DataX · Java

    DataX is an offline data synchronization tool/platform from Alibaba Cloud, open-sourced from the DataWorks Data Integration product, used widely inside Alibaba to move data between heterogeneous data sources.

  4. 7

    BigData-Notes

    17Kheibaiying/BigData-Notes · Java

    This is a comprehensive Chinese-language study guide covering the big data ecosystem: Hadoop, Hive, Spark, Storm, Flink, HBase, Kafka, Zookeeper, Flume, Sqoop, Azkaban, and Scala.

  5. 8

    presto

    16.8Kprestodb/presto · Java

    Presto is a distributed SQL query engine for running interactive analytic queries against big data sources.

  6. 9

    hadoop

    15.7Kapache/hadoop · Java

    Apache Hadoop is the open-source big data platform for reliable, distributed storage and processing of very large data sets across clusters of machines.

  7. 10

    deeplearning4j

    14.3Kdeeplearning4j/deeplearning4j · Java

    Eclipse Deeplearning4J is a suite of JVM libraries for building, training and deploying deep learning applications, usable from Java, Scala, Kotlin and Clojure.

  8. 11

    fluentd

    13.6Kfluent/fluentd · Ruby

    Fluentd is an open-source log collector that sits at the center of your logging setup: it gathers events from many data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and more.

  9. 12

    trino

    13.3Ktrinodb/trino · Java

    Trino is a fast distributed SQL query engine for big data analytics — you run it as a server (or cluster) and query large datasets with standard SQL. It was formerly known as PrestoSQL.

  10. 13

    predictionio

    12.5Kapache/predictionio · Scala

    Apache PredictionIO is an open source machine learning server that lets developers and data scientists collect events, deploy algorithms, evaluate them, and query predictive results through REST APIs.

  11. 14

    God-Of-BigData

    10.6Kwangzhiwubigdata/God-Of-BigData

    A curated Chinese-language knowledge base for learning big data engineering and passing big data interviews, covering Java fundamentals, distributed systems theory, and deep dives into Hadoop, Hive, Spark, Flink, HBase, Kafka, and Zookeeper.

Often tagged together