What is this?
Apache Spark is a unified analytics engine for large-scale data processing, offering high-level APIs in Scala, Java, Python, and R on top of an optimized execution engine.
Apache Spark is a unified analytics engine for large-scale data processing, offering high-level APIs in Scala, Java, Python, and R on top of an optimized execution engine.Is it a library I code against, or a service I run?It is an engine you build applications on top of, and it ships higher-level tools: Spark SQL for SQL and DataFrames, a pandas API for pandas workloads, MLlib, GraphX, and Structured Streaming.
It is an engine you build applications on top of, and it ships higher-level tools: Spark SQL for SQL and DataFrames, a pandas API for pandas workloads, MLlib, GraphX, and Structured Streaming.How do I see it actually do something?Get an official release and run a bundled example - the Pi example is the classic one; many examples print usage help if given no parameters.
Get an official release and run a bundled example - the Pi example is the classic one; many examples print usage help if given no parameters.Any gotchas when it talks to Hadoop storage?Yes - it uses the Hadoop core library to reach HDFS and other Hadoop-supported storage, and you must build Spark against the same Hadoop version your cluster runs.
Yes - it uses the Hadoop core library to reach HDFS and other Hadoop-supported storage, and you must build Spark against the same Hadoop version your cluster runs.Do I need a whole cluster for that, or will my laptop do?A laptop is enough - examples run locally with a single thread, or with local[N] threads.
A laptop is enough - examples run locally with a single thread, or with local[N] threads.And when I outgrow my laptop, how do I point it at a real cluster?Set the MASTER environment variable when running examples: a spark:// URL or 'yarn' submits the job to a cluster.
Set the MASTER environment variable when running examples: a spark:// URL or 'yarn' submits the job to a cluster.The R API is marked deprecated - should I start something new with it?The README lists R among the high-level APIs but flags it as '(Deprecated)', so new projects are better off with Scala, Java, or Python.
The README lists R among the high-level APIs but flags it as '(Deprecated)', so new projects are better off with Scala, Java, or Python.Can Spark run on Kubernetes?Yes - the repository includes a Kubernetes integration under resource-managers/kubernetes with its own integration-tests README, though the main README only brings it up in the testing context.
Yes - the repository includes a Kubernetes integration under resource-managers/kubernetes with its own integration-tests README, though the main README only brings it up in the testing context.Does it work with Hive?Yes - the build documentation covers building for particular Hive and Hive Thriftserver distributions, so Hive support depends on building Spark against the right distribution for your cluster.
Yes - the build documentation covers building for particular Hive and Hive Thriftserver distributions, so Hive support depends on building Spark against the right distribution for your cluster.