Apache Spark™ FAQ
How does Spark relate to Apache Hadoop?
Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.
Who is using Spark in production?
As of 2016, surveys show that more than 1000 organizations are using Spark in production. Some of them are listed on the Powered By page and at the Spark Summit.
How large a cluster can Spark scale to?
Many organizations run Spark on clusters of thousands of nodes. The largest cluster we know has 8000 of them. In terms of data size, Spark has been shown to work well up to petabytes. It has been used to sort 100 TB of data 3X faster than Hadoop MapReduce on 1/10th of the machines, winning the 2014 Daytona GraySort Benchmark, as well as to sort 1 PB. Several production workloads use Spark to do ETL and data analysis on PBs of data.
Does my data need to fit in memory to use Spark?
No. Spark's operators spill data to disk if it does not fit in memory, allowing it to run well on any sized data. Likewise, cached datasets that do not fit in memory are either spilled to disk or recomputed on the fly when needed, as determined by the RDD's storage level.
How can I run Spark on a cluster?
You can use either the standalone deploy mode, which only needs Java to be installed on each node, or the Mesos and YARN cluster managers. If you'd like to run on Amazon EC2, AMPLab provides EC2 scripts to automatically launch a cluster.
Note that you can also run Spark locally (possibly on multiple cores) without any special setup by just passing local[N] as the master URL, where N is the number of parallel threads you want.
Do I need Hadoop to run Spark?
No, but if you run on a cluster, you will need some form of shared file system (for example, NFS mounted at the same path on each node). If you have this type of filesystem, you can just deploy Spark in standalone mode.
Does Spark require modified versions of Scala or Python?
No. Spark requires no changes to Scala or compiler plugins. The Python API uses the standard CPython implementation, and can call into existing C libraries for Python such as NumPy.
What’s the difference between Spark Streaming and Spark Structured Streaming? What should I use?
Spark Streaming is the previous generation of Spark’s streaming engine. There are no longer updates to Spark Streaming and it’s a legacy project. Spark Streaming provides a high-level abstraction called discretized stream or DStream, which represents a continuous stream of data. Internally, a DStream is represented as a sequence of RDDs.
Spark Structured Streaming is the current generation of Spark’s streaming engine, which is richer in functionality, easier to use, and more scalable. Spark Structured Streaming is built on top of the Spark SQL engine and enables you to express streaming computation the same way you express a batch computation on static data.
You should use Spark Structured Streaming for building streaming applications and pipelines with Spark. If you have legacy applications and pipelines built on Spark Streaming, you should migrate them to Spark Structured Streaming.
Where can I find high-resolution versions of the Spark logo?
We provide versions here: black logo, white logo. Please be aware that Spark, Apache Spark and the Spark logo are trademarks of the Apache Software Foundation, and follow the Foundation's trademark policy in all uses of these logos.
Can I provide commercial software or services based on Spark?
Yes, as long as you respect the Apache Software Foundation'ssoftware licenseand trademark policy.In particular, note that there are strong restrictions about how third-party productsuse the "Spark" name (names based on Spark are generally not allowed).Please also refer to ourtrademark policy summary.
How can I contribute to Spark?
See the Contributing to Spark wiki for more information.
Where can I get more help?
Please post on StackOverflow's apache-spark tag or Spark Users mailing list. For more information, please refer to Have Questions?. We'll be glad to help!
