Īpache Spark requires a cluster manager and a distributed storage system. Īmong the class of iterative algorithms are the training algorithms for machine learning systems, which formed the initial impetus for developing Apache Spark. The latency of such applications may be reduced by several orders of magnitude compared to Apache Hadoop MapReduce implementation. Spark facilitates the implementation of both iterative algorithms, which visit their data set multiple times in a loop, and interactive/exploratory data analysis, i.e., the repeated database-style querying of data. Nodes represent RDDs while edges represent the operations on the RDDs. Inside Apache Spark the workflow is managed as a directed acyclic graph (DAG). Spark's RDDs function as a working set for distributed programs that offers a (deliberately) restricted form of distributed shared memory. Spark and its RDDs were developed in 2012 in response to limitations in the MapReduce cluster computing paradigm, which forces a particular linear dataflow structure on distributed programs: MapReduce programs read input data from disk, map a function across the data, reduce the results of the map, and store reduction results on disk.

The RDD technology still underlies the Dataset API. In Spark 1.x, the RDD was the primary application programming interface (API), but as of Spark 2.x use of the Dataset API is encouraged even though the RDD API is not deprecated. The Dataframe API was released as an abstraction on top of the RDD, followed by the Dataset API. Apache Spark has its architectural foundation in the resilient distributed dataset (RDD), a read-only multiset of data items distributed over a cluster of machines, that is maintained in a fault-tolerant way.

Spark 1.0 Chess Engine