What exactly is Apache Spark? | Big Data Tools

Overview

Apache Spark is a fast, general-purpose framework for processing big data across clusters. Developed at Berkeley Lab in 2009, it was open-sourced in 2010 and later became an Apache top-level project. Spark was designed in part to address limitations of Hadoop’s MapReduce approach.

Core Abstraction: RDDs

Spark’s core abstraction is the Resilient Distributed Dataset (RDD), a read-only collection of objects distributed across a cluster. RDDs can be created from sources such as text files, SQL and NoSQL databases, HDFS, and cloud storage.

RDDs support operations such as MapReduce-style processing, joins, filtering, and aggregation, with processing performed in memory.

Execution Architecture

• The driver starts the application and creates a Spark Context to coordinate work.
• The Spark Context builds a physical plan and uses a cluster manager to arrange execution.
• A DAG (directed acyclic graph) scheduler determines task dependencies and execution order.
• Executors run tasks on worker nodes and return results to the driver.

Spark Libraries

• Spark SQL works with structured data and DataFrames, using SQL or languages such as Java, Python, and Scala.
• Spark Streaming processes incoming data in small batches.
• MLlib supports distributed machine learning.
• GraphX supports distributed graph processing.

Deployment and Ecosystem

Spark can run with cluster systems such as Hadoop YARN, Apache Mesos, Kubernetes, and Docker Swarm. Managed options include Amazon EMR, Google Cloud Dataproc, Azure HDInsight, and Databricks.

A common architecture uses HDFS for storage and Spark for processing and data interaction.

Advantages and Trade-Offs

The video highlights Spark’s speed, flexible language support, near-real-time stream processing, and support for advanced analytics. Compared with MapReduce, Spark is presented as better suited to real-time processing, filtering, and iterative workloads.

Its main drawback is its high RAM requirement for in-memory processing, though the video notes that many users consider the benefits worth the memory cost.

Content Type and Source

Content type: An educational overview of Spark’s architecture, components, deployment options, and trade-offs. It does not provide a hands-on tutorial or product review.

Main speaker/source: A narrator from the nullQueries channel.