Tag: Spark

Spark SQL

Spark SQL is a Spark module for structured data processing. Unlike the basic Spark RDD API, Spark SQL provides interfaces that give Spark more information about the data structure and the computations being performed. Internally, Spark SQL uses this extra information to perform additional optimizations. There are multiple ways to interact with Spark SQL, including SQL and the Dataset API. When computing results, the same execution engine is used regardless of the API/language you use to express the computation.

Ruochen ChenMarch 26, 2025About 2 min

Spark Core

RDD (Resilient Distributed Dataset)

Can be understood as a data pipeline.

    public static void main(String[] args) {

        SparkConf conf = new SparkConf();
        conf.setAppName("Spark01");
        conf.setMaster("local");
        JavaSparkContext sc = new JavaSparkContext(conf);


        // read from memory
        List<String> list = Arrays.asList("alice,bob");
        JavaRDD<String> rdd = sc.parallelize(list,3); //3 partitions\

        // read from disk
        // JavaRDD<String> rdd = sc.textFile("/Users/chenruochen/Documents/IT-Study/Spark/spark/data/1.txt");


        List<String> collect = rdd.collect();
        collect.forEach(System.out::println);

        rdd.saveAsTextFile("output");// save as text file in relatived path

        sc.close();
    }

Ruochen ChenMarch 19, 2025About 5 min

Spark Intro

Differences between Spark and Hadoop MapReduce

Feature	Apache Spark	Apache MapReduce
Processing Model	In-memory processing, DAG-based	Disk-based, Map and Reduce operations
Performance	Faster due to in-memory computation	Slower, as data is read/written to disk after each operation
Ease of Use	High-level APIs (Java, Scala, Python, R)	Low-level APIs (Java)
Fault Tolerance	Uses lineage for fault tolerance (re-computation)	Replication of data blocks for fault tolerance
Data Processing	Supports batch and real-time processing (streaming)	Primarily batch processing
Memory Consumption	High, due to in-memory processing	Lower, because it relies on disk storage
Data Shuffling	More efficient due to DAG and in-memory storage	Less efficient, as it involves disk I/O
APIs and Libraries	Rich libraries (MLlib, GraphX, SparkSQL, etc.)	Limited libraries, mostly for basic processing
Latency	Low latency due to in-memory processing	High latency due to disk I/O
Fault Tolerance Mechanism	Lineage-based re-computation	Data replication at HDFS level
Resource Management	Built-in support for YARN, Mesos, and Kubernetes	Requires Hadoop YARN or similar systems
Scalability	Easily scalable, supports large-scale clusters	Scalable but with higher overhead due to disk-based operations
Suitability	Suitable for both batch and real-time workloads	Best for batch processing workloads
Popularity	Gaining widespread adoption for big data processing	Mature but less popular for newer workloads

Ruochen ChenMarch 18, 2025About 1 min