Apache Spark Tuning Configurations Tutorial

Here are some pyspark configuration items that can be used for tuning spark jobs.

1. Memory Management


        spark.executor.memory = 4g

        spark.driver.memory = 2g

        spark.memory.offHeap.enabled = true


        spark.executor.cores = 4

        spark.default.parallelism = 8