Advanced Spark Configuration

SparkClient

das.execution-framework=SparkClient

Runs Spark in YARN-client mode

SparkCluster

das.execution-framework=SparkCluster

Runs Spark in YARN-cluster mode

SparkSX

das.execution-framework=SparkSX

Runs Spark Smart Execution

spark.executor.instances

1

The number of executors if dynamic resource allocation is disabled

For dynamic mode, refer to auto-scaling

spark.executor.cores

1

The number of cores allocated to each executor

spark.executor.memory

1g, auto in v6.3

Amount of memory allocated per executor process (e.g. 512m, 2g, 8g)

Heap size settings can be set with spark.executor.memory

Auto allocates the correct amount of memory based on your settings.

spark.yarn.executor.memoryOverhead

executorMemory * 0.10, with minimum of 384

The amount of off heap memory (in megabytes) allocated per executor. This memory that accounts for things such as VM overheads, interned strings, and other native overheads. This tends to grow with the executor size (typically 6-10%).

spark.serializer

org.apache.spark.serializer.KryoSerializer

Class to use for serializing objects that are sent over the network or need to be cached in serialized form. The default of Java serialization works with any serializable Java object but is quite slow, so Datameer recommends using org.apache.spark.serializer.KryoSerializer and configuring Kryo serialization when speed is necessary. This can be any subclass of org.apache.spark.Serializer .

spark.executor.extraJavaOptions

A string of extra JVM options to pass to executors, such as GC settings or other logging. It is illegal to set Spark properties or heap size settings using this option. Spark properties should be set using a SparkConf object or the spark-defaults.conf file used with the spark-submit script. Heap size settings can be set with spark.executor.memory.

spark.driver.extraJavaOptions

A string of extra JVM options to pass to the driver, such as GC settings or other logging. 
Note:  In client mode, this config must not be set through the  SparkConf  directly in your application, because the driver JVM has already started at that point. Instead, set this through the  --driver-java-options  command line option or in your default properties file.

spark.yarn.am.extraJavaOptions

A string of extra JVM options to pass to the YARN Application Master in client mode. In cluster mode, use  spark.driver.extraJavaOptions  instead.

spark.driver.extraClassPath

Extra classpath entries to prepend to the classpath of the driver. 
Note:  In client mode, this configuration must not be set through the  SparkConf  directly in your application, because the driver JVM has already started at that point. Instead, set this through the  --driver-class-path  command line option or in your default properties file.

 Can be used to specify the classpath for Snappy libraries.

spark.executor.extraClassPath

Extra classpath entries to prepend to the classpath of executors. This ensures backwards-compatibility with older versions of Spark. Users typically should not need to set this option.

Can be used to specify the classpath for HBase libraries.

spark.driver.extraClassPath

Extra classpath entries to prepend to the classpath of the driver. 
Note:  In client mode, this configuration must not be set through the  SparkConf  directly in your application, because the driver JVM has already started at that point. Instead, set this through the  --driver-class-path  command line option or in your default properties file.

spark.yarn.am.extraLibraryPath

Set a special library path to use when launching the YARN Application Master in client mode. For example, this propertycan be used to set a path to the cluster's native libraries.

spark.executor.extraLibraryPath

Set a special library path to use when launching executor JVMs. For example, this property can be used to set a path tothe cluster's native libraries.

spark.driver.extraLibraryPath

Set a special library path to use when launching the driver JVM. For example, this propertycan be used to set a path to the cluster's native libraries.
Note :  In client mode, this config must not be set through the  SparkConf  directly in your application, because the driver JVM has already started at that point. Instead, set this through the  --driver-library-path  command line option or in your default properties file.

das.spark.launcher.spark-submit-opts

-XX:MaxPermSize=48m -Xmx16m

JVM options for the spark-submit process launched in the SparkCluster framework.

Enables overriding JVM options for the spark-submit process launched in the SparkCluster framework.

das.spark.launcher.app.status.poll.interval

10s

Spark Launcher polls application status, the interval between app status fetching can be set with this property.

das.spark.context.max-idle-time

100ms

 Max idle time to keep the SparkClient application running.

The Spark context times out after this amount of time remaining idle.

das.splitting.map-wave-count

6

The count to influence the number of map tasks DAS uses for internal and for file-based import jobs. Determines how many times a single Hadoop job should use the available map slots.

das.map-tasks-per-node

10

Used as a foundation for calculating the optimal number of splits based on the number of physical nodes.

spark.ui.port

4040

Port for your application's dashboard, which shows memory and workload data

spark.driver.port

0

Port for the driver to listen on, which is used for communicating with the executors and the standalone master

spark.replClassServer.port

0

Port for the driver's HTTP class server to listen on, which is only relevant for the Spark shell

spark.executor.port

0

Port for the executor to listen on, which is used for communicating with the driver

spark.fileserver.port

0

Port for the driver's HTTP file server to listen on

spark.broadcast.port

0

Port for the driver's HTTP broadcast server to listen on, which is not relevant for torrent broadcast

spark.blockManager.port

0

Port for all block managers to listen on, which exists on both the driver and the executors

spark.yarn.am.port

0

Port for the YARN application ,aster to listen on. In YARN client mode, this is used to communicate between the Spark driver running on a gateway and the YARN application master running on YARN. In YARN cluster mode, this is used for the dynamic executor feature, where it handles the kill from the scheduler backend.

spark.port.maxRetries

16

Maximum number of retries when binding to a port before giving up.
When a port is given a specific value (non-0), each subsequent retry increments the port used in the previous attempt by 1 before retrying.
This allows it to try a range of ports from the start port specified to port + maxRetries.

spark.io.compression.codec

lzf

Used to compress internal data such as RDD partitions, broadcast variables, and shuffle outputs. By default, Spark provides three codecs: lz4, lzf, and snappy. You can also use fully qualified class names to specify the codec, e.g. org.apache.spark.io.LZ4CompressionCodec, org.apache.spark.io.LZFCompressionCodec, and org.apache.spark.io.SnappyCompressionCodec.

spark.io.compression.lz4.blockSize

32k

Block size used in LZ4 compression. Lowering this block siz also lowers shuffle memory usage when LZ4 is used.

spark.io.compression.snappy.blockSize

32k

Block size used in Snappy compression. Lowering this block size also lowers shuffle memory usage when Snappy is used.

spark.rdd.compress

false

Determines whether to compress serialized RDD partitions (e.g. for StorageLevel.MEMORY_ONLY_SER). Can save substantial space at the cost of extra CPU time.

spark.broadcast.compress

true

Determines whether to compress broadcast variables before sending them, which is recommended.

spark.shuffle.compress

true

Determines whether to compress map output files, which is recommended. Compression uses spark.io.compression.codec.

spark.shuffle.spill.compress

true

Determines whether to compress data spilled during shuffles. Compression uses spark.io.compression.codec.

das.spark.context.auto-scale-enabled

true

Whether Datameer should dynamically scale up or down Spark containers. This property should typically be turned on in order to free up unneeded cluster resources.

When enabled, the Spark context auto-scales executors based on load.

datameer.yarn.available-node-vcores

das.yarn.available-node-vcores in v6.3

8, auto in v6.3

The amount of cluster cores allocated for Spark auto-scaling.

datameer.yarn.available-node-memory

das.yarn.available-node-memory in v6.3

8g, auto in v6.3

The amount of cluster memory allocated for Spark auto-scaling.
Auto-scaling also makes sure that the overhead is included into the calculation. 

spark.executor.instances

4

Minimum number of Spark executors to kept up per Datameer SparkClient application. These are also kept up when all executors are idle. One Spark executor is equivalent to one YARN container.

das.spark.context.max-executors

-1

Maximum number of Spark executors to use per Spark job. One Spark executor is equivalent to one YARN container.

The Spark context allocates executors up to this amount when required. < 0 means no max

spark.eventLog.enabled

false

Whether to log Spark events. This property is useful for reconstructing the web UI after the application has finished.

spark.eventLog.dir

file:///tmp/spark-events

Base directory in which Spark events are logged, if  spark.eventLog.enabled  is set to true. Within this base directory, Spark creates a sub-directory for each application and logs the events specific to the application in this directory. Users might want to set this to a unified location like an HDFS directory so history files can be read by the history server.

das.sparksx.small.max-uncompressed-size

10g

Max data input size threshold for small data sets in SparkSX

das.sparksx.small.execution-framework

SparkClient

SparkSX framework for small data sets

das.sparksx.medium.max-uncompressed-size

100g

Max data input size threshold for medium data sets in SparkSX

das.sparksx.medium.execution-framework

SparkCluster

SparkSX framework for medium data sets

das.sparksx.large.execution-framework

Tez

SparkSX framework for large data sets

das.sparksx.unknown.execution-framework

SparkCluster

SparkSX default framework for cases when data input is an unknown size

spark.yarn.am.cores

1

Number of cores allocated for the YARN Application Master

spark.yarn.am.memory

2g

Amount of memory allocated for the YARN Application Master

spark.yarn.am.memoryOverhead

AM memory * 0.10, with minimum of 384

The amount of off heap memory (in megabytes) allocated for YARN Application Master  

spark.driver.cores

1

Number of cores allocated for the driver process

spark.driver.memory

2g

Amount of memory allocated for the driver process

spark.yarn.driver.memoryOverhead