Spark Tuning Guide
- Charles Bishop (Deactivated)
This tuning guide provides examples showing specific recommendations for tuning for different Hadoop cluster configurations and workloads. Use the guide to adjust your settings based on your cluster and workload characteristics. For a more in-depth explanation of these settings, refer to the Spark Tuning Concepts guide.
Caveats
This guide assumes the YARN cluster has been properly configured for running with more than 1 vcore per YARN container. By default, YARN ignores vcores when performing resource calculations and only takes into account memory usage. Datameer still tracks the amount of vcores Spark is actually using, but YARN reports the each Spark Executor uses 1 vcore. This method might lead to problems on the cluster if other applications or systems are running jobs that require exact vcore allocation or if the cluster has been configured to use CGroups. If exact vcore allocation is required then the DominantResourceCalculator should be enabled on the cluster's ResourceManager, since the DefaultResourceCalculator only takes into account memory resources. In most cases, all the cluster jobs run without problems and use as many CPU(s) as configured, but YARN doesn't report it. Here is an example configuration for the DominantResourceCalculator. During Datameer's testing of Spark we ran with both the DefaultResourceCalculator and DominantResourceCalculator, and in both cases the Spark jobs ran exactly the same.
Small Cluster
Example environment
This environment is an example, and does not cover all possible environments. Make sure you are adjusting your settings based on your particular environment, not adapting these settings as a one-to-one recommendation.
Our example environment uses the following setup:
One master node
Four worker nodes with:
20 GB memory
Four vcores
Custom properties
The following custom properties are set:
das.execution-framework=SparkClient das.spark.context.auto-scale-enabled=false das.map-tasks-per-node=4 das.spark.context.max-idle-time=10m spark.executor.cores=2 spark.executor.memory=9g spark.executor.instances=7
Use this table to understand the settings.
Property | Reason |
|---|---|
| With this configuration's small cluster, SparkSX is unnecessary. |
| Auto scaling is unnecessary because other applications don't need free resources. |
| The map tasks per node should be set to the available vcores of a node on the cluster. |
| Increasing the idle time leads fewer startup costs. |
| Use two vcores per container. |
| (Memory per node / max number of instances per node) * 0.9 E.g., ( 20 / 2 ) * 0.9 = 9g The memory per node is 20GB, divided by the two instances per node specified, multiplied by 90%, giving the final value of 9g to set for the property |
| Four nodes multiplied by the space for two instances per node, minus one AM instance. |
In this case SparkClient should be used as the execution framework, and it should use most of the cluster all of the time which leaves some container slots available for AM(s) and other processes.
Cluster with Mixed Workloads
Example environment
This environment is an example, and does not cover all possible environments. Make sure you are adjusting your settings based on your particular environment, not adapting these settings as a one-to-one recommendation.
Our example environment uses the following setup:
One master node
10 worker nodes with:
56GB memory
12 vcores
Mix of small and larger workloads
Custom properties
The following custom properties are set:
das.execution-framework=SparkSX spark.executor.cores=2 spark.executor.memory=8g das.map-tasks-per-node=12 datameer.yarn.available-node-memory=56g datameer.yarn.available-node-vcores=12
Use this table to understand the settings.
Property | Reason |
|---|---|
| SparkSX is the best option for clusters with mixed workloads. |
| For Tez the map tasks per node should be the number of available vcores per node. |
| Up to six instances per node. 12 vcores per node/2 instances per node= 6 vcores per instance. Due to auto-scaling, you don't need to leave a vcore for the AM instance, as auto-scaling uses one instance fewer to account for the AM. |
| 56g per node/6 instances per node = 9g per instance. The 9g value includes 10% overhead, so the setting should be 8g to account for that necessary overhead. |
| Available memory on each cluster node. |
| Available vcores on each cluster node. |
This configuration allows each SparkCluster job to process up to 60 tasks concurrently. The configuration also leaves open some memory and vcore resources for AM containers and other processes.
Cluster with Mixed Workloads (v6.3 and above)
The following settings are based on version 6.3 of Datameer.
Example environment
This environment is an example, and does not cover all possible environments. Make sure you are adjusting your settings based on your particular environment, not adapting these settings as a one-to-one recommendation.
Our example environment uses the following setup:
1 master node
10 worker nodes with:
56GB memory
12 vcores
Mix of small and larger workloads
Custom Properties
The following custom properties are set:
das.execution-framework=SparkSX spark.executor.cores=2 spark.executor.memory=auto das.yarn.available-node-memory=auto das.yarn.available-node-vcores=auto
Use this table to understand the settings.
Property | Reason |
|---|---|
| SparkSX is the best option for clusters with mixed workloads |
| Up to 6 instances per node 12 vcores per node/2 instances per node= 6 vcores per instance Due to auto-scaling, you don't need to leave a vcore for the AM instance, as auto-scaling uses one instance fewer to account for the AM |
| Automatically calculates the amount of memory required to perform tasks |
| Automatically calculates available memory on each cluster mode |
| Automatically calculates available vcores on each cluster mode |
This configuration allows each SparkCluster job to process up to 60 tasks concurrently. The configuration also leaves open some memory and vcore resources for AM containers and other processes.