2. BestConfig for Spark¶
2.1. Experimental Settings¶
We executed Bestconfig for the spark cluster with 4 nodes. The spark cluster consists of 1 master node and 3 slave nodes. All nodes used in our experiment are shown below.
| Node | OS | CPU | Memory |
|---|---|---|---|
| Master | CentOS | 16 Intel(R) Xeon(R) CPU E5620 @ 2.40GHz | 32GB |
| Slave 1 | CentOS | 16 Intel(R) Xeon(R) CPU E5620 @ 2.40GHz | 32GB |
| Slave 2 | CentOS | 16 Intel(R) Xeon(R) CPU E5620 @ 2.40GHz | 32GB |
| Slave 3 | CentOS | 16 Intel(R) Xeon(R) CPU E5620 @ 2.40GHz | 32GB |
2.2. Performance Surface¶
We use HiBench that is a widely adopted benchmark tools in the workload generator for Spark to generate the target workload. Figure 1 plot the highly differed performance surfaces for Spark Pagerank workload.
The performance surface of Spark under Hibench-Pagerank workload
2.3. Test Results¶
The test result of Spark pagerank workload pagerank. The test result of Spark kmeans workload kmeans.
2.4. Interface Impl¶
The source files of SparkConfigReadin and SparkConfigWrite implement the interfaces of ConfigReadin and ConfigWrite respectively.