Focused on the problem of unbalanced computing load distribution and low resource utilization in the native scheduling mechanism of big data streaming computing platform, this paper proposed a load balancing strategy based on tabu search algorithm in heterogeneous environments and applied to the Apache Flink platform. Firstly, this strategy set up a job topology model and abstracted the topology of streaming computing jobs as a directed acyclic graph. Therefore, each task slot became a node, which established the foundation for performance evaluation of computing nodes. Secondly, the method imported the performance evaluation model to nodes with performance weights in the directed acyclic graph, and obtained the performance of the nodes through normalization processing. Then the evaluation parameters were passed into the tabu search for job path optimization, so as to obtain the optimal job path. Finally, by using the CustomPatitionerWrapper interface, this strategy allocated data to the nodes included in the optimal job path and completed the balancing of computational load. The algorithm then passed evaluation parameters into the tabu scheduling algorithm for job path optimization, thereby obtaining the optimal job path. The experimental results show that the load balancing strategy optimized by the tabu scheduling algorithm reduces the average computing latency by 10~20 ms compared to the native Flink platform. The strategy significantly improves resource utilization, and increases average throughput by about 15%. This effectively proves the effectiveness and optimization effect of the load balancing strategy.

[2023-07-13] Accepted Paper /doc/2023.04.0180/v1

[2023-12-05] Printed Article /doc/2023.04.0180/v2