In the ETL system, when the data generation speed encounters large fluctuations, in order to make the existing resources meet the user's needs to a great extent under limited conditions, this paper proposed an ETL flexible scheduling mechanism based on resource urgency. First, it built a time series prediction model based on historical data to predict the speed of data generation. Then, according to the resource list each ETL process submits and the load of each server, it adjusted the resources of multiple ETL processes for a single user. This scheduling mechanism turned resource scheduling problem into a multi-dimensional knapsack problem, and proposed urgency based dynamic schedule algorithm. Experimental results show that when user's resources are scarce for all ETL processes, the proposed scheduling algorithm makes better use. The experiments verify that the flexible scheduling has advantages over the traditional scheduling method in resource utilization.

[2021-07-05] Printed Article /doc/2020.10.0385/v2