Self-learning dynamic adjustment scheduling algorithm based on Hadoop

DOI

10.2991/icmmita-15.2015.62How to use a DOI?

Keywords

Hadoop; Scheduling algorithm; Speculative execution; Slow tasks

Abstract

Speculative execution is the key technology to improve the execution efficiency of Hadoop cluster. In the large-scale cluster reasonable Speculative execution can effectively reduce the execution time for the job. At present, there is still a big problem in the detecting of the Hadoop's Speculative execution .Therefore this paper proposes a self-learning dynamic adjustment scheduling algorithm. The algorithm firstly study the impact of the historical information of the task execution, dynamically adjusts the time proportions of each stage of Reduce tasks And taking into account the tasks of the different types of load effect on the detection of slow tasks .Found out the real impact of the job execution time of the slow tasks to avoid performing unnecessary backup tasks. Through the experiment, the average execution time of SLDA scheduling is compared with LATE, and the average execution time of the job is significantly reduced.

Copyright

© 2015, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Fucong Li
AU  - Zhuyu Li
AU  - Guohui Chen
AU  - Xiangxin Li
PY  - 2015/11
DA  - 2015/11
TI  - Self-learning dynamic adjustment scheduling algorithm based on Hadoop
BT  - Proceedings of the 2015 3rd International Conference on Machinery, Materials and Information Technology Applications
PB  - Atlantis Press
SP  - 314
EP  - 318
SN  - 2352-538X
UR  - https://doi.org/10.2991/icmmita-15.2015.62
DO  - 10.2991/icmmita-15.2015.62
ID  - Li2015/11
ER  -