A Highly-Efficient Fault Tolerance Method for a Scalable Stream Processing System
- DOI
- 10.2991/caai-17.2017.50How to use a DOI?
- Keywords
- SPS; dynamic scale out; fault tolerance
- Abstract
An effective fault-tolerant mechanism becomes essential to the credible stream processing system (SPS). However, the traditional introduction of fault-tolerant mechanisms has a relatively negative influence on the systemic calculation efficiency. And enhancing the system scale out is an effective way to solve this problem. This paper proposes a highly-efficient fault recovery method that is for an extensible SPS. On the one hand, based on a reasonable partition and upstream backup of its internal calculation status. The SPS can monitor the bottleneck state of the operators and then migrate the upstream backup state of the operators that is in need of scale out to the new nodes to achieve dynamic scale out of the SPS; On the other hand, when a node fault in the SPS, the system can dynamically expand a secondary node for the erroneous one, and then use the upstream backup algorithm to achieve fault recovery. In this paper, a mathematical model is built for the SPS. And based on this, the fault recovery time and computational efficiency of the rapid fault tolerant method are analyzed. Experimental results have shown that the one proposed in this paper is more effective in fault tolerance which gives the system a higher calculation efficiency by reducing the frequency of upstream backup of node state.
- Copyright
- © 2017, the Authors. Published by Atlantis Press.
- Open Access
- This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).
Cite this article
TY - CONF AU - Guanghui Chang AU - Peizhen Li AU - Guangxia Xu PY - 2017/06 DA - 2017/06 TI - A Highly-Efficient Fault Tolerance Method for a Scalable Stream Processing System BT - Proceedings of the 2017 2nd International Conference on Control, Automation and Artificial Intelligence (CAAI 2017) PB - Atlantis Press SP - 226 EP - 230 SN - 1951-6851 UR - https://doi.org/10.2991/caai-17.2017.50 DO - 10.2991/caai-17.2017.50 ID - Chang2017/06 ER -