Proceedings of the Advances in Materials, Machinery, Electrical Engineering (AMMEE 2017)

Performance-sensitive components exploration in Spark Streaming

Authors
Ying Hou, Yi Liang, Chao Su
Corresponding Author
Ying Hou
Available Online June 2017.
DOI
10.2991/ammee-17.2017.74How to use a DOI?
Keywords
big data, spark streaming, performance-sensitive components.
Abstract

Streaming data processing has become a hot topic in the big data research. To ensure the timeliness of data processing, it is important to explore the performance-sensitive components in the streaming data processing platform, which can contribute to the more efficient performance optimization. In this paper, we describe the data processing model in the Spark Streaming, the process can be divided into multiple phases. We propose a simple yet useful method to explore performance-sensitive component components among these phases. Experimental results show that the proposed method is suitable for a wide range of workloads. At last, we demonstrate a detail example of the application of this method on the typical Spark Streaming workload Word count and prove its practicability.

Copyright
© 2017, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

Volume Title
Proceedings of the Advances in Materials, Machinery, Electrical Engineering (AMMEE 2017)
Series
Advances in Engineering Research
Publication Date
June 2017
ISBN
978-94-6252-350-0
ISSN
2352-5401
DOI
10.2991/ammee-17.2017.74How to use a DOI?
Copyright
© 2017, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

TY  - CONF
AU  - Ying Hou
AU  - Yi Liang
AU  - Chao Su
PY  - 2017/06
DA  - 2017/06
TI  - Performance-sensitive components exploration in Spark Streaming
BT  - Proceedings of the Advances in Materials, Machinery, Electrical Engineering (AMMEE 2017)
PB  - Atlantis Press
SP  - 390
EP  - 394
SN  - 2352-5401
UR  - https://doi.org/10.2991/ammee-17.2017.74
DO  - 10.2991/ammee-17.2017.74
ID  - Hou2017/06
ER  -