Proceedings of the 8th International Conference on Social Network, Communication and Education (SNCE 2018)

Hadoop Performance Tuning based on Parameter Optimization

Authors
Wei Wang, Yong Shi, Xin Liu, YiHong Feng, Ning Tao
Corresponding Author
Wei Wang
Available Online May 2018.
DOI
10.2991/snce-18.2018.118How to use a DOI?
Keywords
Hadoop, TeraSort, Parameter optimization, Performance tuning
Abstract

In order to better verify that Hadoop performance can be improved through optimization of parameters, we can use the following test methods: benchmarking, stability testing, high availability testing, scalability testing, and security testing. In this paper, the benchmark test method is used to verify the optimization of parameters and to optimize the performance of Hadoop. This article mainly focuses on the 17 parameters in Tab.1. The optimization results are shown in Tab.3. The optimization of the parameters was verified by the execution time of the TeraSort algorithm in the benchmark test. During the experiment, the CPU and memory utilization rate, disk IO and network IO throughput and other indicators were collected. Fig.1-3 fully illustrates the comparison between Hadoop and TeraSort algorithm after parameter default value and parameter adjustment. The experimental results show that after the Hadoop parameters are adjusted and optimized, the Hadoop performance tuning is achieved under certain conditions.

Copyright
© 2018, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

Volume Title
Proceedings of the 8th International Conference on Social Network, Communication and Education (SNCE 2018)
Series
Advances in Computer Science Research
Publication Date
May 2018
ISBN
978-94-6252-505-4
ISSN
2352-538X
DOI
10.2991/snce-18.2018.118How to use a DOI?
Copyright
© 2018, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

TY  - CONF
AU  - Wei Wang
AU  - Yong Shi
AU  - Xin Liu
AU  - YiHong Feng
AU  - Ning Tao
PY  - 2018/05
DA  - 2018/05
TI  - Hadoop Performance Tuning based on Parameter Optimization
BT  - Proceedings of the 8th International Conference on Social Network, Communication and Education (SNCE 2018)
PB  - Atlantis Press
SP  - 577
EP  - 580
SN  - 2352-538X
UR  - https://doi.org/10.2991/snce-18.2018.118
DO  - 10.2991/snce-18.2018.118
ID  - Wang2018/05
ER  -