A kind of Prefetching Data Way to Hadoop MapReduce Environments
- 10.2991/icmmita-16.2016.234How to use a DOI?
- Hadoop; MapReduce; Prefetching ;distributed computing.
In MapReduce environment, problem of data redundancy,a large number of tasks to be processing and mass data storage come up, in order to solve these problems, we put forward the way of data prefetching , preprocessing of hid the remote data access latency, by adjusting the allocation of resources to Reduce the business, we put forward the method of than before, caused by the Reduce tasks of remote data access performance problems caused by time delay and resource competition system, the hidden by prefetching data the method Reduce task of remote data access latency, and Reduce task control through the resource allocation, to Reduce resource competition caused the Reduce task, the experiment results show that with the default Hadoop graphs and Hadoop Online Prototype (HOP), compared with the method the system performance can be improved by more than 10%.
- © 2017, the Authors. Published by Atlantis Press.
- Open Access
- This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).
Cite this article
TY - CONF AU - Hui Xia AU - Peng Wu PY - 2017/01 DA - 2017/01 TI - A kind of Prefetching Data Way to Hadoop MapReduce Environments BT - Proceedings of the 2016 4th International Conference on Machinery, Materials and Information Technology Applications PB - Atlantis Press SN - 2352-538X UR - https://doi.org/10.2991/icmmita-16.2016.234 DO - 10.2991/icmmita-16.2016.234 ID - Xia2017/01 ER -