Storage Solution: A Virtual Distributed Storage And Migration Architecture For Big Data

DOI

10.2991/jimec-17.2017.57How to use a DOI?

Keywords

Virtualization; data Migration; big data; distributed computing; cloud computing.

Abstract

The big data revolution has provided organizations with a large variety of data which assist in making decisions as well as providing data analysts with high volumes of data for prediction and pattern recognition. This data is stored in the cloud as it has proven to be a good storage environment due to its accessibility and security benefits .The cloud provides data for various applications such as gaming activities and data for prediction analysis in various sectors of the economy. The provision of these data to end-users such as data analysts is best provided through the use virtual desktops which require data regularly (real-time) and at a high speed, efficiency and performance levels. To achieve this users expect services to be hosted on virtual machines in interrelated data centres and that these virtual machines will migrate dynamically to locations best suited for the user as well as connect the new users .This leads to our current problem which is how can we provide high performance to manage large volumes of data from the cloud as well as how can data can be stored in such a manner that they can be easily retrieved and migrated between servers. We propose an Architecture in this paper by using Alluxio and our novel Dynamic Virtual Machine Server (DVMS) to speed up the process as well as ensure there is no delay. We further apply two plugins to Hadoop which are Sqoop and Network levitated Merge (NLM) which will assist to improve the transfer speed of data from the cloud to Hadoop to increase efficiency. The dynamic virtual machine manages the large and growing data load by categorising the data into 3 categories of pools called (1) Raw aggregated data pool, (2) Aggregated data to send and (3) Processed aggregated data pool which works in a loop to increase data migration speed as well as provide a medium to store data in preparation for new users.

Copyright

© 2017, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Randle Oluwarotimi
AU  - Matsebula Fezile
AU  - Zuva Tranos
PY  - 2017/10
DA  - 2017/10
TI  - Storage Solution: A Virtual Distributed Storage And Migration Architecture For Big Data
BT  - Proceedings of the 2017 2nd Joint International Information Technology, Mechanical and Electronic Engineering Conference (JIMEC 2017)
PB  - Atlantis Press
SP  - 260
EP  - 264
SN  - 2352-538X
UR  - https://doi.org/10.2991/jimec-17.2017.57
DO  - 10.2991/jimec-17.2017.57
ID  - Oluwarotimi2017/10
ER  -