Proceedings of the 3rd Workshop on Advanced Research and Technology in Industry (WARTIA 2017)

Network music big data platform storage system construction

Authors
Ouyang Tong, Yi-zhen Cao
Corresponding Author
Ouyang Tong
Available Online November 2017.
DOI
10.2991/wartia-17.2017.46How to use a DOI?
Keywords
Big data platform, Online music, HBase, Distributed storage, Sqoop
Abstract

To build the network music big data platform, We need to collect a lot of music resources from the Internet, And store it on big data platforms. Therefore, it is a key problem to build a storage system that is high performance, scalable and capable of supporting big data. The network music data is very large and the metadata structure is not consistent, HBase and other distributed databases are widely used to build it. However, single RowKey index query in HBase is inefficient, it is difficult to meet the demand of big data platform for real-time or quasi-real-time query and statistical analysis. To this end, this paper proposes a solution, First, we migrated to HBase through Sqoop, which stored the music resources stored in multiple relational databases; Then use Phoenix to establish a secondary index query mechanism, build the Phoenix + HBase storage structure, and store management of the heterogeneous music resources; Finally, the read-write performance and query performance of the storage architecture are tested in the Spark parallel computing framework. Compared with the experimental results, the efficiency of Phoenix + HBase query performance is significantly higher than that of single computer and direct call to HBase, and it can operate unstructured data and structured data, which is in line with the requirement of the storage system of music big data platform.

Copyright
© 2017, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

Volume Title
Proceedings of the 3rd Workshop on Advanced Research and Technology in Industry (WARTIA 2017)
Series
Advances in Engineering Research
Publication Date
November 2017
ISBN
10.2991/wartia-17.2017.46
ISSN
2352-5401
DOI
10.2991/wartia-17.2017.46How to use a DOI?
Copyright
© 2017, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

TY  - CONF
AU  - Ouyang Tong
AU  - Yi-zhen Cao
PY  - 2017/11
DA  - 2017/11
TI  - Network music big data platform storage system construction
BT  - Proceedings of the 3rd Workshop on Advanced Research and Technology in Industry (WARTIA 2017)
PB  - Atlantis Press
SP  - 229
EP  - 236
SN  - 2352-5401
UR  - https://doi.org/10.2991/wartia-17.2017.46
DO  - 10.2991/wartia-17.2017.46
ID  - Tong2017/11
ER  -