Proceedings of the 2016 International Conference on Artificial Intelligence and Engineering Applications

Exploiting FastDFS Client-based Small File Merging

Authors
Haimeng Chen, Hua Zhang
Corresponding Author
Haimeng Chen
Available Online November 2016.
DOI
10.2991/aiea-16.2016.40How to use a DOI?
Keywords
Masses of small files; small file merging; Client-based.
Abstract

How to store masses of small files is a generally acknowledged problem in industry and academia. Small file merging is the most successful strategy to solve this problem, which has been supported by many distributed filesystems, such as FastDFS. However, in FastDFS, our experiments indicate that excessively wide striping causes performance degradation, and pre-allocated space causes data loss or error. Therefore, this paper presents a client-based small file merging scheme based on the characteristics of online web application. This scheme makes full use of the resources of the web server and optimizes the small file merging with the annealing algorithm to improve the throughput. Experimental results show that the write performance are improved by 52.03% and query performance by 27.69% compared with the original FastDFS small file merging scheme.

Copyright
© 2016, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

Volume Title
Proceedings of the 2016 International Conference on Artificial Intelligence and Engineering Applications
Series
Advances in Computer Science Research
Publication Date
November 2016
ISBN
10.2991/aiea-16.2016.40
ISSN
2352-538X
DOI
10.2991/aiea-16.2016.40How to use a DOI?
Copyright
© 2016, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

TY  - CONF
AU  - Haimeng Chen
AU  - Hua Zhang
PY  - 2016/11
DA  - 2016/11
TI  - Exploiting FastDFS Client-based Small File Merging
BT  - Proceedings of the 2016 International Conference on Artificial Intelligence and Engineering Applications
PB  - Atlantis Press
SP  - 213
EP  - 217
SN  - 2352-538X
UR  - https://doi.org/10.2991/aiea-16.2016.40
DO  - 10.2991/aiea-16.2016.40
ID  - Chen2016/11
ER  -