Analysis of CDN Website Logs Based on Hadoop

DOI

10.2991/amcce-15.2015.110How to use a DOI?

Keywords

log analysis mass data Hadoop MapReduce

Abstract

This paper designs a framework of CDN website log system based on Hadoop and a set of algorithm on the basis of user action mode excavation to analyze and process logs from searching engines. The monitor and regulation of colonies can be realized in platform monitoring modules. Under the guideline of data excavation process, this paper adopts Hadoop, an analysis tool for mass data as the experiment platform. The MapReduce reflection/excavation programming model is used. Simple and applicable HIVE from SQL and Hbase mass data pool are used to process mass logs. The writer conducts a detailed analysis on user searching action from such perspectives as topics, hits, URL order and conversational analysis to optimize platform performance and compare the system before and after the optimization. Experiment data is shown in this paper to explain that the log platform here is quite stable and efficient.

Copyright

© 2015, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Qing Song
PY  - 2015/04
DA  - 2015/04
TI  - Analysis of CDN Website Logs Based on Hadoop
BT  - Proceedings of the 2015 International Conference on Automation, Mechanical Control and Computational Engineering
PB  - Atlantis Press
SP  - 599
EP  - 604
SN  - 1951-6851
UR  - https://doi.org/10.2991/amcce-15.2015.110
DO  - 10.2991/amcce-15.2015.110
ID  - Song2015/04
ER  -