Analysis and Design of Improved Intelligent Search Strategy for Web Crawler

DOI

10.2991/mcei-18.2018.41How to use a DOI?

Keywords

Intelligent search; Network crawler; Spider; URL; Link filtering

Abstract

This paper mainly studies the design and implementation of the search engine's searcher Spider program, and introduces the concept and technical essentials of Spider program in detail. Network crawler is a web crawler program which can run in the background with configuration file as the initial URL crawling down with the width first algorithm and saving the target URL. The paper presents analysis and design of improved intelligent search strategy for Web crawler. Based on multi-thread web crawler, the client can access the server through socket, and the client sends its own set request to the server.

Copyright

© 2018, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Hongsheng Xu
AU  - Bin Zhao
AU  - Ganglong Fan
PY  - 2018/06
DA  - 2018/06
TI  - Analysis and Design of Improved Intelligent Search Strategy for Web Crawler
BT  - Proceedings of the 2018 8th International Conference on Mechatronics, Computer and Education Informationization (MCEI 2018)
PB  - Atlantis Press
SP  - 214
EP  - 218
SN  - 2352-538X
UR  - https://doi.org/10.2991/mcei-18.2018.41
DO  - 10.2991/mcei-18.2018.41
ID  - Xu2018/06
ER  -