HIA: Autonomic and General Collecting of Real-time Data Streams

DOI

10.2991/icitmi-15.2015.132How to use a DOI?

Keywords

Event Streams Collecting, Autonomic Computing, I/O Driver

Abstract

Power manufacturing enterprises process events by hundreds of devices interacting with their plants. Rich statistical data distilled from combining such interactions in real-time could generate much business value. In this paper, we describe the architecture of HIA, a system for ingesting multiple geographically distributed data source in real-time with high scalability and low latency, where the data streams may be real-time or batch. The system strongly self-manages infrastructure degradation without any manual intervention. HIA guarantees that there will be no data lost in the ingested output at any point in time, that most ingested events will be present in the output in soft real-time. Our product deployment ingests millions of events per second at peak with an average end-to-end latency of less than 1second. We also present challenges and solutions in maintaining large persistent ingesting state across geographically distant locations, and highlight the design principles that emerged from our experience.

Copyright

© 2015, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Changfeng Chen
AU  - Xiang Wang
AU  - Jinying Zhou
PY  - 2015/10
DA  - 2015/10
TI  - HIA: Autonomic and General Collecting of Real-time Data Streams
BT  - Proceedings of the 4th International Conference on Information Technology and Management Innovation
PB  - Atlantis Press
SP  - 792
EP  - 799
SN  - 2352-538X
UR  - https://doi.org/10.2991/icitmi-15.2015.132
DO  - 10.2991/icitmi-15.2015.132
ID  - Chen2015/10
ER  -