Machine Learning-based Anomaly Detection of Ganglia Monitoring Data in HEP Data Center

¹ Institute of High Energy Physics, Chinese Academy of Sciences, 100049, Beijing, China
² University of Chinese Academy of Sciences, Beijing 100049, China

^* e-mail: chenj@ihep.ac.cn
^** e-mail: wanglu@ihep.ac.cn
^*** e-mail: huqb@ihep.ac.cn

Published online: 16 November 2020

Abstract

This paper introduces a generic and scalable anomaly detection framework. Anomaly detection can improve operation and maintenance efficiency and assure experiments can be carried out effectively. The framework facilitates common tasks such as data sample building, retagging and visualization, deviation measurement and performance measurement for machine learning-based anomaly detection methods. The samples we used are sourced from Ganglia monitoring data. There are several anomaly detection methods to handle spatial and temporal anomalies within the framework. Finally, we show the rudimental application of the framework on Lustre distributed file systems in daily operation and maintenance.