Anomaly Detection in Resource Constrained Environments With Streaming Data

Abstract:

Isolation Forest (or iForest) is a well-known technique for anomaly detection. It is, however, a bulky approach that assumes the luxury of large storage space and is also ineffective with dynamic streaming data so common nowadays in varied application domains. In this work, we present the Preprocessed Isolation Forest (PiForest) approach for anomaly detection that works well in resource constrained environments and is also effective on streaming data. PiForest is largely based on the iForest algorithm and to effectively handle the streaming data includes a pre-processing stage. In the pre-processing stage, Principal Component Analysis (PCA) is first harnessed to significantly reduce the dimension and bulk of the data. Subsequently, the streaming characteristic of the data is handled through a sliding window mechanism that creates sequential blocks of data for systematic processing. PiForest is able to identify anomalies as effectively as iForest and other state-of-the-art anomaly detection techniques but has substantially low storage and prediction complexity. We conduct empirical evaluation of the proposed approach with standard data sets and show that it performs comparably with standard techniques in terms of Area Under the Receiver Operating Characteristic Curve (AUC-ROC) and is able to work with high-dimensional, streaming data. Subsequently, we do a real-world hardware implementation of PiForest and demonstrate that the approach is realistic and practicable in resource-constrained environments.