Outlier detection over streaming data of human activities

Nowadays, due to the continuous generation of huge volumes of data from many devices spread around the world, the term "Data Stream" comes into existence. Data Stream refers to an unbounded continuous flow of data happening at different rates, and coming in an open-ended fashion. New challenges emerged with streaming data in regards to outlier detection due to the special characteristics this data has in comparison with Static Data. Stream Data characteristics can include the following: massive infinite size, multi-dimensionality, high arrival rate, and more. Hence, there is high demand for solutions and frameworks that are highly effective in processing large-scale data in real-time (scalable, fast & fault-tolerant, etc.), and achieve outlier detection in real-time. This thesis addresses the problem of outlier detection in real-time over streaming data by proposing a pipeline methodology capable of meeting the high demand for streaming data solutions. The proposed methodology was evaluated by implementing it in a solution for a real-life problem. The methodology-based solution and its Machine Learning model was successfully capable of detecting outliers in human activities in real-time utilizing the continuous stream of data from a smart phone's built-in accelerometer. The Machine Learning model was developed with Isolation Forest algorithm as the outlier detection algorithm. The selection of Isolation Forest was due to the limitations of other algorithms in dealing with high dimensional, unbalanced data, high memory usage, and time consumption challenges as presented in literature. Moreover, Apache Spark was utilized as big data distributed processing unified engine for its capability of providing high fault tolerance, applying machine learning, and execute complex computations in real-time over streaming data. Also, Apache Kafka was employed as a messaging system for its capability of providing a scalable pipeline with high throughput, low latency, and decoupling features over data streams. The principles of Apache Spark and Kafka form the underlying infrastructure of our developed pipeline. The obtained evaluation results of our proposed methodology prove the adequacy and effectiveness of the developed pipeline with achieved accuracy of 97% in performing outlier detection in real-time.