Malware detection using machine learning and evolutionary algorithms

As cyber threats grow in complexity and frequency, the need for robust and adaptive malware detection mechanisms is critical due to their impact on national security and economic stability. Given malware's evolving nature to evade detection, selecting effective parameters by examining the interactions between malware characteristics is crucial. Traditional signature-based systems are becoming inadequate as malware adapts to new technologies. To address this, a new system is developed in this thesis that focuses on malware behavior and feature relationships. Multi-objective genetic algorithms (MOGAs) are employed to identify critical features for detection, which are then utilized by machine learning (ML) algorithms within a hybrid framework to accurately detect and classify malware. The objective of this thesis is to determine the optimal feature selection and classification methods that yield the highest accuracy within the Cuckoo Sandbox environment. Specifically, classifiers such as the J48 Decision Tree (J48), Reduced Error Pruning Tree (REP Tree), Adaptive Boosting Model 1 (AdaboostM1), Multilayer Perceptron (MLP), and Naive Bayes (NB) were evaluated. As a result of the analysis, the feature set was reduced from 335 to 200 by considering the relationships between features, resulting in a high accuracy of 93.33% and a performance improvement of 40% due to the reduction in the number of features. The obtained metrics were meticulously compared and evaluated with respect to the employed algorithms and methodologies. Furthermore, Mc Nemar's test was utilized to assess the performance of different malware detection classifiers by comparing their correct and incorrect classifications. The results from Mc Nemar's test indicated significant improvements, highlighting the effectiveness of the proposed system.