Malware detection with machine learning using executable files numeric and textual features

Keywords: Portable Executable (PE) file format, Static malware analysis, Machine learning, Malware detection, Text Classification, Cyber Security. Since malicious software or malware has become too agile, evolves quickly and can distribute itself, the numbers of cyber-attacks continue to grow as well. Therefore malware detection and relevant research became a serious issue in the information security community. Most of the commercial proposed solutions for malware detection are signature-based methods. Signature-based method fundamentally requires prior malware variant knowledge which will not work on malwares that have never been "captured" for analysis, called zero-day malware. This work explores an approach to overcome the deficiencies in signature-based malware detection, namely usage of the static analysis method to extract valuable features of Windows Portable Executable (PE) file format. We extract raw features of Windows executables which are PE header information, used Compilers, DLLs, and API functions inside each DLL of Windows PE file. Thereafter, for PE header's information, four different feature selection techniques (Univariate feature selection (SelectKBest), Recursive Feature Elimination (RFE), L1-based feature selection (L1) and Tree-based feature selection (ExtraTree)) are used to remove irrelevant features and so reduce dimension of feature set, where each technique votes whether they have selected the feature. Finally, the vote is counted and the features with higher votes score are used with Compilers, DLLs and API functions to train our classifiers. For Compilers Information, DLLs and API functions Text Classification techniques tools such as TF-IDF were used for feature selection. To evaluate the performances of our detection models (kNN, Decision Tree, Random Forest and Light GBM) and their ability to detect unseen and new malware we conducted three experiments: First by using PE headers features (Numerical) only where we achieve 99.49 % of accuracy with Random Forest, second by using only Compilers Information, Dlls and API functions features (Textual) where 99.06% of detection accuracy was achieved again with Random Forest, and finally we combined Numerical and Textual features where an overall best performance was achieved by Random Forest with a recall of 99.44%, a false positive rate of 1.2%, a precision of 99.25%, and an accuracy of 99.13%.