Hybrid system design for malicious software detection

Malicious software is a known threat in computer security for a long time. However, in recent years, malicious software was started to use with different purposes, e.g., attacks for high-class governmental & commercial organizations and large scale crypto-ransom attacks. Widely used signature-based methods are mostly ineffective against attack vectors like zero-day attacks. Updated or newly installed computer systems, including critical infrastructure, also face zero-day attacks. Most of the time, this type of attack spotted after at least one incident. Computer systems will be vulnerable until the incident detected. Both static and dynamic analyses benefit machine learning methods for shorten analysis processes and prevent zero-day attacks. Machine learning is both expected to be robust and fast as commercial security products, also recognize malicious patterns like humans. Although most research done on this topic shows promising results, most commercial products still use signature-based methods. The purpose of this thesis is to develop a portable, scalable, and interpretable machine learning model. For ensuring an interpretable and portable model, basic features extracted from executable files were used as inputs. An ensemble model developed according to experimental results. The developed model was observed to be more successful than individual models on multi- class malware dataset represented as static feature vectors. In this way, a model which made using two different feature sets and five different classifiers presented as a hybrid ensemble model. Also, this hybrid model can be used for different executable file types without any change. As a result of this work, the developed hybrid machine learning model showed higher than %98 accuracy on the multi-class malware dataset.