Davranış tabanlı zararlı yazılım tespiti ve sınıflandırılması

In recent years, cyber-attack, one of the most apparent subjects in social media, will continue to be on the agenda with increasing its importance. Malicious software which is used heavily in cyber-attacks has become indispensable object of the cyber war. Malicious softwares are used for different purposes such as steal sensitive information, create a backdoor to access system persistently, create a botnet in order to drive distributed denial of service attacks, etc. They may include one or more objectives and the objectives depend on the purpose of the malicious software writer. Stuxnet which contains tree 0-day exploits and target to Iran?s nuclear facilities, is a good example to show us that how malicious software may be used during cyber war.As is knows to all, to analyze malicious software first of all malware analyst need to detect these files and then classify them appropriately. The analysis methods are collected under two headings. These are static and dynamic analysis methods. In this study the new cascade malicious software detection method was developed by combining the mention two methods. In the study the dynamic analysis methods were used in order to detect malicious software. By the way static properties of the file were used during the classification phase.Static analysis method is performed via only the static properties of the file and does not include the execution of the malicious file. In this study, we purpose the new classification algorithm that uses byte sequences (n-gram) of the malicious file.In dynamic analysis the malicious file is need to be executed in secure environment because it investigates the behavior of the malicious software. In dynamic analysis network connections, file system operations, the active processes on the system, etc. are tracked. To carry out the dynamic analysis we created a tool called as Dynamic Malware Analysis (DMA). The developed tool which is quite simple to use, can be run as system tray icon on the Windows operating system and it is capable of alerting the user if there is any malicious activity on the system.Nowadays, malicious softwares use anti-debugger and anti-virtualization technique in order to prevent detection by dynamic analysis methods. In the study the dynamic analysis was performed on VMware virtualization environment. To bypass the anti-virtualization methods used by malwares the Pin tool was used. It is a free tool provided by Intel for the dynamic instrumentation of programs. With the help of this tool the processes, files and registry keys searching methods which are used by malicious software can be bypass.The following features are considered when analyzing malicious software:Network connection changes on the systemRegistry changesProcess changesService changesAfter detection step, the found malicious softwares are classified by using n-gram based malware classification mechanism. In this method, each malicious file is pointed by n-gram vector (byte sequences obtained from malicious file) of the file and the classification is performed over these vectors.To evaluate the developed malicious software detection and classification modules the public Pahadus malware set was used and promising results were obtained. We obtained 86% accuracy over the specified public malware set. After the detection of the file n-gram based classification method was used. In the learning phase of the classification algorithm the malicious softwares which are provided by BILGEM were used. We were obtained 92% success rate when we choose n as 4 and L as 60.