Dinamik sembolik uygulama ve API çağrı sıralamaları kullanarak ikili programlarda zararlı davranış tespiti

Program analysis becomes an important part of malware detection as malware become stealthier and more complex. For example, modern malware may detect whether they are under analysis and they may use certain triggers such as time to avoid detection. However, current detection techniques turn out to be insufficient as they have limitations to detect new, obfuscated, and intelligent malware. In this thesis, we propose a behavior based malware detection methodology using API call sequence analysis. In our methodology, we combine dynamic symbolic execution and API function models to extract call sequences of a given binary program and decide whether it has a malicious sequence. In our experiments, we showed that our methodology is capable of detecting malware hiding behind evasion techniques and our methodology is applicable to a real-world problem.