Agent based and ontological data leakage prevention system against advanced persistent threats

Data leakage caused by Advanced Persistent Threats (APTs) is a growing concern for organizations and governments. Although recent studies have made progress addressing APT risks, they still lack sufficient capabilities for data leakage prevention (DLP). APTs employ content-based methods for data exfiltration alongside their sophisticated methods, increasing the risk of data exfiltration and reducing the effectiveness of the current solutions in the literature. On the other hand, using system calls has proven to be effective in malware detection and semantic analysis has been used for inferring relations regarding an attack successfully. This thesis proposes APTONSYS, an agent-based system that utilizes ontology-driven reasoning mechanism to prevent the data leakage caused by APTs. The proposed approach establishes semantic connections between low-level details of an attack, such as system call, process, and content information with the APT Technique and Tactics defined within the MITRE's ATT&CK framework. A novel content classification method and mechanism to detect content-based attacks executed by APTs are also integrated into the solution. Further, an APT Risk definition is introduced by using Techniques and Tactics that are applied in the system. The effectiveness of the solution is presented using experimental tests using data from real-life scenarios and open-source APT simulation tools. DLP systems are applications that prevent the transfer of sensitive content to inappropriate locations outside or inside an organization. By employing various content tracking, matching, and classification methods, they determine the sensitivity level of data at rest, in-motion, or in-use. Regarding the outcome, they allow or block actions. While they are successful in preventing simple and often accidental leaks, they have been unable to maintain their effectiveness in deliberate and targeted leaks, as evidenced by the increasing number of leakage incidents in recent years. APTs are attackers that utilize sophisticated methods, often with a command-and-control mechanism. They employ targeted methods and can utilize tools existing within target systems. They bypass DLP detection mechanisms by remaining in the target system for extended periods, elevating privileges, spreading within the system, and repurposing existing processes for their objectives. DLP solutions typically utilize static policy-based control mechanisms that lack "situational awareness". These rules take into account components such as the executable file, source, and destination of the process in question. However, in APT-based attacks, these policy rules can be circumvented by methods like altering system programs, accessing memory areas of other processes, saving content in different locations, using different network protocols, employing PowerShell scripts, and elevating privileges. Besides the structural vulnerabilities of DLP solutions against APTs, their employed content matching and classification methods also possess weaknesses against purposeful data exfiltration attacks. Methods such as statistical analysis, fingerprinting, and regular expressions employed in content classification can be bypassed through simple and minor alterations to the content. Attacks involving changes in sections, words, or letters on the content can evade these methods. Additionally, using synonymous or polysemous words, employing book ciphers, extracting summaries, and many other different methods can render the content ineffective against DLP systems by altering it from its original form. APTs, whose primary goal is data leakage bypass DLP systems by utilizing such content-based attacks. In this thesis, a comprehensive system proposal has been developed to prevent APT-sourced data leaks in DLP systems. Within this scope, an agent-based system relying on ontology knowledge has been presented. This system enables scalable and online detection of APT risks and establishes a connection with data leakage attacks. Additionally, the system incorporates a novel content classification algorithm that facilitates resilient content classification against purposeful data leakage attacks. In the context of research conducted towards APT and malware detection shows that while the steps taken by APTs may vary until they reach their target, their behavioral aspect remains consistent. This signifies the need for a behavioral analysis model for successful detection. Accordingly, it is observed that associating Techniques and Tactics with an APT based on MITRE ATT&CK is a crucial method for the development, maintenance, and robustness of the defense system. Therefore, the proposed solution includes such an inference mechanism. Due to the characteristics of APTs, conducting executable file analysis alone is insufficient to detect APT-originated attacks at all stages. A dynamic analysis is also needed on top of this static analysis by continuous monitoring of system processes. While methods like leveraging event logs exist for dynamic analysis, they can only identify predefined, specific events. Moreover, as these event logs are constantly monitored by Intrusion Detection Systems and triggered by certain conditions, APTs avoid performing actions that generate such event logs. In this context, utilizing Application Programming Interface (API) calls of operating system libraries becomes crucial. These calls are commonly known as "system calls" in literature. Regardless of the tool used, every operation executed on the system is conducted through a system call. Hence, APTs cannot evade using system calls. If the type, source, user, and operation of a system call can be associated with APTs' Tactics, Techniques, and Procedures (TTPs), the progression of an attack can be determined. Therefore, within the framework of this thesis, APT behaviors are identified by evaluating components related to system calls and their relationships. The literature shows that the use of ontology and semantic analysis holds significant importance in identifying relationships among components constituting a system. Numerous studies have highlighted the significance of semantic relationships between components such as the source, target, operation, and content concerning attack detection. Therefore, within the scope of this thesis, a novel ontology, APTON has been presented. It allows to define and correlate system call properties, such as category, source, target and user account with process security assessment, content classification, and content attack detection results. Novel APT technique, tactic, and risk detection rules are presented using the classes and relations in the ontology. This presents a novel solution that merges APT detection with DLP approaches. APTs can utilize multiple executable files, leverage existing software and tools within the system, and persist operations across different computers. Therefore, it is imperative to conduct an analysis across the entire system, encompassing all computers on the network. Hence, in the presented system, the collection and evaluation of system calls are facilitated through agents operating on endpoint systems. These agents can share results with each other, leading to a comprehensive detection across the system. While there is no comprehensive solution in the literature addressing the weaknesses of DLP solutions against content-based attacks used by APTs, there exist methods that can enhance resistance against such attacks and improve classification accuracy. By using these methods, a more resilient, multi-stage content classification method against content-based attacks is presented in this thesis. Additionally, by continuously monitoring the content accessed by processes, the detection of known content-based attacks is performed, aiding in APT behavior detection. Finally, the proposed models and system were tested in two stages to demonstrate their performance. Firstly, the proposed ontology model was tested independently with data collected from the system. The content classification algorithm was also independently tested with data sets. The comprehensive performance of the agent-based APT data leakage detection system was demonstrated by executing APT attacks on the established network and subsequently detecting them by the system.