Network intrusion detection with payload-based approach

Rapidly growing network systems become more vulnerable to threats with the improved sophistication of attack techniques. Various types of network attacks affect networks in different ways and continue to be a serious threat despite developing intrusion detection mechanisms. Early detection of network intrusions is crucial to taking precautions and reducing the damage to the system. In addition, the ability to distinguish attacker flows from legitimate ones ensures that the network continues to provide service safely to the clients. In this thesis, payload-based features that characterize network flows are proposed to provide early detection of network attacks and to identify attacker flows. Besides the features conventionally used in application classification, features based on greedy algorithm-based metrics that allow comparing defined probability distributions over different sample spaces at various lengths are also used. Moreover, features based on spectral domain analysis of payload sequences are extracted to capture the complicated patterns that are not observed in the original domain. Also, features based on discrete cosine transforms are utilized in the characterization of these network flows. These features are extracted using N-gram analysis for various N values. In the classification stage, SVM models trained with these features are used. Performance evaluation is given for publicly available IDS 2012 and IDS 2017 datasets that contain different kinds of attack traces. Early detection of network intrusions based on features extracted from the first 3 and 5 packets of a flow achieves high detection rates while detecting network intrusions early.