Evaluation of machine learning based twitter social bot detection systems

Social media platforms such as Twitter, provide an incredibly effective way to communicate with people. While these platforms have many benefits, they can also be used for deceiving people, spreading misinformation, manipulation and verbal harassment. Social bots are usually employed for these kind of activities to artificially increase the amount of a particular post. To mitigate the effects of social bots, many bot detection systems are developed. However, the evaluation of these methods are challenging due to lack limited available datasets and the variety of bots people might develop. Therefore, in this thesis, it has been investigated whether the performance of bot detection systems can be accurately evaluated and the social bot detection problem is solved. The experiments carried out, the performances of 4 different bot detection systems are compared on different experimental setups to find which model has the highest performance. Although it was observed that the scores in the original studies where the models were very high, the models showed poor performance in different test sets. However, since it is understood that the model with the highest performance is the Botometer, a more detailed examination of the performance of the Botometer should be done. Experiments were carried out with a different perspective and the performance of the Botometer is evaluated using 5 different bot scenarios. As a result of these experiments, Botometer shows low performance in all scenarios except one and the problem of social bot detection is still an open problem to investigate.