A new content-free approach to identification of document language: angle patterns

Language identification (LI) in text mining is the study of detecting the natural language in which a document or part of it is written. In this study, a new language indetification approach from the text using the angle information between the UTF-8 values of the characters is proposed. The proposed angle method is used for feature extraction from texts. Angle patterns method is a statistical approach. Four datasets generated in various ways were used to test the proposed approach. Obtained features were used with different classification methods such as Random Forest (RF), Support Vector Machine (SVM), Linear Discriminant Analysis (LDA), Naive Bayes (NB), and K-nearest neighbor ( Knn).The LI performance results determined from four different data sets were observed as 96.81%, 99.39%, 93.31% and 98.60%, respectively. Angle pattern method has two parameters, R and L. Trials were also carried out for different values of these parameters in order to capture different patterns. Although high successes are observed for R=1 and L=1 in general, the values of these parameters need to be decided by trial and error. In addition, the proposed approach has been tested according to character length. LI was performed for texts containing #100, #200, #300, #400 and #500 character numbers. When the results are examined, it has been determined that the success is better as the length of the text increases. It has been determined that text lengths should be #200 characters or more in order to achieve an acceptable success rate. According to the performance results achieved as a result of the study, it was determined that the proposed angle pattern method gave important distinguishing information in LI application.