Sentiment analysis in Turkish texts

There is a remarkable curiosity inside us to know what others think. It is also important for production, marketing, service and advertising firms to learn the attitude of people towards their goods, brands and services. Firms used to monitor customer attitudes by receiving feedback forms from their customers and analyze them manually or made some questionnaire to survey companies with charge of money. However these methods were not able to capture statistically sufficient size masses and were costly in terms of human labor and money. Social media platforms, which are easily accessible platforms, provide remarkable sources to get feedbacks from target masses, but it is impossible to analyze these feedbacks by human labor. Therefore, automated sentiment analysis tools are crucial for companies' customer services to have the capability of capturing complaints and/or positive feedbacks in the right time. Processing by computers allows these data to be used in the market. Implementing an efficient sentiment analysis tool will increase the customer satisfaction and will decrease the costs. This is the motivation of sentiment analysis research area. We can say that: sentiment analysis is one of the most useful tool for social media monitoring. A text with sentiment generally includes opinions, attitudes and evaluating. Opinions and attitudes can be towards a topic, a person, a brand or a politic view. They are not only valuable sources for social researches and surveys but also quite important for firms to analyze responses and feedbacks about their goods and services. Sentiment analysis is needed to capture the attitude of a text towards any topic. Sentiment analysis is the process of determining whether a text includes sentiment or not and classifying the sentiment into positive, negative and neutral classes. Although, sentiment analysis is a hard task for NLP and Data Mining research areas, giving practical solutions and having high market value results increasing academic and market interest in it. Accessing data including sentiment on WEB and processing it are still hard tasks to be solved, because there are huge number of sources including sentiment and most of sentiments are hidden in long texts. Before development of WEB, there were almost no information on digital platforms and no possibility of access to this information. People started to share their opinions on certain topics on digital platforms. The amount of accessible information with opinion on the Web has been increasing with the contribution of forums, columns, blogs, and social media. Processing this information, extracting the subjectivity and classifying the sentiment are the main challenges of the sentiment analysis that need to be solved. Sarcasm and irony also have remarkable importance and interest in both psychology [3] and NLP [4] [5] research area. Increasing the accuracy in detecting sarcasm will increase the performance of the sentiment analysis. Unfortunately it is also a difficult task to identify the sarcasm in a natural text even for a human [4]. Sentiment analysis or opinion mining is the computational study of opinions, sentiments and emotions expressed in text [1]. Extracting opinions and analyzing the polarity of these opinions are the main problems of the sentiment analysis. Various approaches are utilized to solve these problems in academic researches. Most of them are on subjectivity classification and sentiment classification. Subjectivity classification is a problem of classifying any document as objective or subjective and sentiment classification is the classification of these subjective documents into positive or negative [1] classes according to their sentimental polarity. NLP and machine learning techniques are extensively used for Sentiment Analysis. Knowing the characteristics of the language are essential for NLP and Sentiment Analysis because different languages require different preprocessing techniques. Sentiment analysis approaches are mainly based on either machine learning or lexicon based methods. Both methods have advantages and disadvantages in terms of accuracy and human labor. Our goal is to show the comparison of the strengths and weaknesses of these methods on two different types of datasets. As a lexicon based method, we build a framework similar to the systems described in Thelwall et al. [2] and Vural et al. [6]. To implement machine learning based sentiment analysis, we have investigated several machine learning methods like Pang et al. [7] and Ero˘gul [8]. The majority of sentiment analysis approaches are concentrated on English. However,there exists a number of sentiment analysis studies on Turkish [8] [6]. Ero˘gul [8] handled the sentimental analysis problem as a supervised machine learning classification problem and applied different ML techniques with different features like unigrams, bigrams, POS tags and combination of them. Vural et al. [6] presented a lexicon based sentiment analysis framework using Turkish version Sentistrength [2] lexicon. They used an approach based on summing lexicon scores of sentiment oriented words in related text. In this work, we applied both ML based and Lexicon based SA methods on Turkish with additional features. In order to evaluate the performance of lexicon based and ML based sentiment analyzers, we use two datasets exhibiting different characteristics. Our first dataset is comprised of tweets which suffer from orthographic and grammatical problems. Tweets are usually difficult to process for NLP purposes since they frequently contain abbreviations, missing vocals that need devocalization and ungrammatical constructs both due to the character limitation of Twitter and mobile devices with limited text entry capabilities. We collect another dataset that consists of movie reviews which are more grammatical and orthographic than tweets. We applied our tests for binary (positive and negative) and trinary (positive-negative and neutral) classification. Pre-processing is one of the most important steps of the sentimental analysis in Turkish. Having a very productive inflectional and observational morphology, Turkish is a difficult language to process. A number of preprocessing steps are required for both lexicon based and ML based approaches due to the productive Turkish morphology. In this study, we employ deasciification, basic text normalization, morphological analysis, morphological disambiguation and multi-words expressions extraction preprocessing steps. Text normalization pre-processing steps such as spelling correction are necessary prior to morphological analysis step since the data is noisy. A finite-state-machine based morphological analyzer [9] is used to produce root of words, suffixes and morphological tags. This level produce ambiguous results. Since the morphological analysis stage produces ambiguous results, a morphological disambiguation module is required. We used a rule based morphological disambiguator [10]. Multi-words expressions extraction aims to identify the segments of the texts which are generally sequential but not compositional [11]. We use Kemal Oflazer's MWEs extraction application's Perl script to handle the MWEs extraction problem. Finally we identify and combine expressions which have different meanings and may have/haven't sentiment when they separate from each other, e.g. "kafayı ye-" (literally eat the head) none of the words have an sentiment polarity by their self but it means "to get mentally deranged" and has negative sentiment polarity when they are together. We added these sentiment holding MWEs to our lexicon. Our lexicon based sentiment analysis approach depends on comparing features of a given text with a pre-determined sentimentally oriented lexicon. Sentiment analysis does not require a detailed pre-processing [12] phase before classification for English but it is necessary for Turkish and similar agglutinating languages. Turkish is an agglutinating language in which it is possible to add many suffixes to word roots. These derivational and inflectional suffixes can change the POS tag and sentimentally orientation of the word. Important suffixes for sentiment analysis are considered to be the negation suffix (+ma/+me) and absence/presence suffixes (+sız/+siz (without), +lı/+li (with)) which can change the sentiment orientation of a nominal word. Handling these suffixes increase the performance of the sentiment analysis [13] [7]. The morphological analysis is needed to handle linguistic features for sentiment analysis, e.g. roots, POS tags, suffixes and adjuncts of the words. For a lexicon based sentiment analyzer, it is necessary to have a sentimentally oriented lexicon which is effective to detect the sentiment of a sentence. Since there were no Turkish lexicon we manually translated a basic English lexicon (Sentistrength, 2547 words) [2] into Turkish. Although there were some other more detailed lexicon in literature, such as SenticNet [14], WordNet-Affect [15], we used Sentistrength lexicon as a baseline lexicon. We reconstructed it by adding 700 MWEs, 650 words with absence/presence suffixes, 110 extra needed words for Turkish (slangs, curses and some special words) and we removed 350 root words due to adding them again as words with absence/presence suffixes. Actually we use Sentistrength as a starting point. After reconstructing, our final lexicon contains 2784 nominals and 873 verbs totally 3657 terms which have a polarity magnitude between [-5, +5]. Because of negation (-me, -ma) and absence/presence suffixes (+sız/+siz (without), +lı/+li (with)) suffixes, we must be careful when finding root of the words. It is not effective technique to use regular expressions like 'isolat*' which stands for 'isolate' 'isolated' 'isolation' 'isolating' in English, because of differentiation of words with suffixes in Turkish. Negation occurs in two different ways for Turkish. The first is using negation words ("de˘gil", "yok") and second is using negation suffixes (–me, -ma). When negation suffixes met we add negation word ("de˘gil") after related words, so that all negation forms become standardized. During calculating the sentiment score of texts, negation words change the sign of the sentiment score of the related word. We use a booster words list ("çok", "baya", "en" etc.) which have a boosting effect when met before an adjective. We handle punctuations like '!' after sentimental terms as boosters but giving less strength. Instead of with/without words in English we have absence/presence suffixes (+sız/+siz (without), +lı/+li (with)) in Turkish which are added to nouns and change their POS tag to adjective. It is a kind of negation and changes the polarity of the following word. If any absence/presence suffixes met we do not eliminate these suffixes ("umut-suz"). As we mentioned before we also add these sentimental adjectives with absence/presence suffixes to the lexicon. So in sentiment score calculating process we compare these words with Lexicon. The ML approach treats the sentiment analysis as a supervised classification problem. Supervised classification requires a sufficently large labeled dataset for proper training but Lexicon based sentiment analysis does not. Determining of feature set is another key process for ML classification. In order to create the feature vector, we use unigrams and bigrams by using inverse-document-frequency (TF-IDF) feature ranking and selection method. We conduct our experiments using SVM, NB and Decision Trees (J48) classification algorithms. 10 fold cross validation technique is utilized to train and test our supervised classifiers. We use accuracy measure, the number of instances that predicted correctly, to evaluate performance of our systems. We activate and deactivate modules to show the contribution of each module to performance of sentiment analyzers. According to results, each module has a contribution to the performance of Lexicon based sentiment analysis method but the most effective ones are Negation handling and MWEs handling for Twitter dataset and deasciification and negation handling for Movie dataset. The performance of Lexicon based sentiment analysis Method is 75.2% for Twitter dataset and 79.5% for Movie dataset. Results show that MWEs extraction and handling absence/presence suffixes bring reasonable improvement to performance of Lexicon based method. Since Movie reviews are too long and have too many sentimental words, MWEs extraction option does not bring enough improvement. As most researchers [1] [8] reported, our results also show that SVM hashighest accuracy than other algorithms for ML approach. The best performance of ML Based Sentiment Analysis Method is 85.0% (SVM) for Twitter dataset and 89.5% (SVM) for Movie dataset. Using unigrams and bigrams together gives the best performance for almost all classifiers on both datasets. Results indicate that bigrams can handle most of consecutive cases such as negation, boosting and MWEs. As surface forms of words include enough linguistic information such as negation and absence/presence suffixes, the usage of surface forms that combined with unigrams increases the performance of ML based method slightly for Movie dataset. But it decreases the performance for Twitter dataset since Twitter dataset is too noisy and feature selection threshold leaves most of bigrams below the feature selection threshold (min. 20 occurrence in Movie dataset and min. 5 occurrence in Twitter dataset). It decreases the performance when combined with unigrams+bigrams for Movie dataset also. In comparison of these two methods, ML based method performs better than Lexicon based method on both short (Twitter dataset) and long informal texts (Movie dataset). The results show that accuracy of Movie dataset is better than accuracy of Twitter dataset in both Lexicon based and ML based sentiment analysis methods. Although Lexicon based sentiment analysis is unsupervised, it works well when text does not include sarcasm or irony.