Turkish coreference resolution

Coreference resolution (CR) is the task of determining sets of mentions which identify the same real-world entity (e.g. a person, a place, a thing, an event). CR is one of the natural language processing (NLP) tasks at semantic level. Resolving textual mentions as a pre-processing step helps in several higher level NLP applications. For example, machine translation, text summarization, information extraction, question answering are shown as examples of such applications getting benefit of CR systems. Studies categorized as linguistic or theoretical approaches in the literature which uses strict language grammar rules do not seem to be suitable for Turkish CR because of the agglutinative and morphologically very rich nature of the language. This study presents a data-driven coreference resolution system which learns from annotated data by use of machine-learning techniques. Although not strictly encoded, the system also uses linguistic analysis results (e.g. morphological, syntactic) during its decision process. In contrary to some studies which could be listed under Turkish CR task and which cover only limited types of relations (such as pronominal relations), this study do not limit itself to a specific coreference type and covers many types of relations occurring between different types of linguistic entities such as pronouns, adjectival clauses, named entities and possessive constructions. In this study, we use Marmara Turkish Coreference Corpus (MTCC) (Schuller et al., 2017) (a subcollection from METU Corpus (MC) (Say et al. 2002)) where mentions, the coreference relations between them and the formed chains are manually annotated. This thesis basically focuses on pairwise classification technique with two different models used in coreference resolution task: mention-pair model (MPM) and entity-mention model (EMM). Mention-pair models basically process pair of mentions and determine whether the current mention couple is coreferential with each other without considering previous judgments. Also, mention-pair models need a clustering mechanism as a post-processing step to create mention chains from the classification output. However, entity-mention models focus on coreference relation between an entity and a mention unlike MPMs. At this point, entity may be thought as a group of mentions referred to same real world entity. Therefore, EMMs make a decision on coreference relation between a mention and a group of mentions and classifer answers this question "Should this mention be inserted to this chain?". EMMs do not need a post-processor to create chains; because chains are obtained cumulatively during the prediction. Our chain creator is a rule-based post-processor which works on the classification outputs and aims to form chains according to prediction results. Chain creator accepts a mention pair which is classified positively. If these mentions are not assigned to any chain previously, then a new chain is generated and these two mentions are inserted to that chain. When one of these mentions are already added to a chain, the other mention is also appended to that chain. If two mentions occur in two different chains, creator accepts its previous decision and does not change their locations. This study proposes a much rich feature set comprising detailed linguistic information (such as morphological and syntactic properties) extracted from a Turkish NLP pipeline (Eryiğit, 2014) inspiring from earlier studies on Turkish dependency parsing (Eryiğit et al., 2008)-(Sulubacak et al., 2016). The selected features for each mention are: head's lemma, head's coarse and fine POS tags, the status of being/containing a multi-word expression, the status of being an adjectival clause/a noun phrase, string similarity of mentions, the status of having the same head forms/lemmas (see Section 3.1 for usage of these information in MPMs and EMMs). Our initial set of experiments is to see the impact of the negative sampling size and kernels on the CR task with MPMs. With this purpose, we start our experiments with the baseline introduced in (Schuller et al., 2017) by replicating it with exactly the same parameters using SVM with linear kernel and negative sampling within +-100 window size. With our replicated baseline, we obtain slightly better scores than the ones reported in (Schuller et al., 2017) which may be attributed to the difference in the used data set (see Section 2.2 for details). Therefore, mention-pair models are prepared with different negative sampling sizes (+-100, +-70, +-40 and +-10) and polynomial, RBF and linear kernels. Highest results are obtained with +-10 window size and polynomial kernel which yields a 6.49, 8.09, 7.62 and 8.81 percentage points improvement in MUC, B3, BLANC and LEA, respectively in case of linear kernel. The use of polynomial kernel also makes an improvement of more than 1 percentage point in all metrics except MUC. In our second set of experiments, we experiment our MPM to evaluate the influence of our newly introduced feature set (see Section 3.1 for details) by using both Decision Trees (DTs) and Support Vector Machines (SVMs) as the binary classifiers. In this step, we continue with +-10 negative sampling window size and polynomial kernel for SVM. We see that SVMs are much more powerful (based of F-scores) than DTs in this setting where we use many categorical features such as lemma information. In terms of precision, DTs provide much higher results than SVMs but its recall values fall very behind. When we compare the results with the baseline feature model, we observe improvements in all metrics reaching almost 5 percentage points. In our final set of experiments on MPMs, we investigate the impact of our proposed features for the Turkish CR task. We see that the most powerful feature is "The status of having the same head lemmas" which is also a feature existing in the baseline model. Thus, the gain may not be attributed to this feature alone. Since Turkish is an agglutinative language, we see that checking the similarity of lemmas is more powerful than checking surface forms for the head tokens. As a result, the introduced MPM approach obtains 90.68% (MUC), 86.89% (B3), 85.13% (BLANC) and 78.34% (LEA) F1 scores with SVM algorithm by settings of +-10 window size negative sampling, polynomial kernel and newly the selected features yielding an improvement of 9.12, 16.06, 13.08 and 12.57 percentage points respectively over a recent baseline MPM system on Turkish CR (Schuller et al., 2017). After experiments are completed on Turkish MPMs, we also focus on the Turkish CR with entity-mention models. In EMMs, we also experience two different negative sampling approach. Firstly, the existing order of mentions in the document is ignored during the preparation of negative sampling, and as the second approach, this property is also considered. In this manner, the system is improved 9.7 (B3), 3.7 (BLANC) and 7.03 (LEA) percentage points over related evaluation metrics. The best EMM provides 75.83% (MUC), 62.37% (B3), 57.15% (BLANC) ve 50.69% (LEA) F-scores. However, the main drawback of this model is creating many singletons. Therefore, EMMs are still open for improvement for Turkish. As a future work, it is aimed to extend the feature set with word embeddings and test with different deep neural networks, and also we determine to focus on semantic based feature engineering and new approaches to make an improvement on EMMs, especially by reducing the number of creating singletons.