Theses supervised by Prof. Dr. Tunga Güngör
11 theses · Boğaziçi University
Enhancing relation classification by using shortest dependency paths between entities with pre-trained language models
Relation Extraction (RE) is the task of finding the relation between entities from a plain text. As the length of the text increases, finding the relation becomes more challenging. The shortest dependency path (SDP) between two entities, obtained by traversing the terms in the text's dependency tree, provides a view focused on the entities by pruning noisy words. In RE's supervised form Relation Classification, the state-of-the-art methods generally integrate a pre-trained language model (PLM) into their approaches. However, none of them incorporates the shortest dependency paths into their calculations to our knowledge. In this thesis, we investigate the effects of using shortest dependency paths with pre-trained language models by taking the R-BERT relation classification model as our baseline and building upon it. Our novel approach enhances the baseline model by adding the sequence representation of the shortest dependency path between entities, collected from PLMs, as an additional embedding. In experiments, we have evaluated the proposed model's performance for each combination of SDPs generated from Stanford, HPSG, LAL dependency parsers, and baseline with BERT and XLNet PLMs in two datasets, SemEval-2010 Task 8 and TACRED. We improve the baseline model by absolute 1.41% and 3.6% scores, increasing the rankings of the model from 8th to 7th and 18th to 7th in SemEval-2010 Task 8 and TACRED, respectively.
Abstractive text summarization for morphologically rich languages
The exponential growth in the number of documents available on the Web has turned finding the relevant piece of information into a challenging, tedious, and time-consuming activity. Accordingly, automatic text summarization has become an important field of study by gaining significant attention from the researchers. Recent progress in deep learning shifted the research in text summarization from extractive methods towards more abstractive approaches. The research and the available resources are mostly limited to the English language, which prevents progress in other languages which especially differ in terms structure and characteristics such as the morphologically rich languages (MRLs). In this thesis, we mainly focus on abstractive text summarization on two MRLs, Turkish and Hungarian, and address their important challenges. Firstly, we tackle the resource scarcity problem by curating two large-scale datasets for Turkish (TR-News) and Hungarian (HU-News) aimed for text summarization, but are also suitable for other tasks such as topic classification, title generation, and key phrase extraction. Then, we utilize the morphological properties of these languages and adapt them to summarization where we show improvements upon the existing models. Later, we make use of pretrained multilingual sequence-to-sequence models and provide state-of-the-art models for abstractive text summarization and title generation tasks. Evaluation of text summarization for MRLs is very limited. Thus, we show how preprocessing can drastically influence the evaluation results through a case study in Turkish. Finally, morphosyntactic methods are proposed for text summarization evaluation and a human judgement dataset is curated. It is shown that morphosyntactic tokenization processes during evaluation increase correlation with human judgements. All the work and the curated datasets are made publicly available.
Solving Turkish math word problems by sequence-to-sequence encoder-decoder models
It can be argued that solving math word problems (MWP) is a challenging task due to the semantic gap between natural language texts and mathematical equations. The main purpose of the task is to take a written math problem as input and produce a proper equation as output for solving that problem. This thesis describes a sequence-to-sequence (seq2seq) neural model for automatically solving MWPs based on their semantic meanings in the text. The seq2seq model has the advantage of being able to generate equations that do not exist in the training data. It comprises a bidirectional encoder to encode the input sequence and comprehend the problem semantics, and a decoder with attention to track semantic meanings of the output symbols and extract the equation. In this thesis, we investigate the successes of several pre-trained language models and neural models, including gated recurrent units (GRU) and long short-term memory (LSTM) seq2seq models. Our research is novel in the sense that there exist no studies in Turkish on this natural language processing (NLP) task that utilize the pre-trained language models and neural models. There is also no Turkish dataset designed to implement the neural models for MWP task. Due to the lack of data, we translated the well-known English MWP datasets into Turkish using a machine translation system. We performed manual adjustments, and built the corpora to contribute to the literature. Although Turkish is an agglutinative and grammatically challenging language to work on, our system correctly answers 71% of the questions in the corpora.
A comprehensive analysis of subword tokenizers for morphologically rich languages
Transformer language models have paved the way for outstanding achievements on a wide variety of natural language processing tasks. The first step in transformer models is dividing the input into tokens. Over the years, various tokenization approaches have emerged. These approaches have further evolved from character and word-level representations to subword-level representations. However, the impact of tokenization on models performance has not been thoroughly discussed, especially for morphologically rich languages. In this thesis, we comprehensively analyze subword tokenizers for Turkish, which is a highly inflected and morphologically rich language. We define various metrics to evaluate how well tokenizers encode Turkish morphology. Also, we examine how the tokenizer parameters like vocabulary and corpus size change the characteristics of tokenizers. Additionally, we propose a new tokenizer for agglutinative and morphologically rich languages. We demonstrate that our tokenizer reduces overall perplexity and enables better generalization performance. Downstream task experiments show that morphology supervision in tokenization improves model performance.
Intrinsic and extrinsic evaluation of word embedding models
In natural language processing tasks, representing a word is an important issue. After Bengio et al. introduced a simple neural network language model that learns word vector representations in 2003, representing words in continuous vector space has become more popular. Mikolov et al. introduced a method named word2vec and showed that word embedding could capture meaningful syntactic and semantic similarities in 2013. Many methods and implementations have been proposed for English since then. However, there are only a few studies on word representations in Turkish. In this study, we aimed to understand and analyze how word embedding models work on both Turkish and English. We focused on the word2vec word embedding model and tried to modify it to improve the quality of word representations. Additionally, we trained many models with different window sizes and dimensions. The impact of different configurations on the quality of word representations was analyzed both intrinsically and extrinsically. We reported the accuracy on word analogy tasks for intrinsic evaluation and word similarity tasks for extrinsic evaluation. Our results show that our proposed models perform better on most of the word analogy task categories for Turkish. We also showed that increasing window sizes and dimensions does not always affect the accuracy in a positive direction. For some analogy and word similarity tasks, it affects negatively.
Hierarchical mixtures of experts in generative adversarial networks
Generative adversarial networks (GANs) are deep neural networks that are designed to model complex data distributions. The idea is to create a discriminator network that learns the borders of the data distribution and a generator network trained to maximize the discriminator's loss to learn to generate samples from the data distribution. Instead of learning a global generator, one variant trains multiple generators, each responsible from one local mode of the data distribution. In this thesis, we review such approaches and propose the hierarchical mixture of generators that learns a hierarchical division in a tree structure as well as local generators in the leaves. Since these generators are combined softly, the whole model is continuous and can be trained using gradient-based optimization. Our experiments on five image data sets, namely, MNIST, FashionMNIST, CelebA, UTZap50K, and Oxford Flowers, show that our proposed model is as successful as the fully connected neural network. The learned hierarchical structure also allows for knowledge extraction.
Feature analysis for recommender systems using transformer-based architectures
Recommender systems are technology-based solutions that assist users by suggesting relevant items among millions of items. It could be anything like a movie, a meal, a vacation spot, shoes, or a piece of music. Unlike traditional recommender systems, sequential and session-based recommender systems make recommendations by paying attention to the order of items that users interact with. The advantage of such systems is that they take into account varying tastes. Additionally, due to some legal requirements, the users' data cannot be collected from some platforms, and the recommender system has to suggest the session's information without having any previous knowledge. It may only have to recommend products according to a few interactions in that session. These reasons constitute the importance of sequential and session-based recommender systems. In this thesis, we have experimented with sequential and session-based recommender systems using the Transformers4rec framework, which allows us to use transformer architectures in recommender systems. We observed that transformer architectures work better in short interaction sequences than long ones. We showed that additional features enhance the model's performance, particularly time-based features. Additionally, we examined and interpreted that the importance of features changes according to the size, shape, and type of data.
Theme supervised nonnegative matrix factorization for topic modeling
Topic models are often used to organize and interpret large and unstructured corpora of text documents. They try to explain the topics that constitute the semantic infrastructure of the document sets and try to find the distributions of these topics for the documents. Because of its unsupervised nature, the outputs of a topic model has to be interpretable to represent its success. However, the results of a topic model are usually weakly correlated with human interpretation. In this thesis, we propose a semi-supervised topic model called Theme Supervised Nonnegative Matrix Factorization that can benefit from labeled documents to improve and facilitate the interpretation of the topics. Our model constrains the representation of the topics to align with the labeled documents and this enables the topics discovered by the model to be readily understood. To utilize the labels provided by the documents more efficiently and to explore the document sets in more depth, we used a hierarchical topic structure consisting of themes, subtopics, and background topics in our model. We created layers under the themes that permit unsupervised learning for subtopics. This hierarchical structure, with the unsupervised learning capability it provides, enables our model, which was restricted with supervision, to discover new dimensions and make more detailed classifications. We tested our model on Schwartz dataset we created, as well as Brown and Reuters datasets with different supervision ratios. Our model estimates the topics of the documents much better than the traditional nonnegative matrix factorization and latent Dirichlet allocation for any situation; and besides, the effect of supervision is noteworthy, especially at low ratios. Moreover, our new term scoring metric successfully alters the weights of significant and insignificant terms for each topic and makes the topics easier to understand and interpret.
Hierarchical multitask learning for language modeling with transformers
Recent works show that learning contextualized embeddings for words is beneficial for natural language processing (NLP) tasks. Bidirectional Encoder Representations from Transformers (BERT) is one successful example of this approach. It learns embeddings by solving two tasks, which are masked language model (masked LM) and the next sentence prediction (NSP). This procedure is known as pre-training. The pre-training of BERT can also be framed as a multitask learning problem. In this thesis, we adopt hierarchical multitask learning approaches for BERT pre-training. Pre-training tasks are solved at different layers instead of the last layer, and information from the NSP task is transferred to the masked LM task. Also, we propose a new pre-training task, bigram shift, to encode word order information. To evaluate the effectiveness of our proposed models, we choose two downstream tasks, one of which requires sentence-level embeddings (textual entailment), and the other requires contextualized embeddings of words (question answering). Due to computational restrictions, we use the downstream task data instead of a large dataset for the pre-training to see the performance of proposed models when given a restricted dataset. We test their performance on several probing tasks to analyze learned embeddings. Our results show that imposing a task hierarchy in pre-training improves the performance of embeddings.
Multilingual identification of verbal multiword expressions using bidirectional long short-term memory based architectures
Verbal multiword expression (VMWE) identification is a challenging task for many natural language processing studies. In this study, sequence tagging approach accompanied with stochastic models and variants of IOB tagging scheme is used for VMWE identification. In the scope of this thesis, a VMWE annotated Turkish corpus is constructed as the first part of the PARSEME shared task 1.1 which is constructing VMWE annotated corpora in many languages. Additionally, a multilingual system called Deep-BGT is developed as the second part of the shared task which is developing language-independent VMWE identification systems using the corpora constructed in the first part. The Turkish corpus is one of the biggest corpora in the shared task. The training and test corpora that were published in the PARSEME shared task 1.0 are updated as the PARSEME shared task 1.1 training and development corpora according to the new guidelines. A new test corpus is constructed from scratch. Deep-BGT uses the bidirectional Long Short-Term Memory model with a Conditional Random Field layer on top (BiLSTM-CRF). To the best of our knowledge, this study is the first one that employs the BiLSTM-CRF model for VMWE identification. Deep-BGT was ranked the second in the open track in terms of the general ranking metric. Moreover, a novel tagging scheme called bigappy-unicrossy is introduced to rise to the challenge of overlapping VMWEs. Finally, the VMWE identification system is advanced by evaluating a subset of hyperparameters which consists of tagging scheme, number of units, number of BiLSTM layers, and classifier. A comprehensive analysis of BiLSTM based architectures for multilingual identification of VMWEs is presented accordingly.
Identification of verbal multiword expressions using deep learning architectures and representation learning methods
Understanding multiword expressions (MWEs) plays an instrumental role in Natural Language Processing applications such as parsing and machine translation. MWE identification is a task that automatically detects and classifies MWEs in running text. As with the basic characteristics of MWEs, significant challenges exist in MWE identification. Considering the recent attempts of the PARSEME network on verbal multiword expressions (VMWEs), we focus on the identification of VMWEs. We update the PARSEME Turkish train and test corpora 1.0 (2017) as the PARSEME Turkish train and development corpora 1.1 (2018). We construct the PARSEME Turkish test corpus 1.1. In addition, we develop a multilingual VMWE identification system based on bidirectional long short term memory with conditional random fields networks accompanied with the gappy 1-level tagging scheme. To extend our study, we examine the impact of data representation format on the VMWE identification task. We introduce the bigappy-unicrossy tagging scheme to recognize overlaps in sequence labelling tasks. Our results show that data representation format is important to identify discontinuous VMWEs. Moreover, we enhance our neural VMWE identification model with automatically learned embeddings by neural networks to respond to the variability challenge. We compare character-level convolutional neural networks and character-level bidirectional long short-term (BiLSTM) networks. We analyze two different schemes to represent morphological information using BiLSTM networks. Our results demonstrate that character embeddings and morphological embeddings improve performance in general. The choice of representation learning method depends on language.