Count based clustering and classification of RNA-seq data

In molecular biology, gene-expression based studies are frequently used for examining transcriptional activities in different tissue samples or cell populations. Gene expression data can be used for different tasks; e.g differential expression, classification and clustering. In this thesis we focused on clustering and classification of gene expression data obtained from RNA sequencing experiment. Poisson (PLDA) and negative binomial (NBLDA) linear discriminant analyses are selected as discrete, and nearest shrunken centroids (NSC) are selected as continuous classifiers in classification part. We proposed an extension of NBLDA as sparse classifier and compared its performance with other classifiers. In clustering part, we used k-means and hierarchical clustering as continuous, and Poisson and negative binomial clustering as discrete approaches. A comprehensive simulation study is conducted for classification part under different scenarios. Furthermore, we used three different real data sets. Simulation results showed that overdispersion has an important effect on model performances. Overall, discrete models performed better in classification. Among discrete classifiers, NBLDA outperformed PLDA when data set is highly overdispersed. Moreover, our proposed algorithm performed better than NBLDA algorithm in terms of prediction accuracy and sparsity. We also applied the same classifiers to three real data sets and found that results agree with the simulation results. Clustering, on the other hand, is applied to real data sets only. Unlike classification, discrete and continuous clustering approaches performed similar on two real data sets. We did not perform a simulation study for clustering scenarios due to several reasons: (i) simulations were computationally intensive, and (ii) dissimilarity matrices cannot be calculated when data set had several thousands of features. Therefore, we were not able to generalize clustering results. In conclusion, discrete statistical approaches should be preferred for classification while discrete or continuous approaches can be preferred for clustering purpose. However, if visualizing data is of interest in clustering, data should be transformed for better graphical results.