An empirical binary classification performance comparison of ensembling random forest and gradient boosting with oversampling on scarcely populated datasets

The purpose of this thesis is to present an empirical approach to do binary classification when the number of observations available in the dataset is small. In binary response classification, the statistical learning model learns the relationships between the features for each response class. In the case of small sized datasets the model might fail to extract the true relationship between the features. To improve the model performance, we developed an empirical approach consisting of feature selection, oversampling and ensembling Random Forest model with Extreme Gradient Boosting model trained on oversampled dataset. A two phased feature selection is conducted where the features are eliminated according to correlation criteria using pearson correlation coefficients and ranking criteria using lasso feature elimination. After number of features are reduced using these two phases, a kernel with density estimated from the data is fitted and 2,500 observations were drawn from this kernel. For creating the proposed model, Random Forest algorithm is trained on the original dataset and Extreme Gradient Boosting algorithm is trained on the over sampled dataset. The outcomes of these two models are than ensembled using validation set accuracy as a tie breaker, to obtain the final outcome. The approach has been tested on 4 different types of binary classification datasets. It is observed that, training extreme gradient model wiht the oversampled dataset has significantly improved the classification accuracy when compared to the results of extreme gradient boosting model trained on the original datasets.