Improved extreme learning machines and applications

Extreme Learning Machine (ELM) is a simple, fast, and precise machine learning algorithm but still leaves much area for improvement. In standard ELM, the hidden nodes are randomly initiated and fixed without iteratively tuning. However, the optimal number of hidden layer neurons is the key of ELM generalization performance where initializing this number randomly by trial and error is not reasonably satisfied. Using different techniques such as Leave-One-Out Cross Validation (LOO-CV) to optimize the hidden layer size is costly due to the repeatedly training and evaluation of the model on the whole data set with the exception of one sample. In this work, an efficient and reliable approach called Optimum ELM (OELM) was developed to firstly determine the minimum hidden layer size that yields an optimum generalization and performance. Moreover, another approach that exploits the merits of orthogonal projections with Singular Value Decomposition (SVD) was proposed in order to tackle the problems of randomness and correlated features in the input data. This approach that is called hereafter Projected ELM (PELM) achieves more than 2% advance in average accuracy. Tikhonov Regularization in the form of squared loss penalty with ELM (TRELM) that regularizes and improves the matrix computations utilizing L-curve criterion and SVD was implemented as a most appropriate candidate to this work. Unlike the most iterative methods, L-curve can estimate optimum regularization parameter by illustrating a curve with flexible points that represents the trade-off between minimizing the training error and the residual of output weight. The proposed TRELM was tested based on three different scenarios of data sizes: small, moderate, and big datasets. Due to simplicity, robustness and less time consuming of OELM and PELM, it is recommended to use them in small and even moderate data. TRELM demonstrated that enhancing the ELM performance needs to enlarge the size of hidden nodes. As a result, in big data, increasing hidden node number in TRELM is necessary which concurrently leads to better accuracy. A novel extension of ELM termed Graph-based ELM with L_21-norm Regularization (L21GELM), which merges efficient approaches, was developed to improve the robustness and compactness over the Graph Extreme Learning Machine (GELM). Many linear and nonlinear subspace learning (SL) methods were adopted herein to reduce the similar properties of input neighborhood pixels and hence extract only the discriminative features. Due to the nonlinear mapping of the activation functions and some SL approaches that may destroy the local consistency properties in the dataset, the graph Laplacian is intervened into the optimization function as a demand to exploit the class similarity advantages of intrinsic construction to enhance the overall model accuracy. Finally, an iterative algorithm with proved convergence for solving L_21-norm optimization problem was involved to eliminate the irrelevant features and to decrease the outlier's effects. The outcomes demonstrated that the L21GELM was more general and accurate (more than 3% advance in average accuracy) than other ELM variants, also it can handle large-scale data problems with good feature selection and less sensitive to noisy environments. Precisely evaluating the compactness of ELM is difficult due to the randomness in hidden layer nodes number, weight and bias values. To reduce the randomness, resultant overfitting and large variance of the ELM, a selective Weighted Voting ensemble model based on Regularized ELM which called (WVRELM) is investigated. It can strongly enhance the overall performance including accuracy, variance and time consumption. Optimizing the regularization parameter of the RELM using Leave-One-Out Cross Validation (LOO-CV) method is a time-consuming approach. Thereby, efficient PREdiction Sum of Squares (PRESS) criteria that utilizes Singular Value Decomposition was proposed to address the slow execution issue with low human intervention. Moreover, a simple ensemble pruning approach based on the eigenvalues for the input weight matrix was developed owing to its capability to produce the discriminative parameters and select the optimal ensembles. As an advantage, in this thesis, the weights assignment for base classifiers were based on the same PRESS error metric and the variation of the output weight vector (β) solutions used for the RELM, thus, it could reduce computational cost and space requirement. The outcomes demonstrated that the average accuracy of WV-RELM was approximately 3% higher than other ELM variants, stable with low variance, faster with 2.8 folds than RELM with LOO-CV. Various well-known benchmark datasets such as face with facial expression and object recognition as well as different state-of-the-art learning approaches were examined and compared with the proposed approaches. Consequently, the data representations handled by various feature spaces in training data enhanced significantly the classification efficiency as compared to other related popular algorithms.