Optimization of extreme learning machine with sparse recovery algorithms

Recently, the Extreme Learning Machine (ELM) becomes an interesting topic in machine learning area. The ELM has been proposed as a new learning algorithm for Single-Hidden Layer Feed forward Networks (SLFNs). The ELM structure has several advantageous such as good generalization performance, extremely fast learning ability and low computational process. Besides this advantageous, the ELM structure has some drawbacks. Firstly, the ELM encounters over-fitting problems because of using a least squares minimization in calculation of the output weights. Another drawback is that performance of the ELM depends on the number of hidden neurons. On the other hand the ELM may encounter the singularity problem, and its solution may become unstable, when the hidden nodes are greater than the training data. In this thesis, the output weights, which are considered sparse, have been computed by using Greedy Pursuit (GP) algorithms. The investigated GP algorithms are given as following; 1.Iterative Hard Thresholding (IHT), Orthogonal Matching Pursuit (OMP), Compressive Sampling Matching Pursuit (CoSaMP) and Stagewise Orthogonal Matching Pursuit (StOMP); 2.Forward-Backward Pursuit (FBP); 3.Orthogonal Least-Squares (OLS). The proposed GP based ELM methods have been applied to regression (group 1), classification (group 2) and time series prediction (group 3) problems. The experimental results show that a robust ELM architecture which is getting over the singularity and over fitting has been obtained by using the proposed methods. The result also show that the number of hidden neurons has been obtained. Keywords: Single-Layer feedforward network, extreme learning machine, sparsity, sparse recovery, greedy pursuit algorithms.