Extreme learning machine based on L1 and L2 norms

Extreme learning machine (ELM) as a type of single-layer feedforward neural network (SLFN) has been widely used in various disciplines due to its superior properties like fast learning, simplicity, high adaptability for different real-life applications. The main reason of this interest is that ELM eliminates the iterative parameter tuning in the conventional neural networks. However, ELM has some drawbacks such as instability and poor generalizability at the point of estimating the data as well as issues related with the determination of the optimal model size as a consequence of structural risk minimization. In this study, in order to deal with these aforementioned drawbacks and improve the performance of ELM, some statistical estimators are discussed in the context of neural networks and various algorithms are proposed. In the first part of the study, we proposed different selection criteria for determinining appropriate tuning parameter for ELM algorithm based on ridge estimator. Later on, Liu estimator, r-k class estimator, a cascade of Liu and Lasso regression estimators are developed as alternatives to the existing ELM algorithms based on ridge estimator. In the scope of this study, the applications of algorithms written from scratch in the R platform are tested via many real-world data sets and the performance comparison are presented in details. These proposed algorithms present effective shrinkage and/or variables selection performance which is more stable, generalizable and compact than ELM and some advanced algorithms based on ELM.