Development of efficient method and hardware for sparse machine learning

Most natural signals exhibit sparsity which can enhance the efficiency, interpretability, and effectiveness of the related algorithms. Artificial neural networks (ANNs) are widely used in the literature and their efficiency can be enhanced with sparse learning. Extreme learning machine (ELM) is a prominent training method for an ANN with its fast training speed, and good prediction performance. However, due to the need for regularization to prevent overfitting and the large number of neurons required in the hidden layer, such ANNs demand significant computation power for large-scale data. This thesis proposed a sparse mixed norm (l2,1) regularized online machine learning algorithm (MRO-ELM) based on alternating direction method of multipliers (ADMM). A linear combination of the mixed norm and the Frobenius norm regularization is applied and update formulas are derived. Graphics processing unit (GPU) accelerated version of MRO-ELM (GPU-MRO-ELM) is also proposed to reduce the training time by processing appropriate parts in parallel using the implemented kernels. In addition, a novel automatic hyper-parameter tuning method is incorporated to GPU-MRO-ELM with GPU acceleration. The experimental results show that the GPU-MRO-ELM algorithm outperforms the similar online ELM algorithms in the literature in terms of training speed, and prediction accuracy. Moreover, compared to cross-validation (CV), the proposed automatic hyper-parameter tuning method shows a dramatic reduction in tuning time. Furthermore, a parallel hardware architecture was proposed and implemented on field programmable gate array (FPGA). This architecture is the first to support norm regularization, exhibiting lower resource utilization and higher prediction accuracy compared to existing parallel architectures.