Improved artificial neural network algorithms and applications

In this thesis, at first, stability and convergence speed of gradient descent with momentum algorithm is analyzed in the case of a deterministic quadratic performance function. As a consequence of theoretical analyzes, effective learning rate and momentum factor formulas which improve convergence speed are determined from the largest and smallest eigenvalue of the Hessian. This approach is tested on randomly generated test problems and results indicate that the algorithm with effective learning parameters outperforms other conventional gradient descent with momentum algorithms. Effective learning parameters obtained for the quadratic performance function are adapted to the general case where the performance is any nonlinear function of the network weights. Four different versions of gradient descent with momentum algorithm which works compatible with the effective learning parameters are proposed in the general case. Developed algorithms are compared with other conventional gradient descent algorithms on the up-to-date test problems. It is observed that algorithms with the effective learning parameters show better convergence performance than the other conventional gradient descent algorithms in general.