On gradient descent optimization algoritms in machine learning

In this study, the structure, types, advantages, and disadvantages of the gradient-based optimization algorithms which have an important place in machine learning are represented. For this purpose, the most known 1st order optimization algorithms in the literature; Stochastic Gradient Descent, Momentum, Nesterov Momentum, AdaGrad, Adadelta, RMSProp, Adam and Nadam algorithms and Newton, BFGS and L-BFGS algorithms from 2nd order optimization algorithms are handled. The mathematical structures of these algorithms are examined and three different real-life problems are discussed for the comparison of the algorithms. ResNet50, VGG19, and logistic regression models are applied to solve these problems with artificial intelligence models. The results are evaluated through tables and figures. By measuring the performances of the algorithms with metrics, both the performances of the algorithms against each other and the performances in the models are determined and the results are interpreted.