Optimization and especially Gradient method

In this study, optimization problems, which are very necessary and important for neural networks and machine learning, are examined. This review was carried out in two phases. In the first phase, in case that an optimization problem is expressed mathematically with a function, the necessary conditions for a critical point to be a maximum or minimum point (or even a saddle point) expressed and proven by clarifying the relation between the critical points of the function and the nature of these critical points. At the second phase, practical studies (examples) were included. In this study, besides the analytical solution method, two different optimization algorithms were expressed and examined. These algorithms are Full Gradient Method and Newton Method.