Nonconvex multicriteria optimization and portfolio selection problem

In this study, nonconvex multicriteria optimization theory with main definition and theorems has been investigated thoroughly. One of the fundamental aims of the multicriteria optimization theory is finding conditions about existence of optimal solutions. Gasımov (1992), has found a necessary condition for proper minimal points by developing a nonlinear separation theorem which is based on level set of a convex sets. In our study, by developing Gasımov?s study the relation between the recessive function of the separation function and the cone which provides an order in space has been found for given necessary condition to be sufficient and in the event of this relation, given point is the proper minimal point of the set has been proved.Scalarization methods have been investigated to solve multicriteria optimization problem and advantages of the conic scalarization method which can obtain all efficient points of nonconvex multicriteria problem have been dealt with. For solving the obtained nonconvex and nondifferentiable conic scalar problem, the theory and advantages of sharp augmented Lagrangian duality and F-MSG algorithm have been shown.At the last part of this study, the portfolio selection problem in finance theory which is one of the application area of nonconvex multicriteria optimization theory has been investigated. In addition to mean-variance application, it has shown that the problem which takes into account higher order central moments and cardinality constrained has quite complex and difficult form and it has been informed about future studies concerned with the solution of this problem.Keywords: Multicriteria optimization, proper minimal point, nonconvex nondifferentiable optimization, conic scalarization, sharp augmented Lagrangian duality, F-MSG algorithm, portfolio selection problem.