Determining the hyperparameters of YOLO algorithm using artificial bee colony and particle swarm optimization

This thesis investigates the application and effectiveness of Artificial Bee Colony (ABC) and Particle Swarm Optimization (PSO) algorithms in the optimization process of hyperparameters found in YOLOv5, the fifth iteration of the YOLO algorithm. Despite the significant advancements that YOLO has brought to the field of object detection, the requirement for expert knowledge and trial-and-error in tuning its numerous hyperparameters adds complexity to its application. In this study, nine hyperparameters of the algorithm were selected and optimized using ABC and PSO algorithms. When the training performance of the resulting models obtained using images from the Oxford-IIIT pet dataset was compared, it was observed that the model optimized with PSO tended to detect rare and challenging data and was trained with rapid convergence. In contrast, the model optimized with ABC learned all data in a balanced manner, achieving slow and steady convergence with low learning rates. During testing, the ABC-optimized model produced %81,7 mAP score which similar to its training outcomes while the PSO-optimized model produced %76,8 mAP score and exhibited overfitting and failed to perform well on the test data. These findings indicate that in the hyperparameter optimization problem of the YOLO algorithm, the ABC algorithm, which can produce a greater variety of results compared to PSO, is more conducive to achieving successful outcomes.