Hyper-parameter optimization of deep neural networks for cell-vessel segmentation

Deep neural networks (DNNs) are important methods for the segmentation of medical images. However, the performance of DNNs depends on the architecture and hyperparameters. These hyperparameters can be selected manually, which can be time-consuming and difficult. Therefore, Neural Architecture Search (NAS) methods are developed. First, we propose a new NAS method called UNAS-Net. UNAS-Net is optimized by meta-heuristics such as Differential Evolution (DE) and Local Search (LS). UNAS-Net is evaluated on the Optofil and Cell Nucleus datasets and outperforms the U-Net regarding segmentation performance and computational complexity. Second, DE-based NAS approaches for brain vessel segmentation are proposed. These approaches are based on U-Net and Attention U-Net, which are widely used in medical image segmentation. Traditional DE and opposition-based DE (ODE) are chosen as search methods. The experiments are performed on two publicly available cerebrovascular segmentation datasets: vesseINN and KUVESG. The proposed methods achieved better segmentation performance in different segmentation metrics and generated 9.15 times more complex architectures than the baselines. Retinal vessel segmentation (RVS) is very important in medical image analysis as it helps to identify and monitor retinal diseases, and automated methods are needed. Third, a new NAS method is proposed for the RVS that discovers architectures with high segmentation performance and lower inference time: MedUNAS. ODE and genetic algorithm (GA) methods are used to search for the best network structure, and also discrete and continuous encoding strategies are compared. The proposed methods outperformed the baseline U-Net on four datasets with networks with up to 15 times fewer parameters. Furthermore, it is demonstrated that the generated networks can be effectively adapted to new medical tasks. Finally, the BaDENAS approach, which combines predictors with the NAS process, is proposed and tested for the RVS problem. The results show that BaDENAS improves the baselines regarding model complexity, segmentation performance and convergence speed.