Detection of branches and end points in cerebral vessel images with deep learning methods

Advances in microscopy technologies have gained an important place in biomedical research and have played a critical role in the study of complex biological systems, especially in neurology. Accurate and efficient imaging of brain vessels is of vital importance in the field of neuroimaging. Disruptions in the structural integrity of brain vessels can lead to the development of neurological diseases. Therefore, accurate detection of branching points and endpoints of vascular structures plays a critical role in diagnosing and treating diseases. However, traditional image processing methods have limitations due to increasing algorithmic complexity and image size. This thesis is a significant step forward in the field of neuroimaging. It aims to develop a deep learning-based object detection method for detecting branches and endpoints in two-dimensional (2D) microscope images of cerebral vessels. This method, coupled with hyperparameter optimization, promises to revolutionize the way we analyze brain vessels. Raw images of mouse brain slices obtained in the laboratory environment are subjected to noise removal with the BM3D algorithm, segmentation with the triangle thresholding method and skeletonization with the 2D thinning algorithm. The resulting branches and endpoints were labeled with bounding boxes of 5x5 pixels and converted to COCO format. Within the scope of the thesis, models were trained using four different deep-learning object detection algorithms within the Detectron2 framework, and their performance was evaluated with the Intersection over Union (IoU) metric. With some models, a success rate of over 90% was achieved. To improve the model performance further, hyperparameter optimization was performed using Optuna software, and the model accuracy increased by over 98%. The thesis also introduces BrainVasculyzer, a software developed in Python. This software provides a user-friendly interface for automatic analysis of brain vessels and its performance is evaluated by comparing it with existing analysis programs. BrainVasculyzer can automatically extract and visually present information such as the length, branching and endpoints of the vessels. This thesis not only demonstrates the significant advances made in neuroimaging through deep learning and hyperparameter optimization but also instills hope for the future, showing the great potential these technologies have in brain vessels. Keywords: Deep learning, Brain vessel analysis, Hyperparameter optimization, Object detection, Image processing, Branching and endpoint detection.