Tumor segmentation in histopathological images

While the histopathological image segmentation area has a rich literature, the introduction of Convolutional Neural Networks (CNN) into our lives, obtaining high-resolution Whole Slide Images (WSI) of patient tissue surfaces with motorized microscopes, and presenting large data sets labeled by cancer institutes to open access, researchers have more It has led them to discover current ESA architectures which can produce more accurate results. The correct detection of cancerous tissues in the breast lymph nodes plays an important role in determining the stage of the disease and planning the appropriate treatment method. However, difficulties such as the differences in the staining procedures of the tissue samples taken from the lymph nodes of the patient, the different imaging device and imaging format, and the very different shapes, colors and structures of the tissue make the segmentation process difficult. Existing CNN architectures (Faster RCNN and Mask RCNN) have shown great success in object detection and segmentation. The main idea of CNNs is to reveal the general patterns (distinctive features) in the image gradually by using the convolution and pooling layers. Convolution layers combine local filter results to produce features with high display capability. The partnership layer provides a reduction in the size of the data by summarizing the generated attributes. Both layers seem to work in local image areas. These architectures that focus on local context cannot adequately use the global context information in the image. In this thesis, hybrid CNN architectures have been developed which can use local and global context information together in segmentation of histopathological images. A non-local network module has been used to provide CNN architectures with the ability to capture the global context. This module is combined with a multi-scale network model, and a segmentation architecture with a low parameter space has been developed. The boundary sharpening (boundary-aware) module has been added to this architecture, which provides improvement in the boundary lines of the objects in the image, and the use of deep control technique (including middle layer outputs in the cost function) is provided in cost calculation. The performance of the proposed hybrid architecture has been tested in many different experimental studies. As a result of the experimental studies, it has been observed that the segmentation results obtained with the existing CNN architectures can be reached faster with the proposed architecture. Another study involves the use of the Multi-Scale Residual Block (MSRB) approach developed to increase the resolution in the segmentation problem. MSRB architecture consists of two parts. While the first part obtains the multi-scale properties of the input, the second part produces the residual feature map. High resolution images can be produced with the combination of both features. At the stage of integrating this capability into the segmentation architecture, firstly, the kernel size suitable for segmentation was determined. The effect of the Atrous Spatial Pyramid Pooling (ASPP) module on the determined kernel-sized architecture has been investigated. It has been observed by examining the mIoU metric that the architecture with a low number of hyperparameters exhibits high segmentation performance. Finally, the effect of attention mechanisms on segmentation accuracy has been examined. Since the segmentation architectures developed have a multi-scale structure, attention mechanisms have been applied to each scale separately. Thus, the emphasis of the region to be focused on in the local and global context information obtained by the architecture, and the dominance of the background was provided to draw attention to a specific region. As in previous architectures, it is aimed to make the segmentation process fast by keeping the parameter space low. The performance results obtained are compared with existing segmentation architectures. As a result, three different architectures have been added to the literature for faster and more accurate segmentation of histopathological images in the field of breast cancer detection.