Development of a new deep learning architecture for nuclear segmentation in H&E histopathological images

Histopathological images play a crucial role in medical pathology, enabling tasks like nuclei counting, nuclei density estimation, nuclei-to-cytoplasm ratio calculation, and nuclear atypia grading. Automating nuclei segmentation and detection not only reduces pathologists' workload but also enhances the accuracy and consistency of diagnosis and treatment planning. In medical image segmentation, particularly in histopathological image analysis, the U-Net architecture has become the de-facto standard, yielding excellent results. However, while U-Net excels at capturing local information effectively through convolutions, it has limited ability to explicitly capture long-range dependencies. Transformers, initially designed for sequence-to-sequence prediction, have emerged as alternative architectures with inherent global self-attention mechanisms. Nevertheless, Transformers may lack localization abilities and fine-grained details due to insufficient focus on low-level features. In this thesis study, we propose a novel U-shaped architecture called CompSegNet, which utilizes both convolutional and transformer blocks for accurate nuclei segmentation in H&E-stained histopathological images. The proposed convolutional blocks enhance the Mobile Convolution (MBConv) block through a receptive fields enlargement strategy known as the Zoom-Filter-Rescale (ZFR) and incorporate an enhanced global context (GC) block for global context modeling. The bottleneck of CompSegNet consists of a Residual Bottleneck Transformer (RBT) block, which integrates Transformer encoder blocks in a tailored manner to a variant of the Sandglass block. Additionally, a noise-aware stem (NAS) block is designed at the highest level of the U-shaped architecture to minimize the potential impact of noise in the input images. Furthermore, a weighted joint loss function has been formulated to improve the overall segmentation performance of the architecture. The proposed CompSegNet architecture exhibits superior performance compared to existing nuclei segmentation methods in both quantitative and qualitative evaluations while maintaining a reasonable number of parameters. When evaluated on the MoNuSeg 2018 and NuSeC 2023 datasets, CompSegNet achieves a competitive and commendable AJI score of 0.704 and 0.677 respectively, demonstrating its effectiveness in accurate nuclei segmentation.