Video deinterlacing and demosaicing by deep learning

Deinterlacing and demosaicing are commonly used techniques in the image processing pipeline for consumer video. Despite the fact that real-world video deinterlacing and demosaicing are well-suited to supervised learning from synthetically degraded data because both degradation models are known and fixed, learned video deinterlacing and demosaicing have received much less attention compared to denoising and superresolution tasks. This thesis progressively explores feature alignment, integration and reconstruction stages for both tasks tailored to their known and fixed degradation subsampling patterns. We begin by presenting our initial work of a novel multi-field deinterlacing architecture that aligns features from adjacent fields to a reference field (to be deinterlaced) by designing novel deformable residual convolution blocks with two variants of different scales. To the best of our knowledge, this work is the first to propose fusion of multi-field features that are aligned via deformable convolutions for deinterlacing. Next, based on our initial work, we propose a novel multi-field full frame-rate deinterlacing network, which adapts the state of-the-art superresolution approaches to the deinterlacing task. This model incorporates self attention mechanism with deformable convolution residual blocks to align features and additively integrate aligned features for reconstruction. In order to reconstruct odd and even fields directionally, separate reconstruction modules are utilized according to the parity of each reference. Our extensive experimental results demonstrate that the proposed method provides state-of-the-art deinterlacing results in terms of both numerical and perceptual performance. Upon all these previous work, we propose a new multi-picture architecture for both video deinterlacing or demosaicing by aligning multiple supporting pictures with missing data to a reference picture to be reconstructed, benefiting from both local and global spatio-temporal correlations in the feature space using modified deformable convolution blocks and a novel residual efficient top-$k$ self-attention (kSA) block, respectively. Separate reconstruction blocks are used to estimate different types of missing data. Our extensive experimental results demonstrate that the proposed novel architecture provides superior results that significantly exceed the state-of-the-art for both tasks in terms of PSNR, SSIM, and perceptual quality. Ablation studies are provided to justify and show the benefit of each novel modification made to the deformable convolution and residual efficient kSA blocks.