Multi-model and multi-stage learned image super-resolution

When an image processing model is trained for a given task on a training set, the performance of the model varies noticeably over the test set from image to image depending on how well the image patterns in the training set match to those in the test set. Hence, image priors learned by a single generic model cannot generalize well enough for different classes of images. In addition, end-to-end trained networks overfit to the degradation model seen in the training set in inverse problems and they do not generalize to other types of degradations well. Therefore, in this thesis, we consider inverse problems specifically image super-resolution (SR) from a big picture perspective and categorize it into two sub-fields based on the prior information about the degradation kernel. First, once the degradation kernel is known but the training set contains varying classes, we propose an ensemble network for obtaining better visual results. Accordingly, we show that training multiple deep models for an SR task for different classes of images to exploit class-specific image priors and employing a post-processing network that learns how to best fuse the outputs of these multiple SR models surpasses the performance of state-of-the-art single SR models for both images with homogeneous patterns (e.g., pure text or pure texture images) and images with non-homogeneous, spatially-varying characteristics (e.g., images with both text and texture). Furthermore, we propose class-specific training of the fusion model for images with homogeneous patterns and generic end-to-end training of fusion model for images with non-homogeneous patterns. Our experimental results demonstrate that the proposed multiple-model SR (MMSR) approach with a set of pre-trained models and a generic fusion model significantly outperforms a single pre-trained EDSR model both quantitatively and visually. It even exceeds the performance of the best single class-specific EDSR model trained on similar text or texture images. Second, if real-world LR images with unknown degradation is given, an approach to first map images downsampled by unknown filters to bicubicly downsampled look-alike images was proposed to successfully super-resolve such images. In the second part of this thesis, we show that any inverse problem can be formulated by first mapping the input degraded images to an intermediate domain, and then training a second network to form output images from these intermediate images. Specifically, two-stage SR with x2 mapping without intermediate training (i.e., without seeing the Cannon and Nikon training data) performs much better than two-stage SR with the same resolution mapping w/o intermediate training and as good as with intermediate training, which sees the respective training sets. Furthermore, the best intermediate domain may vary according to the task. Our experimental results demonstrate that this two-stage domain-adapted training strategy does not only achieve better results on a given class of unknown degradations but can also generalize to other unseen classes of degradations better.