Deep convolutional neural networks for image inpainting

The goal of inpainting is reconstruction of an image without incurring noticeable changes. It is a widely used technique by the photo and video editing applications for repairing damaged images, removing undesired objects or refilling the missing parts of images. Although fixing the small deteriorations are relatively simple, filling the large holes or removing an object from the scene are still challenging due to complexity of the problem. Deep neural networks have shown promising results in image inpainting even if the missing area is relatively large. They are able to recover certain information by just considering the partial data. In this study, our aim is to provide a solution for the inpainting problem using deep convolutional neural networks. We propose a two staged approach for inpainting. At the first stage a generative model produces an intermediate result, which is satisfactory by itself, to repair the damaged image region. At the second stage, output of the generative model is enhanced by a residual network which finds out the high frequency details. For the first stage, we use an existing architecture and propose a new training approach as an alternative to the Generative Adversarial Networks which produces promising results. Our end-to-end neural network takes an image, which the certain part of its center is extracted, as an input, and then it attempts to synthesize texture for the blank region. One of the essential questions about realistic texture synthesis is: how can we measure the realism? No magical mathematical formula to determine whether an image is real or artificially constructed exists. In order to solve this challenging problem, a crucial step is to construct synthesis models which are trained based on a comparison of real images with generated outputs. Although primitive objective functions like Euclidean Distance assist in measuring and comparing information on the general structure of the image, they tend to converge to the mean of pixel values that cause blurry outputs. To solve this issue, during the training phase, a distinct deep convolutional neural network is used and it is called an Advisor Network. We show that the features extracted from intermediate layers of the Advisor Network, which is trained on a different dataset for classification, improves the performance of the autoencoder. We also train our network by using combination of the Advisor Network and the Generative Adversarial Network. Although deep neural networks are able to predict structure and texture of missing parts closely, most of the existing inpainting networks introduce undesired artifacts and noise to the repaired regions. To solve this problem, we present a novel framework which consists of two stacked convolutional neural networks that inpaint the image and remove the artifacts, respectively. The first network considers the global structure of the damaged image and coarsely fills the blank area. Then the second network modifies the repaired image to cancel the noise introduced by the first network. The proposed framework splits the problem into two distinct partitions that can be optimized separately, therefore it can be applied to any inpainting algorithm by changing the first network. Second stage in our framework which aims at polishing the inpainted images can be treated as a denoising problem where a wide range of algorithms can be employed. Our results demonstrate that the proposed framework achieves significant improvement on both visual and quantitative evaluations.