Non-linear neuron modeling using padé approximants with applications to single image super-resolution and image compression

It is fairly recent for artificial neural networks to gain extreme popularity. However, the building blocks of artificial neural networks, the perceptrons, have been in existence for more than eight decades, and convolution operation has been used in networks for more than thirty years. The popularity of neural networks comes not only from their success in solving many problems they are applied to, but also from their theoretically guaranteed convergence to the solutions under certain conditions. The universal approximation theorem tells that for any required mapping with any desired accuracy, there is a neural network that achieves it, provided with sufficient hidden units. This statement is an existence theorem; it does not specify any feature about the network. Therefore, the research community developed different strategies to fulfill the predictions of the theorem such as developing hundreds of non-linear activation functions and proposing more advanced neuron models. In this thesis, we propose a novel, more powerful and inherently non-linear neuron model, called Padé approximant neuron, or Paon in short. As the name implies, Paon uses the Padé approximant to calculate the rational function approximation on the learned locations of the input features, increasing the representation and learning capacity of the network as well as its non-linear power and receptive field. Moreover, coming in two variants as solutions for the possible singularity problem of rational approximation, Paons are able to replace, and are a super set of, previously proposed neuron models, offering adaptability in various configurations. Experiments on the single image super-resolution and image compression problems demonstrate that Paons surpass their competitors when compared in equal conditions in terms of number of parameters, and are able to bring performance increase even when direct replacement and reduction in number of layers are the cases in point.