Improvement of deep neural networks by intelligent model initialization

Developments in deep learning leveraging the recent abundance of parallel computational power and visual data have resulted in significant advances in visual recognition performance. Among the methods that have emerged from this work, deep convolutional neural networks trained with a large amount of data in a supervised manner has been able to consistently deliver state of the art performance in various visual tasks. However, the amount of data this method requires limits its usefulness in real-world applications. In this study, we investigate methods to relieve deep convolutional neural networks from extreme data dependency. First, we show that transferring representations from ImageNet pretraining reduces overfitting even if the target data distribution is significantly different. In addition, we propose two approaches to stochastically generate training data using analytically designed models. The first approach is to generate entirely synthetic training data based on Gestalt principles, which is suitable when the target pattern to be recognized is low-level. Alternatively, if the target pattern to be recognized is high-level, training data is derived from an existing dataset such as ImageNet. By utilizing analytically designed elements, these two approaches inject knowledge to the model and reduce data-dependency.