Conditional computation techniques in deep neural networks with conditional information gain

Recently, deep neural networks, particularly convolutional neural networks, have excelled in computer vision tasks such as image classification, object detection, and semantic segmentation. Their high performance stems from numerous layers and learnable parameters. However, this complexity poses challenges for efficient inference, especially on devices with limited computing power, like edge devices. Among numerous similar approaches in the literature to address this issue, conditional computing is an efficient inference method where parts of a deep neural network are used or skipped based on the properties of the input. In this thesis, we develop two main conditional computing approaches: "Conditional Information Gain Networks", where a neural network is designed in the shape of a tree, and the samples are routed based on network elements that are trained with information gain. The second one is the "Conditional Information Gain Trellis", which describes a trellis-shaped network that again allows the routing of the samples based on the decisions of routing units trained by information gain objectives. We develop loss functions, training methodologies, and regularizers for both models. For both models, we develop inference methodologies that allow the routing of samples over more than one route in these networks, where we try to achieve a balance between additional model performance and extra computational burden. These multiple-path routing approaches, which we call "Sparse Mixture of Experts" inference, are implemented using algorithms such as Bayesian Optimization, Cross-Entropy Entropy Search, and Reinforcement Learning. We show the results of these model designs with various experiments.