Cellular automata based reservoir computing in sequence learning

Reservoir computing based on cellular automata (ReCA) constructs a novel bridge between automata computational theory and recurrent neural architectures. In this study, ReCA has been developed to solve different types of tasks. Several methods have been proposed to extract the features from the cellular automata reservoir. In most tasks, ReCA results outperform the state-of-the-art results. Concerning the model complexity, a sparsely connected network with simple binary units like elementary cellular automata in ReCA could perform the computational requirements of the reservoir in order to solve hard sequence tasks that have long term dependencies. Thus, ReCA can be considered to operate around the lower bound of complexity. Sequence learning is an essential capability for a wide collection of intelligence tasks such as language, continuous vision, symbolic manipulation in a knowledge base, etc. Therefore, ReCA has been tested using pathological synthetic tasks of sequence learning that are widely used in RNNs field. ReCA achieves zero error in all pathological tasks; using only the CA evolution states, at last time step, as a feature vector to predict the output (LAST method). The CA evolution states at all time steps (ALL method) can also be used, which improves the ReCA accuracy with large feature space. To reduce the feature space size, three options are proposed: Each by using only few states from the reservoir as features, Half by using only one side of CA evolution states, or f by reducing the dimension of the zero buffers. Using these three options together significantly reduces the ReCA complexity in some tasks by up to 98% for training and 94% for testing. The distributed representation of CA in recurrent architecture (ReCA) could solve the 5 bit tasks with minimum complexity, using only two training examples which is the lowest number of training examples for any model. Comparing between different architectures and data representations; ReCA outperforms the local representation in recurrent architecture (stack reservoir), then echo state networks and feed-forward architecture using local or distributed representation. ReCA also could solve nonbinary tasks after using one hot encoding to binarize the dataset. The results are perfect for the signal classification and IRIS tasks where ReCA achieves zero error. While for the Japanese vowels task the results are competitive; less than the state-of-the-art results a little bit. Finally, ReCA has been tested using the 20 QA bAbI tasks from Facebook; These tasks are very hard and require an understanding of the meaning of a text and the ability to reason over relevant facts. Using only supporting facts, ReCA could solve most of bAbI tasks 15 out of 20 has 100% accuracy and 2 tasks above 90%, whilst 3 tasks less than 90%. In addition, the usage of cellular automata in the reservoir computing paradigm greatly simplifies the architecture, makes the computation more transparent for analysis, and provide enough computation for large domain of tasks. Furthermore, the reservoir in ReCA can be implemented using ordinary logic gates or Field programmable gate arrays FPGAs, resulting in reducing the complexity in space, time and power consumption. Thus, our work raises the question of whether real-valued neuron units are mandatory for solving complex problems that are distributed over time.