Audio visual attention for robots from a developmental perspective

Developmental robotics is associated with embedding a human-like behavior in a developmental manner within the robot. The field takes inspiration from the human cognitive model for the syntheses of a cognitive-like decision-making model. The key concepts associated with the topic are hierarchy of the system, incremental learning and multi-sensory cooperation. To achieve a developmental like structure, the system has to be hierarchical in nature. Specifically, the perception level has to be separated from (and put beneath) the decision level. The cognitive-like structure is achieved by ascending from the lower-level to the upper level. The ascension should entail a reduction in the engagement of processing resources and a suppression of response to stimuli of lower priority. The process is similar to filtering where only important information propagated further in the model. Not only that, but it is also modified so that the meaning is conveyed in a symbolic manner rather than transporting the raw information itself. However, for the establishment of such hierarchy to be truly developmental in essence, incremental learning is a crucial factor. After all, the human cognitive model achieves its highly complex dynamics through a lifetime of evolution. As a matter of fact, incremental learning seems to offer, in this case, more than a way out of the scarcity of data at time zero of the experiment dilemma. But it also offers an opportunity of allowing a unified gradual growth of multiple prospective cognitive -functions embedding on a larger scheme and over the long run of the field's research. The multi-sensory cooperation comes then naturally in the picture because the majority of cognitive functions feed on the fused information provided by the different lower level sensory. Among the highly hierarchical cognitive functions, attention is one that has been heavily addressed by the literature. More specifically, audiovisual attention relies heavily on the concept of fusing multi-sensory information. In order to design an audiovisual attention system within a developmental framework, the nature of the attention phenomenon found in the human model has to be understood and translated into the machine learning and artificial intelligence view. Concepts related to attention like "audiovisual objects" and "binding" found in the neuroscience literature seem to provide key insight to how audiovisual attention occurs. "Audiovisual objects" are the cognitive-association representative of an auditory quantity emitted by a visual object perceived by both eyes and ears. The "binding" concept is the one responsible for creating this symbolic association. Examining these concepts for design inspiration, and taking into account that visual and auditory quantities are perceived through "features" from the scope of Artificial intelligence, a system-design that attempts to replicate such concepts can be established. The audiovisual attention system introduced here is of a two level hierarchy. The low level modules are auditory and visual perception modules; the second-level module fuses the information provided in the low level ones to achieve a higher-order cognitive like behavior. The auditory module relies heavily on the concept of incremental unsupervised learning, where the classifier labels the incoming new data instances, assumes the label to be true, append the instance to its past knowledge and retrains again. Three classification model structures that make use of a support classifier (which can be thought of as a short term auditory memory) are introduced to attempt to decrease the drop down of accuracy overtime. These models are Shared base-knowledge life-long model, Separate base-knowledge life-long model, Separate base-knowledge epoch-based model. The models investigate the accuracy behavior of unsupervised learners. Six different classifier algorithms are monitored for their behavior and the results of the models are compared to the offline learning case for the same algorithms. The performance in terms of accuracy and training time of each classifier is also compared within different feature set frames and different amount of data per training set frame. Audiovisual attention is then introduced through a two level architecture and three phases of learning. The first level of the architecture is comprised of the auditory module and the visual module. The auditory module operates with the Separate base-Knowledge life-long model suggested for incremental learning. The visual model uses a pre-trained (in an offline manner) object recognition model of deep neural network with a Single Shot Multi-Box Detector (SSD) architecture with the tensorflow object detection API. The second level module makes use of a meta classifier that operates on the classes labels probability of the low level modules in order to fuse the audiovisual information. The meta classifier is evolved developmentally in three phases. The first phase (Exposure) establishes the base knowledge for the meta classifier and realizes the audiovisual "binding" concept. In this phase the symbolic representation of an audiovisual object is realized through concatenating the label probability vectors of the auditory and visual modules to make up a meta feature vector. This way the association is introduced via correlation between label probability (low level modules decisions) values of the two sensory models. The meta classifier is trained with the meta feature vectors in an offline manner at this stage. The second phase (Expansion) attempts to expand and change the nature of a pre-established class in the first phase through enumerations of mismatched audio visual information. Mismatch events are not forced on the system but are rather taken when the meta classifier makes a false classification of a meta audiovisual instance. Therefore, learning at this stage is supervised by the caregiver, where the system is told of its false labeling and informed of the true label in order to enumerate the instances correctly. The last phase (Reappraisal) attempts to equip the system with the ability to correct its false guess of an instance through previous knowledge of related instances and via storing the wrong guessed instances in memory to go back to later after more knowledge has been obtained. For this phase two methods (subroutines) for correction are used. The first subroutine looks at the past instances in the stream of which the instance at hand was labeled falsely. It takes the maximum vote over the labels of these instances and applies it to the instance at hand. If the first subroutine fails (or if the instance in question is the first in the stream) then the second subroutine tries to infer the label by looking only at the audio meta information or the visual meta information. The subroutine tries first to decide which of these two information vectors is more likely to be accurate (or less noisy). Then, it picks the label whose audio/visual association (that was established in the exposure phase) is voted by the more likely to be accurate among the two types of meta information. The communication mechanism by which the low level modules and the second (meta) module communicate and broadcast information is maintained by the topic/message architecture of ROS. The timestamp feature is utilized for audio and visual capturing to avoid synchronization issues between all modules and insure that all audiovisual information being fed to the system correspond to each other. The system was tested in two parts. The first part addresses the audio classification by incremental learning problem. The second part addresses the audiovisual integration and audiovisual object classification problem. First, for the incremental learning models test, the auditory data were tested with different feature sets that consist of Mel Frequency Cepstrum Coefficients (MFCC), delta MFCC, energy filter banks and a range of other features that include spectrograms and Chroma-grams. Of those, the performance accuracy and training time were compared. Single MFCC features were found to be efficient since they train the fastest and their accuracy is comparable to a more complex feature set. Then, the support based incremental learning models were tested for their behavior. Results showed that most of the classifier such as support vector machines and random forests experienced a lift in performance accuracy with models introduced as opposed to regular incremental learning or even offline learning in the case of multilayer perceptron. Both the Shared Base-Knowledge lifelong model and Separate Base-Knowledge lifelong model performed well and obtained good results. The Separate Base-Knowledge epoch-based model however failed to achieve a good performance except for the case of k nearest neighbor algorithm. As for the audiovisual integration problem, the learning phases were tested with a custom made test set that includes the fused auditory and visual features obtained (as the labels probability ) from the lower level auditory and visual modules. The exposure phase achieved an accuracy of 89% on the test set. The expansion phase failed however, to achieve a good performance and the accuracy fell down to 7% because the meta classifier classified all incoming instances as a "don't-care" class after expanding said class. The reappraisal phase was thus mounted on the exposure phase directly. Its defined subroutines and behaviors managed to correct the false label estimations on the test set and the end accuracy was lifted to 100%.