Sound source identification for scene analysis

Humans are successful at categorization of objects using auditory signals and outcomes of events. The ability to categorize objects and outcomes of events using auditory signals is rather advanced in humans. Thus, by increasing the awareness of people about their environment, they make more accurate and correct decisions. For example, when a person hears glass-like sounds coming from a closed box, he may decide to carry it more carefully due to risk of fracture risk. The ability to categorize objects and events using auditory signals is extremely important, but it is a difficult task in robots. In this thesis, different supervised learning methods using distinctive features from sound data were compared as part of a system for robots to clasify objects and events automatically using auditory features of environmental sounds. The developed system consists of three fundamental processes: preprocessing, feature extraction, and classification. A chain of elementary audio processing blocks constitute the preprocessing phase, which sequentially acquires the audio data from the microphone, creates audio windows (frames) and applies threshold for sound detection. Creating audio windows is quite important due to there can be sudden and unrelated changes within received sounds. By framing effects of these changes are suppressed and more robust preprocessing is provided. Thresholding process consists of two seperate thresholds. First one is onset threshold to detect active and useful sound. After that detection offset threshold is becoming active to detect end of this sound. Besides thresholding, in the preprocessing phase discrete Fourier transform is applied to obtain spectral representation of sounds. Fast Fourier transform (FFT) is used for DFT calculation. FFT algorithm is calculating exactly same results with DFT but it has lower complexity. There are there discrete but simultaneous processes are working after thresholding and discrete Fourier transform is application for feature extraction. First one is extracting features from the spectrum of sound, second one is extracting timbral features and third one is measuring length of detected sound. Timbral features consists of time-domain zero-crossings, spectral centroid, rolloff, flux and mel-frequency cepstral coefficients (MFCC) features. These feature extraction processes are applied exactly the same way in both training and testing sessions. Training and testing sessions constitute classification section of system. Training is part that machine learning algorithm creates a model from traning set to classify test set which will be given later. Machine learning algorithm is trained with labeled sounds and creates a model using information about these sounds. Then in the testing session, machine learning algorithm is used that model to classify test sets. Testing mostly repeated several times in the experiments using same model due to obtain best results in ideal situtation. In the classification session two widely used machine learning algorithm is selected to use. These are Support Vector Machine (SVM) and Multi-Layer Perceptron (MLP). In the experiments it is shown that at some level MLP became insufficient due to processing time length. From that moment, only SVM algorithm is used. In the course of this of thesis, series of experiments are performed in order to develop the system and analyze features. Experiments are evaluated in two sections. These are; 1) Laboratory experiments, 2) Robot experiments. In the first section of experiments, the main purpose is to develop a system and analyze the performance od features of sounds. In the second section of experiments, the system is applied to the robots to the measure performance of system in the real situations. A set of objects is collected to be used in the experiments. This set of objects consists of common household objects. Objects are made of five different materials and every material category comprises five different objects. There is five different materials, \textit{glass, porcelain, plastic, metal} and \textit{carton}. A total of 25 different objects are used in the experiments. The material of the objects are used as a class name indicator. Besides the set of objects, two different events are used to generate sounds out of objects. $Drop$ and $hit$ events are used not only to make sound out of objects but also as a label of sound. By doing that, a pair of labels is obtained. Five different object materials with two different events make 10 different class labels for classification. Received sounds are analyzed to determine the separative features. For analyzing, spectral and timbral features are extracted from detected sounds. Amoung related researches, there is no study using the duration of sound as a feature. In this thesis, it is found that duration of sound can be a distinctive feature between not only materials but also events. On the other hand, timbral features of sounds are useful for many other aplications, but these applications are mostly music related. In this system, another use of timbral features is investigated. Besides spectral and duration features, timbral features extract and add to feature vector to increase the success of classification. Contrary to expectations timbral features did not contribute to the classification process. For classification two different machine learning algorithms are used. These algorithms are widely accepted by the community to classify the unlabelled sounds. The first method called Support Vector Machine (SVM) is primarily a method that performs classification tasks by constructing hyperplanes in a multidimensional space that separates cases of different class labels. More spesifically, the goal of the SVM is to train a radial basis function and classify labelled sounds. Second learning method is Multi Layer Perceptron (MLP). An MLP algorithm is a feed-forward artificial neural network model that maps sets of input data onto a set of appropriate outputs. An MLP consists of multiple layers of nodes in a directed graph, with each layer fully connected to the next one. Except for the input nodes, each node is a neuron (or processing element) with a nonlinear activation function. MLP utilizes a supervised learning technique called back-propagation for training the network. The laboratory and robot experiments were performed using a computer equipped with an Intel Core @2.40GHz CPU, 16GB RAM and Sony ECM-C10 microphone. The microphone is used to capture audio signals at a standard 32-bit resolution and 16 kHz sampling rate. The audio processing chain including the traning and test phases runs on HARK, which is an open-sourced robot audition software. After the system is developed by performing laboratory experiments, system is applied to the robots. Two robots are used in the robot experiments. First one is Pioneer 3-AT, a four-wheel mobile robotic platform. Using this robot only drop sounds are tested. With this test simulation of fault detection is performed. The second robot is a 7-DOF robotic arm. With robotic arm both drop and hitting sounds were recorded and tested. The work demonstrated a feasibility study of a system which is able to discriminate material of objects and events from each other with a high accuracy. The system can be used on different cognitive robots to analyze the environmental changes and detect failures. As a result of all experiments, system proved that detection and classification of objects and events is highly successful by using auditory features.