Data association for object tracking in a shape-based sensory substitution system

In this thesis, a data association approach to real-time tracking of multiple 3D objects as a part of an auditory-to-vision sensory substitution system is presented. In our sensory substitution system, a depth camera is used to access the 3D structure of an environment in the form of point clouds; the point cloud is preprocessed, segmented, tracked and eventually, objects are presented to users as spatial audio in real time. One of challenging parts of the system is keeping track of the segments extracted from each incoming point cloud, which in the sensory substitution system are treated as proto-objects for auditory spatialization and sonification; it is essential for the purpose of generating an informative, clear and pleasant sound. A novel data association approach to object tracking, which allows the system to keep track of different objects or different parts of a single object, is the primary technical contribution of this thesis. To evaluate each set of associations in the presented approach, two cost functions are introduced. One makes use of the primitive image moments, while the other takes into account the pixel overlap between segments. Furthermore, three different search methods are tried for searching among possible associations; (1) a greedy approach, (2) a brute force approach and (3) a genetic algorithm. To evaluate the performance of the implemented tracker, two kinds of experiments are conducted: (1) testing the performance of tracker independently on an off-line dataset (off-line experiments) and, (2) testing the ability of the blindfolded users to localize and recognize objects in the wild using the implemented tracker. To conduct the off-line experiments, two different small data sets are collected; both of them were used to evaluate the performance of the tracker with different cost functions and search methods. For the experiment done in the wild with users, two sonification methods implemented outside the scope of this thesis, which encode the internal and external contour of 3D objects, are used to evaluate the ability of users to localize and recognize objects placed on the floor in a controlled physical environment. Using an ABA/BAB experimental protocol variant, the effect of learning during training and testing as well as order effects of experiments is investigated. On our limited data set, off-line experiments suggest that although the greedy approach is less effective than the brute force and genetic approaches in detecting splitting and merging situations, but in normal cases, they perform similarly. On the other hand, results indicate that the moment-based cost function performs better then overlap-based functions in detecting the moving objects. The result of comparison between genetic algorithm and brute force approach shows that, without using a restart strategy, the GA mostly finds a weaker association than the multi-run and also brute force approaches. The multi-run algorithm performs similarly to the brute force algorithm and most of the time better than the single-run method. Furthermore, although the brute force algorithm sometimes works fast enough when the number of segments is low, when the tracker gets more segments as input it works too slow and not real-time. The implemented multi-run GA method is capable of real-time tracking. Results of experiments in the wild show that using the proposed tracker, our system enables users to localize objects with short learning time, where participants can localize (precisely or poorly) the objects most of the time. Furthermore, using our system participates could successfully recognize objects most of the time. Participants could identify plant and ball in both encoding approaches with 93% and 100% accuracy, respectively, which is the highest recognition accuracy among all the objects. Results show similar performance between the two sonification methods.