Multimodal sensing for manipulation failure detection

Service robots have become more capable of performing daily tasks as their cognitive skills are improved. However, safety still remains almost an unsolved issue in autonomous manipulation. Ideally, robots are desired to accomplish given tasks with no or minimum damage to their environments and themselves. However, unexpected outcomes are inevitable while operating in unstructured physical environments due to sensory and motor failures or external factors. For example, a service robot may fail in manipulating an object due to sensor/motor misalignments, drop the object after grasping due to an instable grasp or collide with other objects during transportation due to perception errors; all of which may result in undesired effects. The robot state and the environment should be continuously monitored in order to detect anomalies or failures during task execution. To achieve this, an execution monitoring system is needed that can detect failures or anomalies in an observed state. Then, recoveries are possible after detecting failures if they are unavoidable. The focus of the thesis is on multi-modal failure detection methods. Proposed failure detection system uses past observations acquired from different sensors as a training set to create models of success and failure cases, and detects failures during object manipulation actions. Rather than relying only on a single sensor modality, it is proposed to fuse visual, auditory and proprioceptive sensory data to achieve a better detection performance in different failure cases. High level proprioceptive, auditory and visual predicates are extracted by processing each modality separately. Then, the extracted predicates are fused. In order to benefit from integrating multiple information sources, some critical issues must be addressed. In particular, difference in data formats, asynchrony between sensors, correlation or independency of modalities, and varying confidence levels of sensors make the fusion process challenging [1]. The contributions of this thesis are two fold. (1) Visual, auditory and proprioceptive sensor modalities are combined for failure detection. This is the first time that these modalities are used together for failure detection in object manipulation domains. (2) A Hidden Markov Model (HMM)-based fusion method is adopted to create temporal and probabilistic failure models on extracted novel sensory predicates. We evaluate our system on real-time tabletop manipulation tasks with a humanoid robot. Experiments on a humanoid robot for tabletop manipulation scenarios indicate that the contribution of each modality is different depending on the action in execution, and fusion results in an overall performance increase in detecting failures compared to the performance attained in all unimodal scenarios.