Learning graspability of unknown objects via intrinsic motivation

Robots need effective grasp procedures to interact with and manipulate unknown objects. In unstructured environments, challenges arise mainly due to uncertainties in sensing and control, and lack of prior knowledge and model of objects. Effective learning methods are essential to deal with these challenges. One classic approach here is to use reinforcement learning (RL) where an agent actively interacts with an environment and learns from the consequences of its actions, rather than from being explicitly taught. An agent selects its actions on basis of its past experiences (exploitation) and also by new choices (exploration). The goal of an agent is to maximize the global reward; therefore the agent needs to rely on actions that led to high rewards in the past. However, if the agent is too greedy and neglects exploration, it might never find the optimal strategy for the task. Hence, to find the best ways to perform an action they need to find a balance between exploitation of current knowledge and exploration to discover new knowledge that might lead to better performance in the future. In our work, we use reinforcement learning (RL) framework for learning and incorporate competence-based intrinsic motivation for guidance in search. The complexity of reinforcement learning is high in terms of the number of state action pairs and the computations needed to determine utility values. Approximate policy iteration methods can be used to alleviate this problem based on sampling. Imitation learning before reinforcement learning is one of the methods for decreasing the complexity in RL. Furthermore, it is also used for robots learn crucial parameters in movement to accomplish the task. Frustration level of the robot is also taken into account for learning mechanism. We further extend this approach by adopting an adaptive frustration level depending on a task. Intrinsic motivation is investigated in earlier works. System of "interestingness" was proposed and curiosity concept for reinforcement learning was introduced. Intrinsic motivation was also considered as learning objective. Different from curiosity and reward functions, ideal level of frustration is beneficial for exploration and faster learning. In addition, competence-based intrinsic motivation for learning was proposed in literature. In our work, main difference is that impulsiveness is adapted into the frustration rate in order to change the learning rate dynamically based on a task in real world environment for robots. We propose an intrinsically motivated reinforcement learning system for robots to learn graspability of unknown objects. The system includes two main phases for determination of grasp points on objects and experimentation of them in the real world. The first phase includes the required methods to determine candidate grasp point pairs in simulation. Note that a robot arm with a two-fingered end effector is selected as the target platform. For this reason, grasp points are determined as point pairs. In the second phase of the system, the grasp points determined in the first phase are experimented in the real world through reinforcement learning. The following subsections explain the details of these processes. The first step in the framework is detecting objects in the scene by using an ASUS Xtion Pro Live RGB-D camera mounted on a linear platform for interpreting the scene for tabletop manipulation scenarios by a robotic arm. For object detection, Organized Point Cloud Segmentation with Connected Components algorithm from PCL is used. This algorithm finds and marks connected pixels coming from the RGB-D camera. Hence, the object's center of mass and its edges are detected to be used by the grasp point detection algorithm that finds candidate grasp point pairs for a two-fingered robotic hand. Next step is detecting candidate grasp points in the simulator. Objects are represented by their center of masses ($\mu$) and 3D edges ($H$). Then candidate grasp point pairs ($\rho=$[${p_{1},p_{2}}$]) are determined with Grasp Point Detection Algorithm. In the algorithm, initially the reference points are determined. The center of mass, the upside and the bottom side center points are chosen as references. Based on these points, cross section points coplanar with the reference points and parallel to the table surface are determined. In the next step, the algorithm detects the closest point to the reference points on the same planar and draws a line crossing with reference points and closest to it. The second step is determining the opposite point to the closest one on the same line. This procedure continues until all points are tested. The algorithm produces the candidate grasp pairs (two grasp points with x,y,z values) and orientation of each pair according to (0,0) point in 2D (x,y) plane. These grasp points are tested in the simulator for finding out only the feasible ones. These point pairs are tried in simulator environment in order to eliminate pairs which are impossible for grasping an object with robotic arm. This process is saving time, so robot can decide and learn faster. In learning process, we propose a competence-based approach to reinforcement learning where exploration and exploitation is balanced while learning to grasp novel objects. In our approach, the dynamics of balancing between exploration and exploitation is tightly related to the level of frustration. The failures in obtaining a new goal may significantly increase the robot's level of frustration, and push it into searching new solutions in order to achieve its goal. However, a prolonged state of frustration, when no solution can been found, will lead to a state of learned helplessness, and the goal will be marked as unachievable at the current state (i.e., object not graspable). Simply speaking, an optimal level of frustration favours more explorative behaviour, whereas low or high level of frustration favours more exploitative behaviour. Additionally, we dynamically change the robot's impulsiveness that influences how fast the robot gets frustrated, and indirectly how much time it devotes to learning a particular task. To demonstrate the advantages of our approach, we compare it with three other action selection methods: $\varepsilon$-greedy algorithm, Softmax function with constant temperature parameter, Softmax function with variable temperature depending on agent's overall frustration level. The results show that the robot equipped with frustration and impulsiveness learns faster than the robot with standard action selection strategies providing some evidence that the use of artificial emotions can improve the learning time. For example, when a robot plays a quick game with a human, it has to learn quickly. However, when the robot is alone, it can spend relatively more time on exploring different states. By changing the impulsiveness, the robot may dynamically control its level of frustration and therefore the time devoted for learning a particular task. Hence, the robot could behave differently in different environments and for different tasks.