Multi-object graph symbol network: Goal-oriented planning through learned compound object symbols

The assembly and disassembly of objects in robotics often relies on policy learning methods and symbolic planners. However, these approaches face significant challenges, such as the need for large amounts of data and expert-defined symbols, which limit their generalizability. The concept of affordances, which refer to the potential actions provided by the environment, widely studied in robotics domain in the aspect of active exploration, solving the need for supervision. Similarly, high-level symbol exploration has been extensively studied for learning and transforming symbols into symbolic planners for high-level tasks. Unlike previous works that focus on single or paired object interactions, our study addresses the affordances and symbols of compound objects with varying shapes and sizes. Our study proposes two novel models: the Multi-Object Graph Affordance Network (MOGAN) for learning affordances of compound objects, and the Multi-Object Graph Symbol Network (MOGSN) for exploring symbols of single and compound objects through effect predictions. By representing the compounds as graphs, we preserve spatial relations and propagate information effectively, enabling the prediction of affordances and symbols of complex structures. Graph Neural Networks (GNNs) are utilized to learn representations of these structures and their relations. MOGAN and MOGSN were tested on tasks involving convex and concave objects, showing successful affordance learning and symbol exploration. The MOGAN model was validated through successful plan generation in both simulation and real-world settings. The MOGSN model demonstrated the emergence of meaningful symbols in compound objects, leading to successful symbolic plannings.