Theses supervised by Prof. Dr. Huriye Işıl Bozma Aydın
13 theses · Boğaziçi University
Autonomous multi-robot topological spatial cognition
This thesis is concerned with topological spatial cognition in multi-robot systems. With topological reasoning, continuous space is discretized into a set of places that are spatially related. Thus, topological spatial cognition is associated with the acquisition, organization, utilization and revision of knowledge of places and their spatial relations. In turn, this can occur either through the direct experience of the robot or indirectly based on the knowledge of other robots. In this perspective, the problem is handled in three stages. First, efficient sensory data representation is studied and a model based on previously developed bubble space is presented. Next, the full range of spatial processing associated with direct experience is considered via introducing a topological spatial cognition (TSC) model. This model enables each robot to detect places, recognize them or learn them as necessary in a completely unsupervised, incremental and organized manner. The robot continually builds and utilizes its long-term spatial memory where knowledge of places and their spatial relations are retained in separate, but related parts. Finally, the problem of expanding each robot's spatial cognition based on other robots' knowledge is addressed and a model that enables each robot to merge its spatial memory with those of other robots is proposed. All of the proposed approaches are evaluated on benchmark data sets as well as on real robots. The experimental results demonstrate that robots are able to autonomously become cognizant of their surrounding through either their individual experience or that of other robots.
Appearance based self localization and navigation using place memory
This thesis is concerned with appearance based self-localization and navigation. First, we assume a place-aware robot. Such a robot can recognize a place (i.e. a particular room etc.) after having learned it based on a collection of appearances. A novel approach to self-localization is presented in this framework. In this approach, all the respective learned appearances contribute simultaneously to the location estimation - assuming them to be tagged with their respective locations during learning. With the extensive set of experiments, it is shown that the proposed approach can be used as a solution to mobile robot self-localization. Secondly, we introduce a novel approach in which the robot couples place awareness and self-localization. The robot first recognizes the place and then uses the retrieved appearances from its place memory. Experimental results demonstrate the advantages of coupling place awareness and self-localization. Lastly, the proposed approach is considered in appearance-based navigation.
From place detection to long-term place memory
This thesis is concerned with automated place detection and its coupling with long-term place memory. Long-term place memory is critical if a robot is to be place-aware as it stores the relevant knowledge for future referral. Place detection is closely coupled to place memory as it determines the appearances belonging to each place and thus plays a key role in regards to which knowledge gets retained in the long-term place memory. In this perspective, the contributions of the thesis are as follows: First, it introduces a novel approach to place detection based on coherent visual segments. Second, a new approach to place representation referred to as 'segments summary graph', is presented. Finally, it is shown that this representation can be utilized for improving the reliability of memory association.
Vision-based robotic plant growth monitoring and learning in polyculture farming
This thesis is focused on vision-based robotic plant growth monitoring and learning in polyculture farming. This type of agriculture dictates special soil preparation and farming a variety of companion plants together and hence differs from monoculture agriculture. The three-axis cartesian robot TarRob endowed with vision is used. First, its workspace is set up as an indoor polyculture garden through preparing the soil and planting a variety of companion plants. The behavioral capabilities of TarRob are extended to collect RGB-D images of these plants regularly. A polyculture plant dataset is initiated by adapting a four times daily schedule six hours apart over three months. Next, learning-based models for polyculture plant detection, plant segmentation, and leaf detection have developed. These models are primarily YOLO-based models that are subjected to further training. For this, the dataset is expanded as to include ground truth segment and leaf data. Following, the computation of the structural phenotypes is considered. A three-dimensional model of each plant is constructed via combining RGB-based detection and segmentation results with the depth data and is then used to compute the major structural phenotypes including height, width, length, area, and leaf count. Finally, the temporal nature of these phenotypes is studied from the resulting time series data. In particular, various statistical methods and learning models can be used to learn how each plant type grows. This thesis presents a comprehensive, fast, and modular approach to the automated monitoring and analysis of plant growth in polyculture farming by integrating robotics with machine vision.
Uzamsal haritalama ve analiz için uçan robot - kara robotu koordinasyonu
This thesis is concerned with the coordination of an aerial robot (AR) and a mobile ground robot (MGR) for aerial spatial mapping and analysis. Although ARs are favorable platforms for environmental mapping, their operation times are limited due to the current battery technologies. This can be alleviated through having the AR coordinate with an MGR acting as a mobile battery replenishment platform. In this thesis, this problem is studied from three aspects. First, a novel rendezvous scheduling algorithm is presented to determine where and when should the robots meet to replenish the battery of the AR. The resulting algorithms are shown to be real-time applicable in a wide range of scenarios varying with respect to landing type (dockless landing, docked landing), muling (done or not), charging times (fixed, varying), and final location constraints regardless of the mission scale while ensuring optimality for the associated mission. Second, three-dimensional spatial mapping and localization algorithms based on the sensory data as collected by the AR are formulated. Three-dimensional mapping is based on the matching of the incoming frames with the so-far constructed map. For localization, an interpolation-based method and a geometry-based method are considered. Thirdly, a real coordination system is designed. The system consists of a replaceable battery design for ARs, a battery replacement platform (BRP), and a MGR that carries this platform. The hardware and software development of this platform are initiated.
Place learning with a human tracking mobile robot
This thesis is concerned with human-guided spatial cognition. This is an important problem in human-robot interaction as it can yield human-like knowledge of places. It is a difficult problem as it requires the robot to have three capabilities that can function in an integrated manner - namely human tracking, human following, and spatial reasoning while doing so. The goal of tracking is human detection and estimating the relative position of the human as long as s/he remains within the robot's field of view. We consider a robot endowed with an RGB-D sensor and propose an approach that consists of five stages: human detection, target region selection, detection improvement, target distance calculation, and relative position derivation. Human detection is based on one of two alternative existing methods depending on whether the processing power or accuracy is of priority. In case the target cannot be found or if more than one target is found, the continuity of tracking is ensured through position estimation. The goal of human following is to ensure that the robot keeps the target human within its field of view - even if the human is bodily moving. This is achieved using a reactive navigation approach - based on previous work. As such, the robot is able to follow the target while avoiding collisions along the way. The final stage is spatial reasoning. Here, we utilize a topological spatial cognition model. In this model, a place refers to an area with spatial extent as defined by its appearances. The model works in conjunction with a place memory and places are detected, recognized or learned if necessary. Our experimental results indicate that the three capabilities can operate in an integrated manner.
Semantic scene analysis through visual exploration and learning
This thesis is concerned with the construction of objects' maps and their usage by a mobile robot endowed with a RGB pan-tilt camera. Such a representation is an important part of semantic scene analysis. However, it is a challenging task as the robot needs to be capable of object detection, scene exploration, object recognition and objects' mapping. We propose a series of approaches that address these problems so that the robot can construct an objects' map completely on its own. First, a novel approach that enables the robot to achieve both object detection and scene exploration simultaneously is proposed. In this approach, camera movements are guided by the so-far discovered object candidates. In parallel, the robot generates object candidates by tracking segments and determining spatio-temporally coherent ones. Following, an approach in which the generated object candidates are used in learning and recognition of object categories. In this approach, the robot has an evolving long-term object categories memory where the knowledge of learned object categories is organized in a hierarchical structure with each terminal node corresponding to an object category. The robot evolves its long-term object categories memory on its own through accumulating the unrecognized object candidates in its working object candidates memory. It waits for its working object candidates memory to fill up and then determines the new object categories which are then added to its long-term object categories memory. As such, the robot is able to construct an egocentric map of objects around it. In the case of revisiting a scene, a validation method that is based on comparing the objects' map of the current scene with those previously constructed is presented. All the proposed approaches are experimentally evaluated using visual data obtained from a mobile robot.
Robotic exploration and navigation on hierarchical spatial maps
This thesis is concerned with topological map building and navigation using these maps. Topological maps are compact spatial representations in which i) the continuous world is modeled as a discrete set of places and their spatial relations; and ii) what is meant by places ranges from particular locations to large spatial regions. This thesis proposes a series of approaches that altogether enable a mobile robot to evolve its knowledge of topological maps and use them in navigating among places. First, navigating reliably between two locations is considered. A reactive strategy that allows the robot to arrive at the goal location with the desired heading is proposed. Next, exploration of unknown environments is addressed and an approach in which the robot determines the nodes of the map through discovering canonical appearances is presented. As such, the robot can build a topological map of its surroundings autonomously. Following, the integration of this knowledge with the existing knowledge in the robot's map memory is considered. As knowledge accumulates, state-of-the-art approaches face real-time applicability issues. This is addressed by proposing an approach based on hierarchical topological maps. The proposed approach assumes that the robot is capable of topological spatial cognition as presented previously. As such, the hierarchical map is constructed so that the level preceding the terminal nodes corresponds to places - where the concept of a place, is defined as a collection of appearances or locations sharing common perceptual signatures or physical boundaries. Finally, navigation using the hierarchical map is considered and a novel algorithm that enables the robot to generate a plan of the route between any given two nodes in the hierarchical map is developed. The proposed approaches are implemented on a mobile robot and extensive experimental results are reported.
Human-like coordination of body-assisted arm movements for object manipulation
Manipulation is an integral capability for service robots. The goal of this thesis is to design and develop an approach that enables a mobile robot to mimic human manipulation abilities. We consider a differential type of mobile robot that is endowed with an arm and gripper. The robot is assumed to have visual sensing so that it can determine the relative position of the object of interest. First, it is observed that humans exhibit various basic modes of interaction with an object of interest, including extension, flexion, gripping, release and translation. As such, the robot can be programmed to have similar capabilities through establishing the correspondence between the robot and a human with respect to the underlying manipulation mechanisms. More complex behaviors such as putting, pulling, pushing, and shaking are defined using a sequential composition of basic operations. Second, humans are observed to achieve these tasks through the coordination of their body and arm movements. For this, a control approach in which the movements of the robot body and manipulator are coupled temporally and spatially is proposed. As such, if the object of interest is within the robot's reach, then only arm movements are made. If this is not the case, the robot starts moving its body. Depending on the vicinity of the object, this may be accompanied by arm motion or not. The control algorithm results in the robot's body and arm movements to be done in a coupled manner. The proposed approach is evaluated through an extensive set of experiments involving various manipulation tasks.
Design of a social robot and safe social navigation with deep reinforcement learning
This thesis is concerned with the design and development of a social robot that can navigate around in a socially compliant manner. The importance of this problem is due to the growing demand of using robots in human-populated environments. In this thesis, this problem is addressed in two concurrent parts. The first part has focused on the physical design and development of a social robot - named as SempRob. SempRob is aimed to have a sympathetic appearance while also having a design in which its visual sensors are located appropriately for environmental sensing. In the second part, the social navigation capability of the social robot is developed. First, a novel navigation method referred to as artificial potential function with reinforcement learning (APF-RL) method. In addition, an ellipse-based representation of obstacles is developed for efficient obstacle representation. Furthermore, environmental complexity measures are defined in order to ensure that learning scenarios incorporate a range of maneuvering difficulties. Both simulation and experimental results with SempRob demonstrate that APF-RL method enables the robot to move safely and efficiently in complex environments. Following, APF-RL method is extended to Social APF-RL method so that the robot additionally respects the comfort zones of the humans while navigating. This requires the robot to detect the humans in its surroundings and to track them spatially. A deep learning based human detection algorithm is combined with a Kalman filter for this purpose. Finally, Social APF-RL method is modified to be applicable in human following as well. All the proposed methods are tested on the developed robot successfully.
Towards autonomous semantic place cognition in mobile robots
This thesis is concerned with the problem of autonomous semantic place cognition in mobile robots. The importance of this problem is due to the growing demand of using robots for service tasks. This requires the robot to construct a semantic place map, retain this model in its memory and use it as necessary. A semantic map is defined by a labeled graph with nodes corresponding to objects and edges encoding their relations. This thesis addresses this problem through introducing a complete framework for building and using semantic place maps. In the proposed framework, differing from previous work, objects are represented as point cloud objects that are detected through tracking the candidate and thus class-independent point cloud objects over the incoming RGB-D stream. The tracking is done through considering features such as position, shape, color, keypoints and neighboring objects. As the robot is tracking the point cloud objects, it attempts to recognize the respective objects through object recognition. These objects are then used to incrementally build the semantic place map. These maps can then be used in the robot's subsequent visits of the associated place. Experimental evaluation indicates the recognition performance of the semantic place map to depend on both object recognition and routes traversed during first-time visits of places.
Autonomous polyculture farming robot
This thesis focuses on the design and development of a three-axis Cartesian robot, TarRob, that has multiple behavioral capabilities and thus can accomplish multiple different tasks associated with the farming of a polyculture garden. In this perspective, the contributions of the thesis are three-fold: First, the robot's hardware and software are overhauled to resolve related problems of the previously developed robot system, as well as forming a robust framework that expands its behavioral capabilities and tasks. Following this, the robot is endowed with the ability to generate a three-dimensional topological garden map. This map is constructed based on the RGB-D data that is acquired by the robot. Finally, a novel navigation algorithm that enables the movement of the robot among the plants is proposed. This algorithm is based on an extension of artificial potential functions to a three-dimensional workspace- considering obstacles modeled as elliptic cylinders. The 3DW-APF Navigation Algorithm is then tested in both simulation environments using real RGB-D data and on TarRob. Performance results demonstrate that TarRob is able to move safely and effectively among plants even as their canopy foliage and heights grow over time.
Human following with environmental sensing and mapping
This thesis is concerned with human following by a mobile robot that also needs to navigate in a socially compliant manner. The importance of this problem is due to the increasing usage of service robots in human-populated areas. This is a challenging task because the robot needs to consider proxemics while navigating. At the same time, it needs to keep the human in sight. In this thesis, this problem is studied from four aspects. First, an extensive evaluation of the previously proposed social navigation method, Social APF-RL, has been conducted to verify that the robot's navigation is socially compliant. Following, in order to better keep the human target within its field of view, a social robot head design has been realized based on a pan-tilt mechanism. Thirdly, human following in a socially compliant manner has been considered using the developed robot head, including body-head coordination while navigating. For this, a deep learning network based on reinforcement learning has been developed. Finally, recovery in case of human target loss is considered through human tracking and environmental sensing of doors using a specially trained YOLOv8 network. All the proposed methods are tested extensively in the Gazebo simulation environment.