Development of follower flight controller for an unmanned aerial vehicle based on long short-term memory

In this thesis, the leader-follower problem is studied using quad-rotor unmanned aerial vehicles (UAVs) in a simulation environment. The leader-follower problem is quite important for scenarios where UAVs need to work together. Within the scope of the thesis, assuming that the leader and follower UAVs fly without any communication between them; the autonomous flight controller that allows the follower to follow the leader UAV is developed utilizing only the images taken from the camera on the follower. In this case where the follower UAV cannot receive any data such as location, speed or direction of movement from the leader UAV, a solution based on Long Short-Term Memory (LSTM) capable of processing time series data is proposed to be able to develop this controller by using only camera images. In this solution, the aim is to estimate the forward velocity and lateral velocity required for the follower to follow the leader and height difference of the follower relative to the leader using camera images and to provide control inputs to the follower UAV according to this estimated information. To achieve this goal, the dataset required for training the LSTM-based model was prepared in the Microsoft AirSim simulation environment by flying the leader and follower UAVs on various routes while the follower follows the leader. In this dataset, the images taken from the camera on the follower UAV throughout the flight were recorded together with simultaneous forward velocity and lateral velocity of the follower and the height difference of the follower relative to the leader. In order to increase the performance of the LSTM-based model, the images taken from the camera were passed through a time-distributed convolutional neural network (CNN) before being fed to the LSTM layer. In this part, MobileNet was preferred because it is a low latency model. In addition, since the spatial relationships are important as the operations are performed on the images in the problem under consideration, the LSTM layer was selected as the Convolutional LSTM (ConvLSTM) layer in the literature, unlike the normal LSTM. Using the prepared data set, 3 separate models were trained to estimate the forward velocity and lateral velocity that the follower should have, and height difference of the follower relative to the leader. While these models were used in the testing phase, the estimated forward velocity and lateral velocity information were fed to the follower as the target speed. The estimated height difference information was used to determine the target height at which the follower should fly. The performance of the controller developed in this way was tested on routes where the leader UAV (1) flies at a fixed altitude by maneuvering to the right and left, (2) flies at a fixed altitude in a way that draws a figure of 8 when viewed from above, and (3) flies on a circular, spiral-shaped orbit that also includes elevation. The results obtained show that the LSTM-based controller can successfully estimate both the forward and lateral velocities in the horizontal plane and the height changes in the vertical axis. Thus, the follower followed the leader by reliably adapting to its complex and dynamic movements. This study makes a significant contribution to the literature with an autonomous leader-follower system that does not require direct communication and works only with visual data. This approach provides a robust solution against communication interruptions, especially in multi-UAV systems, and increases the applicability in the field.