Automated molecule generation using graph neural networks and deep reinforcement learning

The idea of building molecular structures with certain desirable properties underlies several major problems in drug discovery and materials science. However, the chemical space is so vast that it is impossible to explore the entire space with the available computational resources today. Therefore, finding a new drug can be very costly in terms of time and money. Fortunately, advances in computer hardware and software, combined with increasing data in recent years, have led to a revolution in deep neural networks affecting areas ranging from natural language processing to computer vision. Deep learning has influenced several fields in drug discovery, including the design of new pathways for the prediction and optimization of molecular properties. Due to these developments in deep learning and the increase in large databases that can be used for molecule generation, encouraging results have been obtained in recent years. In this thesis, a model that can generate new molecules for certain desired properties has been developed. This model consists of a combination of two different models. The first model has a Deep Reinforcement Learning architecture that handles the generation of molecules. The other model is a Graph Neural Network that generates molecular representations used in training the Deep Reinforcement Learning architecture. Two different architectures are used for this Graph Neural Network model. The first of these architectures is the Graph Convolutional Network and the other is the Graph Attention Network. By using the proposed model, molecule optimization and generation were carried out for basically five different jobs. These tasks include optimization of the QED value, optimization of the penalized LogP value with similarity constraint, optimization of the QED value with similarity constraint, optimization of the docking score for the desired target protein receptors (F2, PPAR, JAK2), and optimization of the random starting molecules using the trained model in the previous task for which the docking score is optimized. To compare this proposed model, MolDQN, a model with the best results in similar tasks in the literature, was preferred as the baseline method. This baseline method has the same Deep Reinforcement Learning architecture but uses the Morgan Fingerprint as the molecule representation while being trained. The results of the conducted experiments show that the molecules generated using the Graph Neural Network architecture were significantly more successful than the molecules generated using the Morgan Fingerprint method.