Monte Carlo tree search with temporal-difference learning for general video game playing

General Video Game Playing (GVGP) is a problem where the objective is to create an agent that can play multiple video games with different properties successfully, with no prior knowledge about them. By being an important field in Artificial General Intelligence (AGI) research, GVGP has drawn a considerable amount of interest, and the research in this field got intensified with the release of General Video Game AI (GVG-AI) framework and competition. As of today, even though this problem has been approached with many different techniques, it is still far from being solved. Monte Carlo Tree Search (MCTS) is one of the most promising baseline approaches for GVGP in literature. Despite having a large set of enhancements including evolutionary algorithms, pathfinding techniques and multi-objective methods, an overall best performance still could not be achieved with MCTS in GVG-AI domain. Moreover, only a little portion of these extensions have utilized temporal-difference learning which is a widely used reinforcement learning technique especially in game domains. In this study, the weaknesses of baseline MCTS method in GVG-AI such as its shortsightedness and not being able to exploit any sorts of domain knowledge are analyzed and addressed with a novel hybrid method. With the aim of making MCTS adapt to the specific games while preserving its generality, it is enhanced with a recently developed temporal-difference learning method, namely true online Sarsa(lambda) for the first time in literature. In this algorithm, MCTS uses its own experience to perform online learning by employing linear function approximation in order to evaluate game states accurately and consequently make better decisions. Experiments have shown that the proposed algorithm significantly outperformed the baseline MCTS in GVG-AI in terms of win percentages and average scores obtained in two sets of 20 games in total with very different properties. It was concluded that incorporating an online learning in MCTS is a way to overcome most of its weaknesses and temporal-difference learning is a viable alternative to the other modifications done in MCTS for GVGP problem. Analyzing MCTS with temporal-difference learning in a larger set of games in order to develop a game-adaptive parameter tuning technique and a more robust feature extraction methods will be important extensions to this work.