Reinforcement learning based active queue management with probabilistic actions

With the increasing internet traffic density, queue management problems in routers have become more important. Classical Active Queue Management (AQM) algorithms are inadequate to adapt to variable internet traffic conditions due to fixed threshold values and parameter adjustment requirements. In this thesis, a Q-learning based AQM algorithm named Continuous Adaptive Learning Active Queue Management (CoAL-AQM) is developed to improve the performance of AQM algorithms. CoAL-AQM analyzes the network status and makes decisions not only according to the queue occupancy but also according to the protocol type of the incoming packet and the packet drop rate of the flows. CoAL-AQM uses probabilistic actions labeled according to action probabilities ranging from 0.0 to 1.0 when making pass, mark, and drop decisions for packets. In addition, CoAL-AQM adapts better to changing traffic conditions with its continuous learning approach instead of being trained for a certain period of time and then deployed. Experimental studies are carried out in the OMNeT++ simulation environment, and CoAL-AQM showed better performance than classical algorithms. CoAL-AQM maintained its performance superiority in traffic scenarios with varying densities. This thesis study shows that better queue management approaches can be provided with Reinforcement Learning based AQM approaches with different state, action, and reward mechanisms to queue management problems. CoAL-AQM offers an AQM solution that can adapt itself to different conditions, without the need for parameter tuning, and can balance delay, throughput, and packet loss ratio.