A learned post-processing model with quality-gated convlstm for video compression

Recently, significant progress has been shown in applying deep learning to video compression and enhancement in terms of coding efficiency and quality improvement. In this work, a learned post-processing network is proposed for video compression and restoration tasks, which contains Quality-Gated Convolutional Long Short-Term Memory (QG-ConvLSTM) cells to enhance the quality of the compressed video frames considering the relative quality. With the proposed QG-ConvLSTM cell-based post-processing, the network exploits and takes full advantage of the inter-frame correlation and quality fluctuation between neighboring compressed frames. Since high-quality compressed frames provide more helpful information than low-quality compressed frames, the network can adjust the input and forget gate weights in QG-ConvLSTM cells. To show the enhancement of the proposed network that uses relative quality information between frames, video frames given to the network as inputs are compressed hierarchically with different qualities using the standard VVC (H.266) codec. In the proposed network, the weights given to the QG-ConvLSTM network are determined by extracting quality-related features from compressed video frames. Additionally, there is no reference frame for the quality feature extraction, so a no-reference image quality assessment method via Transformers, relative ranking, and self-consistency is suggested. The quality enhancement performance of the proposed network is measured with frequently used metrics, namely PSNR, MS-SSIM, and VMAF.