Theses supervised by Prof. Dr. Ahmet Murat Tekalp
13 theses · Koç University
End-to-end rate-distortion optimization for bi-directional learned video compression
Conventional video compression methods employ a linear transform and block motion model, and the steps of motion estimation, mode and quantization parameter selection, and entropy coding are optimized individually due to the combinatorial nature of the end-to-end optimization problem. Learned video compression allows end-to-end rate-distortion optimized training of all nonlinear modules, quantization parameter and entropy model simultaneously. Most of the works on learned video compression considered training a sequential video codec based on end-to-end optimization of cost averaged over pairs of successive frames. It is well-known in conventional video compression that hierarchical, bi-directional coding outperforms sequential compression because of its ability to selectively use reference frames from both future and past. To this effect, a hierarchical bi-directional learned lossy video compression system is presented in this thesis. Experimental results show that the rate-distortion performance of the proposed framework outperforms both traditional and other learned codecs in the literature yielding state-of-the art results.
Optimizing learned image compression models for complexity and rate-distortion-perception performance
Lately, the rate-distortion performance of learned image compression models has surpassed that of traditional codecs by the virtue of recent advancements in learned entropy and context models. However, state-of-the-art learned models currently exhibit higher complexity and slower processing times compared to conventional image codecs. Furthermore, optimization of models for just rate-distortion performance as currently done does not result in the best perceptual image quality. This thesis addresses these issues. One of the contributions of this thesis is to explore the impact of the activation function on the performance of image compression, considering both objective and subjective evaluation criteria, as well as runtime efficiency. The widely used generalized divisive normalization (GDN) activation function is one of the reasons for its high complexity. Our findings reveal that the latent variables generated by hard shrinkage activation align more closely with a Laplacian distribution. Our method achieves comparable rate-distortion results, along with superior visual performance, at reduced computational complexity. The second contribution of this thesis lies in the exploration of practical approaches to the optimization of rate-distortion-perception (RDP) performance. To date, the use of mean squared error (MSE) in rate-distortion optimization (RDO) has remained the standard practice in the field of image and video compression. This has been beneficial for gauging codec performance by offering a quantitative measurement of results through peak-signal-to-noise ratio (PSNR). However, it's broadly accepted that PSNR does not accurately reflect the perceptual quality of images, making RDO unsuitable for codec optimization in terms of perceptual quality. Recently, the notion of RDP has been formally defined by Blau and Michaeli [1]. Yet, there's still a lack of practical methodology for setting the RDP function at a desired level in a feasible way. We propose a practical method to enable perception-distortion analysis by keeping the rate constant. This approach allows for a principled perceptual evaluation of the codec at predetermined bitrates. Additionally, we present a method for compressing a set of images at a desired RDP point by converting the problem to an integer linear programming model. Our experimental results provide essential insights into the practical analysis of RDP in learned image compression.
A learned post-processing model with quality-gated convlstm for video compression
Recently, significant progress has been shown in applying deep learning to video compression and enhancement in terms of coding efficiency and quality improvement. In this work, a learned post-processing network is proposed for video compression and restoration tasks, which contains Quality-Gated Convolutional Long Short-Term Memory (QG-ConvLSTM) cells to enhance the quality of the compressed video frames considering the relative quality. With the proposed QG-ConvLSTM cell-based post-processing, the network exploits and takes full advantage of the inter-frame correlation and quality fluctuation between neighboring compressed frames. Since high-quality compressed frames provide more helpful information than low-quality compressed frames, the network can adjust the input and forget gate weights in QG-ConvLSTM cells. To show the enhancement of the proposed network that uses relative quality information between frames, video frames given to the network as inputs are compressed hierarchically with different qualities using the standard VVC (H.266) codec. In the proposed network, the weights given to the QG-ConvLSTM network are determined by extracting quality-related features from compressed video frames. Additionally, there is no reference frame for the quality feature extraction, so a no-reference image quality assessment method via Transformers, relative ranking, and self-consistency is suggested. The quality enhancement performance of the proposed network is measured with frequently used metrics, namely PSNR, MS-SSIM, and VMAF.
Video deinterlacing and demosaicing by deep learning
Deinterlacing and demosaicing are commonly used techniques in the image processing pipeline for consumer video. Despite the fact that real-world video deinterlacing and demosaicing are well-suited to supervised learning from synthetically degraded data because both degradation models are known and fixed, learned video deinterlacing and demosaicing have received much less attention compared to denoising and superresolution tasks. This thesis progressively explores feature alignment, integration and reconstruction stages for both tasks tailored to their known and fixed degradation subsampling patterns. We begin by presenting our initial work of a novel multi-field deinterlacing architecture that aligns features from adjacent fields to a reference field (to be deinterlaced) by designing novel deformable residual convolution blocks with two variants of different scales. To the best of our knowledge, this work is the first to propose fusion of multi-field features that are aligned via deformable convolutions for deinterlacing. Next, based on our initial work, we propose a novel multi-field full frame-rate deinterlacing network, which adapts the state of-the-art superresolution approaches to the deinterlacing task. This model incorporates self attention mechanism with deformable convolution residual blocks to align features and additively integrate aligned features for reconstruction. In order to reconstruct odd and even fields directionally, separate reconstruction modules are utilized according to the parity of each reference. Our extensive experimental results demonstrate that the proposed method provides state-of-the-art deinterlacing results in terms of both numerical and perceptual performance. Upon all these previous work, we propose a new multi-picture architecture for both video deinterlacing or demosaicing by aligning multiple supporting pictures with missing data to a reference picture to be reconstructed, benefiting from both local and global spatio-temporal correlations in the feature space using modified deformable convolution blocks and a novel residual efficient top-$k$ self-attention (kSA) block, respectively. Separate reconstruction blocks are used to estimate different types of missing data. Our extensive experimental results demonstrate that the proposed novel architecture provides superior results that significantly exceed the state-of-the-art for both tasks in terms of PSNR, SSIM, and perceptual quality. Ablation studies are provided to justify and show the benefit of each novel modification made to the deformable convolution and residual efficient kSA blocks.
Non-linear neuron modeling using padé approximants with applications to single image super-resolution and image compression
It is fairly recent for artificial neural networks to gain extreme popularity. However, the building blocks of artificial neural networks, the perceptrons, have been in existence for more than eight decades, and convolution operation has been used in networks for more than thirty years. The popularity of neural networks comes not only from their success in solving many problems they are applied to, but also from their theoretically guaranteed convergence to the solutions under certain conditions. The universal approximation theorem tells that for any required mapping with any desired accuracy, there is a neural network that achieves it, provided with sufficient hidden units. This statement is an existence theorem; it does not specify any feature about the network. Therefore, the research community developed different strategies to fulfill the predictions of the theorem such as developing hundreds of non-linear activation functions and proposing more advanced neuron models. In this thesis, we propose a novel, more powerful and inherently non-linear neuron model, called Padé approximant neuron, or Paon in short. As the name implies, Paon uses the Padé approximant to calculate the rational function approximation on the learned locations of the input features, increasing the representation and learning capacity of the network as well as its non-linear power and receptive field. Moreover, coming in two variants as solutions for the possible singularity problem of rational approximation, Paons are able to replace, and are a super set of, previously proposed neuron models, offering adaptability in various configurations. Experiments on the single image super-resolution and image compression problems demonstrate that Paons surpass their competitors when compared in equal conditions in terms of number of parameters, and are able to bring performance increase even when direct replacement and reduction in number of layers are the cases in point.
Perceptual video restoration: Task-specific versus pre-trained large generative models
Video restoration problems are ill-posed inverse problems with a very large null space. Perceptual video restoration refers to choosing a feasible solution that is perceptually pleasing and temporally coherent. This thesis compares two paradigms for perceptual video restoration: supervised task-specific generative models and pre-trained large generative models applied in a zero-shot manner with some inference-time adaptation across diverse video restoration tasks. In the first part, we present a novel task-specific Video Super-Resolution (VSR) model trained under a perception-aware framework, which trades fidelity for spatio-temporal perceptual quality. Our architecture employs two discriminators: one that focuses on the realism of spatial texture, and the other on the perceptual naturalness of motion. This design is supported by a hypothesis called perceptual straightening of natural videos, which states that natural image sequences that appear curved in the intensity domain tend to follow straighter trajectories in the perceptual domain of human visual system. Extensive experiments demonstrate that this model achieves superior spatio-temporal perceptual quality compared to baseline approaches. In the second part, we explore a zero-shot, training-free paradigm to adapt large pre-trained image-based latent diffusion models (LDMs), for general-purpose video restoration tasks including video super-resolution, inpainting, deblurring, motion blur removal, and a combination of aforementioned tasks. While these models are rather flexible and offer quite good spatial generative capabilities, being inherently image-based, suffer from temporal inconsistencies when applied frame by frame, and mechanisms for enforcing temporal coherence remain relatively underexplored in this zero-shot setting. To address this, we introduce three contributions: (1) a cross-frame attention approach to foster feature propagation without any motion estimation; (2) perceptual straightness regularization to guide sampling in finding perceptually smoother temporal paths, and, (3) a multi-path ensemble sampling scheme that improves overall quality—including temporal consistency—during the diffusion denoising process. Quantitative and qualitative results reveal a trade-off between task-specific models and general zero-shot large-scale models: task-specific models achieve the best fidelity and perceptual scores when test degradations match training, but their performance drops under degradation kernel or dataset shift, where the zero-shot model remains competitive or superior and provides a flexible, training-free option—albeit with higher runtime. This thesis presents a detailed exploring of both paradigms, with a focus on the different approaches employed to address the challenge of temporal consistency—leveraging explicit training-based mechanisms in task-specific models and introducing inference-time temporal consistency techniques within the zero-shot diffusion framework.
Ölçeklenebilir ve çok betimlemeli video kodlama için kodlama düzenleşiminin eniyilemesi
Ölçeklenebilir bit katarının desteklediği özütleme noktalarının erimi ne kadar genişse, bu özütleme noktalarındaki sıkıştırma verimliliği de o kadar düşüktür. Hatta genelde sıkıştırma verimliliği kullanılan ölçeklenebilirlik türlerine göre değişir. Belirli bir bit katarı erimi içinde ölçeklenebilir video kodlama için en iyi kodlama düzenleşimini belirlemek amacıyla çoklu hedef optimizasyonu oluşturduk. Bu sebeple, kapsam, sıkıştırma verimliliği ve hız-bozunum verimi gibi bazı hedef fonksiyonları belirledik.Ayrıca, tekgözlü ve çok bakışlı videolar için ölçeklenebilir bir çoklu betimle video kodlama öneriyoruz. Hız uyarlaması ile DCCP üzerinde aktarılabilecek şekilde, kendi içinde ölçeklenebilir özelliğe sahip her betimleme ölçeklenebilir video kodlamaya uyumlu bir bit katarından elde edilmektedir. Betimlemeler oluşturulurkenki değişkenler ölçeklenebilir video kodlama düzenleşimini ve çoklu betimleme oluşturulma yöntemini içerir. Bu değişkenleri belirlemek amacıyla bir çoklu kriter eniyilemesi öneriyoruz. Belirlenen değişkenler elde edilen ölçeklenebilir çoklu betimleme video kodlama yönteminin belirli kriterler arasında en iyi dengeyi oluşturmasını sağlıyor, bu kriterler ölçeklenebilir betimlemelerin özütleme noktalarının eriminin en büyütülmesi, bir grup paket kayıp olasılığında özütleme noktaları erimi boyunca hız-bozunum veriminin en büyütülmesi ve artıklığın en küçültülmesi. En iyilemeyi taban bit hızı (nicemleme parametresi) ve betimleme oluşturma yöntemleri üzerinden yaptık. Eniyileme modeli istenildiğinde farklı kodlama değişkenleri içerebilmeye uyumlu. Tekgözlü ve streo videolar için bulunan Monte-Carlo benzetiminin sonuçları önerilen eniyileme yönteminin başarımını göstermektedir.
Özgürakış ağlarda servis kaliteli uyarlanır görüntü aktarımı
Multimedia streaming applications have stringent Quality of Service (QoS) requirements which cannot be always met by the best-effort Internet. To provide QoS, several QoS architectures have been explored over last two decades, but none of them has been truly successful and globally implemented. This thesis presents a novel QoS architecture for multimedia streaming based on OpenFlow, a Software Defined Networking (SDN) paradigm that has already attracted many commercial vendors and recently being deployed throughout the world. We leverage off OpenFlow's enhanced network control capabilities to deliver multimedia with QoS. On top of OpenFlow, we propose an optimization framework for dynamic QoS routing which fulfills the required end-to-end QoS by dynamically optimizing the routes of the multimedia traffic. Our extensive simulation results show that the proposed architecture and the optimization framework on routing significantly improves the QoS of the multimedia streaming compared to traditional shortest path routing in the Internet. In addition, we extend our framework for large scale multi-domain OpenFlow networks. We propose a distributed control plane architecture and present new methods for dynamic inter-domain QoS routing by addressing the messaging between OpenFlow controllers and the network scalability. We show that the proposed solution to the distributed routing closely approaches the globally optimal routing and nicely scales to large networks. We also implement a controller software to demonstrate the performance of our approach over a real OpenFlow network deployed in our campus. Our experimental results on the real network acknowledge our simulation results and show that we can guarantee seamless video delivery with little or no video artifacts experienced by the end users.
VSP (video servis sağlayıcıları) tarafından yönetilen DASH (dinamik adaptif iletim) video servisleri
Dynamic adaptive streaming over HTTP (DASH) clients compete with each other over one or more bottleneck links in a network, which results in fluctuations in TCP throughput and QoE, QoE unfairness among clients, and under utilization of the network capacity. We propose both centralized and distributed architectures for collaboration between network service provider (NSP), video service provider (VSP), and users (DASH clients) to provide DASH service over software-defined networks (SDN) with quality-of-service (QoS) reserved network slices. We show that QoS reservation alone is not sufficient to overcome QoE fluctuations per client and unfairness between clients, and clients also need to employ TCP receive window adaptation knowing their fair-share bitrate. To this effect, we propose two collaboration architectures to inform clients about their fair-share bitrates. We first present a centralized collaboration architecture between the NSP, VSP, and the users, where VSP assigns a fair-share bitrate to each DASH client. We then present a distributed collaboration architecture, where a group of DASH clients sharing a reserved network slice collaborate among themselves. The collaboration groups are identified by the VSP using information provided by the NSP, and we propose a protocol that allows clients within a group to share critical parameters with each other so that each client can estimate its fair-share bitrate in a distributed manner. We show that collaboration rather than competition between clients also improves utilization of the reserved capacity in addition to avoiding throughput fluctuations and achieving a smooth QoE. Experimental results demonstrate the superior performance of both types of collaboration over competition between the clients.
Managed video services over software defined networks
Mevcut İnternet'in en iyi çabalı hizmeti doğası, bir video hizmetinde yer alan tüm taraflar, yani ağ hizmeti sağlayıcıları (NSP), üst düzey (OTT) video servis sağlayıcıları (VSP) ve kullanıcılar (istemciler), için verimsizliklere yol açmaktadır. NSP tarafından bakılırsa, NSP'ler genellikle en iyi çabalı İnternet hizmeti için aylık bir abonelik ücreti aldığından, video hizmetlerinin artan hacmi gelirlerine katkıda bulunmaz. OTT VSP'nin perspektifinden bakılırsa, özel bir IP ağı üzerinden yönetilen IPTV hizmetlerinden farklı olarak, kullanıcılarına belirli bir deneyim kalitesi (QoE) vaat edemezler çünkü NSP'ler tarafından sunulan en iyi çabalı İnternet üzerinden servis vermektedirler. Son kullanıcılar tarafından bakılır ise, ekstra bir ücret karşılığında bir miktar QoE ile video hizmetleri alma seçeneği mevcut sistemlerde mümkün değildir. Bu tezde, bu eksikliklerin farkında olarak, farklı hizmet düzeyleri ve hizmet seviyesinde kullanıcı farkındalığı da dahil olmak üzere, açık çoklu-hizmet arası ağ oluşturma vizyonunu uygulayarak, 5G ağları için anahtar bir teknoloji olan yazılım tanımlı ağlar (SDN) üzerinde yönetilebilen video hizmeti mimarileri önerilmektedir. Öncelikle, NSP, VSP ve kullanıcılar arasında, hizmet kalitesi (QoS) ayrılmış ağ dilimleriyle SDN üzerinden NSP tarafından yönetilen veya VSP tarafından yönetilen akış hizmetleri sağlamak için merkezi ve dağıtılmış mimariler öneriyoruz. QoS rezervasyonunun, kullanıcı başına QoE dalgalanmalarını ve heterojen video istemcileri arasındaki adaletsizliği gidermek için tek başına yeterli olmadığını ve son kullanıcıların adil paylaşım bit hızlarını bilerek TCP gelen veri penceresi uyarlamalarını kullanmaları gerektiğini gösteriyoruz. Bunun için NSP tarafından yönetilen (merkezi) ve VSP tarafından yönetilen (merkezi veya dağıtılmış) hizmet modellerini öneriyoruz. İkinci olarak, NSP'lerin birden fazla hizmet seviyesi ve ilgili iş modelleri de dahil olmak üzere tek operatörlü SDN üzerinde katma değerli video hizmetleri (VAVS) sunabilmelerini sağlamak için bir video hizmeti mimarisi ve yeni bir kaynak ayırma optimizasyonu çerçevesi öneriyoruz. Bu amaçla, küçük bir grup akış için (yeni servis talepleri ve bazı mevcut olanlardan oluşan) kaynakların (yol, bit hızı ve kabul kontrolü) tahsislerinin aynı anda gerçekleştirildiği yeni bir parti optimizasyon çerçevesini sunuyoruz. Dinamik kaynak tahsislerini çevrimiçi olarak hesaplamak için, toplam NSP gelirini en üst düzeye çıkarırken, aynı hizmet düzeyine sahip bir grup talepler arasında kaynakların adil bir şekilde tahsis edilmesini amaçlayan, sezgisel grup kısıtlamalı en kısa yol prosedürünü öneriyoruz. Üçüncü olarak, her bir NSP'nin kendi ağ kaynaklarını yönetirken bireysel olarak e2e QoS garantileriyle birlikte operatörler arası hizmetler sunmasını sağlayan, çoklu operatör ağları üzerinden uçtan uca (e2e) QoS yollarının dinamik optimizasyonu için SDN üzerinde dağıtımlı bir açık değişim çerçevesi öneriyoruz. Önerilen sistemde, tüm NSP'lerin SDN denetleyicileri, birbirleriyle iletişim kurarak, kendi ağları üzerinden bir dizi QoS yolunun satış fiyatlarının bildirilmesi için iletişim kurarlar. Her bir NSP, duyurusu yapılan QoS yollarının bir alt kümesine teklif vererek müşterileri için en iyi e2e fiyat performans yolunu dinamik olarak belirleyerek, NSP ağları üzerindeki bu yollar birleştirilir ve uçtan uca yollar oluşturulur.
Video sıkıştırma için öğrenilmiş çerçeve tahmini
Motion compensation is one of the most essential methods for any video compression algorithm. Video frame prediction is a task analogous to motion compensation. In recent years, the task of frame prediction is undertaken by deep neural networks (DNNs). In this thesis we create a DNN to perform learned frame prediction and additionally implement a codec that contains our DNN. We train our network using two methods for two di erent goals. Firstly we train our network based on mean square error (MSE) only, aiming to obtain highest PSNR values at frame prediction and video compression. Secondly we use adversarial training to produce visually more realistic frame predictions. For frame prediction, we compare our method with the baseline methods of frame di erence and 16x16 block motion compensation. For video compression we further include x264 video codec in the comparison. We show that in frame prediction, adversarial training produces frames that look sharper and more realistic, compared MSE based training, but in video compression it consistently performs worse. This proves that even though adversarial training is useful for generating video frames that are more pleasing to the human eye, they should not be employed for video compression. Moreover, our network trained with MSE produces accurate frame predictions, and in quantitative results, for both tasks, it produces comparable results in all videos and outperforms other methods on average. More speci cally, learned frame prediction outperforms other methods in terms of rate-distortion performance in case of high motion video, while the rate-distortion performance of our method is competitive with x264 in low motion video.
Ölçeklenebilir video kodlaması kullanılan çok partili WebRTC video konferansı: En iyi erişim gösteren internetten, katma değer konulmuş yönetilen hizmetlere
At present, multi-party WebRTC videoconferencing can be configured in mesh or selective forwarding unit (SFU) architectures over the best-effort Internet. Scalable video coding (SVC) enables efficient service to peers with heterogenous network connections and terminals. In the mesh architecture each peer sends a separate video stream to each other peer where peer upload bandwidth capacity is not enough as the number of peers increase, and the peer with least download bandwidth capacity becomes limiting factor for the rest of the peers because of single non-SVC encoder. SVC enabled SFU architecture, where each peer sends a single upstream video to a central SFU, requires high capacity servers with high bandwidth network connection and the SFU becomes a single point of failure. Also, SFU architecture has more network resource consumption and end-to-end delay performance comapring to mesh architecture. To these effects, we propose one SVC enabled best-effort streaming architecture for mesh connection and three network service provider (NSP) managed SVC enabled multi-party WebRTC architectures with i) mesh, ii) distributed edge-SFU and iii) software-defined network (SDN) assisted IP multicast. Proposed best-effort mesh connection technique solves limiting overall service quality effect of least receiving peer by embedding motion-adaptive SVC. Yet, best-effort videoconferencing services require very quick responsiveness for variant network conditions. Proposed NSP-managed architectures solves this basic problem by bandwidth allocation for WebRTC services. NSP-Managed mesh architecture has the best service quality comparing to proposed best-effort and default WebRTC mesh implementations, however, it still suffers from required high upload bandwidth capacity of the peers. Distributed edge-SFU architecture introduces a novel SFU streaming type by using multiple SFU servers in a single videoconferencing session. NSP-managed distributed edge-SFU architecture reduces total network resource consumption and end-to-end service delay considerably comparing to default SFU implementation. Nonetheless, this architecture is dependent on distributed service deployment over edge networks. SDN-assisted IP-multicast architecture is a completely new streaming type for multi-party WebRTC services. This architecture overcomes limited uplink capacity, service deployment, and single point of failure drawbacks while having better network resource consumption and the best possible end-to-end service delay performance.
Uç erişimli ağlar üzerinden HTTP uyarlamalı canlı video akışı için YTA tabanlı kontrol edilebilir-P2P destekli CDN
Live video services over the Internet, including over-the-top (OTT) and IPTV services, are becoming ever more popular. These services are based on the client-server model using hyper-text transmission protocol (HTTP) adaptive streaming (HAS). Since there's a maximum number of concurrent streams that a server can handle, popular solutions to provide demand-scalability include content-distribution networks (CDN) and peer-to-peer (P2P) streaming. Today OTT video services employ cloud CDN solutions to cache content in distributed edge points of presence. Cloud CDNs not only ease the load on the origin server, but also accelerate content delivery with lower latency. However, hosting or renting CDN infrastructure can be costly. Recent developments in the WebRTC protocol that offers easy to use P2P media and data channels have led to renewed interest in P2P video streaming. However, the uncontrolled nature of P2P video delivery over the Internet remains as a critical problem. In the meantime, we observe two major industry trends: i) cloud service providers are moving towards network edges for lower latency and higher bandwidth access (also known as fog computing), ii) network service providers (NSP) are replacing closed and proprietary hardware-based access technologies with disaggregated and virtualized software running on edge clouds to manage their edge access networks. Recognizing the shortcomings of current CDN and P2P video solutions and the potential of SDN-based unified cloud and edge-access network technologies, we propose a hybrid P2P-assisted CDN architecture hosted at edge-access datacenters operated by network service providers. In the proposed architecture, the NSP employs an SDN-enabled cloud platform to manage each edge network domain. An important feature of the proposed hybrid service architecture is that both the CDN access by clients and P2P video streaming between clients are controlled by the network service provider within each SDN-based edge network domain to optimize video service key performance indicators (KPI). This controllable P2P-assisted edge-managed video service reduces the load on CDN servers while overcoming quality of experience (QoE) fluctuations per flow and unfairness between multiple heterogeneous video-resolution clients over a reserved network slice. Other advantages of this service include: i) better video quality, lower delay and no buffering for clients; ii) minimization of cross-ISP traffic; iii) avoiding illegal, unauthorized usage of P2P services. Experimental results show that the proposed managed P2P-assisted CDN service deployed at SDN-enabled network edge platforms performs better than the state of the art CDN or P2P services alone according to our KPI. Finally, there are no solutions to the best of our knowledge that address all of these problems in the literature using P2P-assisted services combining SDN, WebRTC and edge computing.