Learning from rare clicks: A comparative study of imbalance-handling techniques for deep CTR prediction

Click-through Rate (CTR) prediction is an essential but difficult task in online advertising and recommendation systems. The imbalanced nature of click data, where positive click instances are much fewer in number than non-click instances, is one of the major issues in this task. This work presents an extensive comparison of rare event handling methods combined with three state-of-the-art deep learning models for CTR prediction tasks, namely, Deep and Cross Network (DCN), Deep Factorization Machine (DeepFM), and Deep Neural Network (DNN). The rare event handling methods being considered are data-level methods such as Synthetic Minority Over-sampling Technique (SMOTE) and NearMiss undersampling, and algorithm-level methods such as class weight, PolyLoss, and Dice loss functions. The experiments are conducted on two large real-world datasets: the Avazu CTR prediction dataset, containing 1.4 million training samples and a CTR of 16.16 percent; and the Trendyol JFY Ads dataset, a 5 million training sample dataset provided by the company Trendyol, an e-commerce company, where the CTR is around 5 percent. The experimental results, validated through multi-seed Wilcoxon signed-rank tests ($n = 7$, $p < 0.05$), show that PolyLoss achieves the best mean AUC on Avazu (0.7675 with DNN, +0.44\% over baseline) and on Trendyol (0.6710 with DCN, +0.56\% over baseline). These results demonstrate that algorithm-level methods show greater consistency in performance improvements across diverse datasets and models, compared to data-level approaches.