Tamamlanmamış kararlar ile sıralama öğrenme

Research in learning to rank has been placed on developing sophisticated learning algorithms mainly, assuming the training set as a given. However, the quality of the training set directly affects the quality of the learned ranking systems. Considering the expense of obtaining relevance judgements and the budget constraint, one can get limited number of judgements to construct a training set. Much research has been devoted to distribute the judgement effort across different queries, and efficient and effective evaluation of retrieval systems given a limited judgement budget. However, little research has been done regarding the effect of incomplete judgements in learning-to-rank and available studies do not propose a solution to the problem that can be applicable given any objective metric and any sampling distribution that used to generate the incomplete judgements. In this work, we focus on how to better utilize training sets with shallow (less) judgements per query and obtain better ranking performance using such training sets. For this aim, we generate the incomplete judgements using stratified sampling strategy and use StatAP method to compute unbiased estimates of the objective evaluation metric, i.e. mean average precision (MAP), given these judgements. Then, we use LambdaMART algorithm to train a ranking model by using the estimated and actual objective metrics. By using two different datasets, we show that estimated MAP performs significantly better than actual MAP as the training objective in learning to rank when judgements are incomplete.