Analysis of the frequency distributions of query terms on document collections & per-query selection of best term weighting model

Many term-weighting models have been proposed for information retrieval but the effectiveness of each term-weighting model varies across queries (i.e., information needs of users). Thus, using a single term-weighting model to process all kinds of queries may not be appropriate for fulfilling every information need of users. Instead of using a single term weighting model, it is an empirical fact that using different term weighting models for different queries could provide an increase in information retrieval effectiveness by an order of magnitude. However, for any given query, automatically selecting the term-weighting model that could provide the highest achievable retrieval effectiveness in the current state-of-the-art of information retrieval technology is still an open and challenging research problem. This issue is, in general, referred to as selective term weighting or selective weighting function or selective retrieval model in the field of selective information retrieval. In this PhD dissertation, we will investigate a novel statistical/probabilistic approach to the selective term weighting problem, based on the frequency distributions of query terms on document collections. A term-weighting model that works well for one query, may not work well for another. We are not capable of determining or justifying in advance the best term-weighting model to use with a given query. We know little of the characteristics of queries and document collections that affect the effectiveness of term-weighting models. This PhD dissertation aims to shed some light on this mystery by analyzing the frequency distributions of query terms on document collections. All the results presented in this dissertation are fully repeatable and reproducible with data and code available online.