Performance evaluation of library discovery tools

Libraries subscribe to tools that allow all their print and electronic collections to be browsed through a single search engine. These tools, called discovery tools, or discovery services, and mostly preferred by university libraries, continue to show diversity and development. Issues such as the scope of discovery tools, the relevance of the results they bring to search queries, and the diversity of topics they provide, are among the important indicators which determine the performance of these tools. The aim of this study is to analyze the EDS, EKUAL Keşif, and Piri Discovery Tool, in terms of the scope, ranking quality, and topical diversity index, and to compare the results within that framework. For this purpose, the iSearch test collection prepared from the articles in arXiv, and the search terms used in this collection, were run as queries in the evaluated discovery tools. The number of results returned to the queries was determined, and the extent to which these results matched articles in the iSearch collection was analyzed. The articles in the iSearch collection, which were randomly selected by experts and evaluated for relevance, were checked against the results returned from these queries, and the ordinal rank in which each result was placed was determined. The ranking quality was measured by calculating the DCG (Discounted Cumulative Gain) and the NDCG (Normalized Discounted Cumulative Gain) values of these rankings. In addition, the Shannon Diversity Index of the first 20 results returned to the queries was calculated, and compared in terms of topical diversity. As a result of this research, it was revealed that, in the first 10,000 results, that EDS and EKUAL Keşif produced results of the same quantity and ranking order, while the Piri Discovery Tool provided access to more relevant articles than the two other discovery tools. In addition, it was determined that more relevant results were reached with higher rankings by the Piri Discovery Tool, but these results were ranked within a wider range. While the discovery tools showed similar performance in terms of ranking quality, it was determined that EDS and EKUAL Keşif provided more topical diversity than the Piri Discovery Tool. The results of this research can be a reference point for university libraries which want to make the most of discovery tools, and to identify areas which might need improvement in these tools. Keywords Discovery tools, EBSCO Discovery Service, EKUAL Keşif, Piri Discovery Tool, performance evaluation, relevance ranking, Shannon Diversity Index