Comparison of baseline inverted index compression techniques by using a new document similarity removal method

Forming inverted indexes from datasets takes an important place in the compression process. Using the d-gap method to inverted indexes and then applying non-parameterized baseline compression methods are indispensable parts of the compression process for a long time. In this paper, baseline compression techniques and a new method named as Document-Document Similarity Removal (DDSR) were applied to various datasets, and the results were compared with the results obtained from the classical methods. As a result of the comparison, it has been observed that the compression ratios can be improved with the newly tried DDSR method, but this method may have effects in terms of disk access time. For this reason, different DDSR threshold values were determined and the most ideal usage range for the DDSR method was tried to be determined. Looking at the results obtained, it can be said that this method is promising for achieving even more successful results in the field of text compression in the future.