M188: A New Preprocessor for Better Compression of Text and Transcription Files

ABSTRACT: Compression of natural language text files is worthwhile for communities such as Project Gutenberg in terms of their storage space and even for text messaging applications' bandwidth efficiency. Thus, there has been extensive research on preprocessing techniques. The thesis proposes a new word-based preprocessor named METEHAN188 (M188). The proposed method provides better compression of text and transcription files when concatenated with some well known data compression algorithms. M188 and state-of-the-art preprocessors; starNT, WRT, ETDC, SCDC and RPBC are compared while concatenated with PPMD and PPMonstr. M188 differs from the other methods; it has larger dictionary which provides coverage of more words, the disadvantage is that it slows down the process; it has longer alphabet which gives M188 the opportunity of assigning shorter codewords; it does not code space and punctuation characters which speeds up M188 also output a more predictable scheme. During experiments, Wall Street Journal, Calgary, Canterbury, Large, Gutenberg and Pizza & Chili corpora are used. For the files in Calgary corpus the experimental results yield that M188 can overcome all other preprocessing techniques in terms of compression effectiveness. For the files selected from the project Gutenberg and Canterbury corpora WRT+PPMonstr has 1.22% gain in over M188+PPMonstr on the average. The results showed that best two preprocessors for compression effectiveness are M188 and WRT and for timing performance ETDC and SCDC are the fastest preprocessors. Keywords: LIPT, StarNT, WRT, Universal Preprocessor, PPMonstr, M188, ETDC, SCDC, RPBC, PPM, Data Compression. …………………………………………………………………………………………………………………………