A word based approach for compression and compressed matching in big text data

With the popularity of "Internet of Things" and correspondingly increasing amount of data create the concept of "Big Data". NoSQL databases could be used to store and manage large data. Compression algorithms can also be used to reduce space and improve performance. The purpose of this thesis is to develop a compression algorithm that provides multiple streaming support and can be used with NoSQL databases. By using this algorithm, load on the network traffic and NoSQL databases could be reduced. The amount of data requested from the database is reduced by the algorithm's compressed search support and there is no need for the entire stream during search. In the thesis, a word-based compression algoritm named MWCA (Multi-Stream Word-Based Compression Algorithm) is developed and compared with existing algorithms in terms of speed and compression ratio. Word-based compression algorithms are included in comparison in addition to the general purpose algorithms. Afterwards, NoSQL databases in three different categories are obtained and the effect of using MWCA before writing operation on these databases is examined. Also, MWCA is added as a block compression algorithm on LevelDB which is a key-value database and performance comparison has been made with Snappy, the existing block compression algorithm on LevelDB by default. In the last stage, because the data we want to keep is document based and offers indexed search support, MongoDB was selected and performance measurement was measured for different configurations. MongoDB was selected and performance was measured for different configurations. On MongoDB read/write operation is performed on three configurations: a single node, standard partitioning and partitioning by language with given rules. In addition, these partitioning configurations have been used with three different compression Options supported by MongoDB which are None, Snappy and Zlib. MWCA could be used before other compression algorithms to increase compression ratio like other word-based methods. Therefore, the results of using MWCA as a preliminary stage for all of these later configurations were also obtained and the effect of integration of MWCA on the process was examined. As a result of the performance analysis, it can be said that the use of MWCA as a block compression algorithm or as a preprocessing algorithm gives better results in terms of ratio and time.