Pyramid self organizing maps (psom): a new approach to analyze large data sets

The aim of this study is to develop a new method capable of processing and analyzing large data sets, by less time consumption and furthermore, to include extra features with the ability of visualizing the correlation between attributes of the analyzed data sets. Another additional goal that the study aims to fulfill is, designing and developing a software package that provides new features of data analysis and data visualization techniques, with the main purpose of interactively analyzing and visualizing the correlation between data attributes. Data analysis is the process of reading, cleaning, correcting, transforming and modeling data with the purpose of subtracting important information. Analyzing big data is an actual challenge of data analysis techniques. Self-Organizing Maps is an ANN algorithm that successfully processes data sets and visualizes the correlation between attributes. However, it is very slow when processing huge amount of data. In this study, a Pyramid Self-Organizing Maps (PSOM), based on Self-Organizing Maps algorithm, is proposed. The suggested method successfully overcomes SOM, in terms of time consumption and quality of results. PSOM is a hierarchy-based approach, where within a training process it constructs and trains several maps of several levels (several map-sizes). To further improve the performance of proposed algorithm, Batch version of PSOM is parallelized and adapted to work in multi-core environments. Parallelized Batch - PSOM (PB-PSOM) version significantly overcomes both, PSOM and SOM, in terms of execution time and in terms of quality. A software package, which implements SOM, PSOM and PB-PSOM, and includes extra user-interactive features to analyze data sets, is another contribution of this study. Suggested algorithms are tested and compared with results of classical SOM using different sizes of input data (100, 500, 1000, 5000, 10000, 50000 and 100000 input patterns) and evaluation of results is done in terms of topology preservation, data set mapping and execution time. Results have shown that PSOM decreases the amount of time required to process large data sets and in the same time, it improves the produced results. In the other hand, PB-PSOM significantly outperforms both, PSOM and SOM. Key Words: Data Analysis, Self-Organizing Maps, Large Data Sets, Pyramid Self Organizing Maps, Parallelized Batch Pyramid Self Organizing Maps