Analysis and prediction in sparse and high dimensional data with using dynamic social networks

Machine learning techniques and applications of these techniques became very popular after the incremental of different data sources and with the ease of accessing the data. Prediction models can be trained with using these datasets which are collected from different sources. In addition, natural language processing techniques are also very useful for data mining and information extraction on text based data. In this study, with using natural language processing techniques, a large collection of news and social media data is analysed and feature sets are created with results. Then, a scalable prediction system for sparse and high dimensional feature sets to predict stock market movements is built with these feature sets and results. For building that prediction system, 12,560 articles from New York Times covering 1 year time period and 2,854,333 tweets from Twitter covering 4 month time period are collected. The collected data are analysed with named entity recognition, sentiment analysis and topic modelling techniques. As another output of the designed system, social networks are created and analysed according to the various range of timeframes. Feature sets are created and elastic network regression based prediction models are trained with using the natural languages processing results, analysis results and social networks. With using the proposed approach, different dataset can be analysed and different prediction systems can be created. To show an example of this, predicting direction of the Dow Jones Index, is selected as a case. Different prediction models are trained and used for predicting to stock market movements for Dow Jones Index. As a result of different sets of experiments, the models which are created with the proposed method made promising predictions. In different sets of experiments, highly accurate (up to 70.90% accuracy) predictions are made by the proposed approach. These predicted values also correlated (up to 0.2315 correlation coefficient value) with real Dow Jones Index values. Further, performance tests are made to show scalability of proposed method for various prediction models that are trained with different set of features. Experiment results show that it is possible to make reasonable stock movement prediction by integrating news and related social media data, analysing them using named entity recognition, sentiment analysis and topic modelling techniques together with prediction models which use features that are created from these analysis results.