Analysis of electric consumption data by data mining methods and determining the right tariff

With the development of technology, which takes more place in our lives day by day, problems such as big data volume, storage and analysis arise. The development of technology also increases dependence on electricity. The increase in electricity consumption raises the cost problem. For these reasons, individuals seek the optimum cost-benefit relationship. To find the optimum cost-benefit relationship, it is necessary to make the big data obtained meaningful. Therefore, Apache Spark is a specially developed platform for big data analytics. Apache Spark is an integrated calculation engine that enables data analysis on large data using various machine learning algorithms. Apache Kafka, another data analytics system, is a messaging protocol developed as a solution to integration problems between systems, enabling real-time communication between producers and consumers. In this thesis, firstly, software controlled smart plug was developed to examine the electricity consumption information of the users and to analyze the consumption data obtained from the users. Developed this smart plug is obtained through the default data 5000's by examining the power consumption by the electricity billing system existing in Turkey are made optimum consumer type selection to the user. While selecting the consumer type, two different machine learning applications were developed by using Logistic Regression with the same data set. Real-time data analysis is performed with Spark Structured Streaming in Application-1. In application-2, real-time data analysis is performed with Apache Kafka. There are differences between these two applications in terms of approaches and technologies. Results obtained according to applications show that the proposed architecture is available for real-time data analysis.