Extracting electricity consumption patterns using big data and machine learning

Most of the activities carried out in the energy industry so far mainly focused on the generation and transmission of energy. Besides, especially with the introduction of technologies such as smart grid and Advanced Metering Infrastructure (AMI) into our daily lives in recent years, a significant increase has been observed in the number of applications carried out at the distribution and consumer levels. In terms of electricity utilities, AMI systems, and smart meters help to monitor and control the power flow more quickly and reliably by eliminating the need for on-site meter reading. As opposite, in terms of the end-user, smart meters create awareness about the effective and efficient use of energy by increasing the quality of the service received by the consumer from the electricity utility. In this thesis, short-term load forecasting and non-technical loss (NTL) detection application at the consumer level is carried out by using Big Data analytics and machine learning approaches. In the short-term load forecasting application, the load profiles and consumption patterns of the consumers are analyzed using smart meter data. Load forecasting at the consumer level requires more complex processes to be designed compared to aggregated load estimation at the distribution level due to the issues resulting from the high variability, uncertainty, and data privacy. Therefore, a hybrid deep learning model is designed for the short-term load forecasting application. The proposed model consists of 1-Dimensional (1-D) Convolutional Neural Network (CNN), Long Short Term Memory (LSTM) networks, and advanced data preprocessing methods in an integrated framework. In the advanced data preprocessing stage, density-based outlier analysis, parameter optimization, and feature extraction using 1-D CNN network operations are performed on the load profiles of consumers. The proposed hybrid deep learning model has been tested on both private and public smart meter datasets. In the NTL detection application, six different False Data Injection (FDI) scenarios were examined using the AMI-level observer meter and smart meter data of a regional electricity utility. Besides, comprehensive statistical feature extraction and selection processes are performed for abnormal load profiles using the Highly Comparative Time-Series Analysis (HCTSA) software package and Neighborhood Components Analysis (NCA) and then a deep learning-based classifier model is designed for NTL detection. Finally, the proposed deep learning model has been integrated with Big Data platforms and tested on different NTL scenarios.