Quality assesment of internet infrastructure using Big Data methods and logistic regression analysis

In this study, it is aimed that Internet Infrastructure Providers (IIP) determine the potential locations to be invested and the failure rates are determined at the month and/or year level. The failure rates are determined by the Big Data method as single and total failures and the locations where potential failure will occur from the existing failure rates are determined by using Logistic Regression Analysis. The meaningless data obtained from different databases were stored on Hadoop Distributed File System (HDFS) by using Big Data methods. Data analysis was performed by using pig script with stored meaningless data. Pig script has been used in data analysis to save the data sets on HDFS into a single file by combining them according to common column names, allowing complicated operations to be performed. Complicated cleanup is goal-oriented for the forecasting department by evaluating and interpreting it with Big Data logic in a meaningless view. It is aimed to form a binary result of the estimation, so the last report, which can be applied Logistic Regression Analysis, was obtained. The result file obtained on HDFS was used to determine potential fault locations with python programming language and Logistic Regression Analysis. With the Logistic Regression Analysis, the pre-determined number of failures (single and total failures) and other existing columns are evaluated as a whole and potential fault locations are determined in the new data. As a result, predetermining the locations to be invested; generated result data and potential fault locations will be obtained more clearly. This study aims to facilitate the interpretation of meaningful data through the joint use of Big Data and Logistic Regression Analysis. Keywords: big data, hdfs, logistic regression analysis, confusion matrix, pig script