Creating a machine learning based method for predicting defective classes in object oriented systems

Today, because of the need to reduce the increasing competition and maintenance costs, software quality assessment has a very important place in the software world. The software is available on almost all electronic devices and software faults can cause great financial and moral loss. Therefore, it is very important to understand the causes of software defects and to prevent these defects before they appear. Using a good design in software systems will reduce complexity and costs. With a good design, changing the software with new customer demands will have less impact on functionality. Software developers designing with inadequate knowledge and experience or coding without paying attention to the design cause the design to be defective. As a result of the design defects, the software system loses its flexibility. Software requirements must be fully specified at the beginning of the process. If software requirements are not completely determined at the beginning of the process, the requirements will constantly change over time. Constantly changing requirements cause software developers to incorrectly code and corrupt designs. Limited time given to software developers to complete the software system plays an important role in the development of design mistakes. As a result of the limited time given by the administrators, the software developer can not afford the time to design the software or even write unit tests. Software developers have to make software to meet the needs of the software without considering the problems that may arise in the future. For this reason, software defects occur. The need to make very flexible software and to think too much with detail and complex design are also causing software problems. When designing software, it is necessary to evaluate the qualities such as flexibility, complexity, and reusability at the same time. Where flexibility is given a lot of importance and less attention is given to other qualities, complexity is high and software is low in clarity. This causes software defects. If sufficient information is gathered and analyzed, it can lead to the design of incorrect assumptions. Designing on incorrect assumptions indicates that the design at the beginning of the process is faulty. Designing the software on incorrect assumptions causes software defects. Missing or incorrect information about the design patterns is another reason for software malfunction. Developers may have incomplete or incorrect information about design patterns, because the software developer has not used design patterns for a long time and because they have not been adequately trained in design patterns. As a result of this situation, design faults occur. Design defects are in the design of the software. These defects are found in the functional definitions, interfaces, data structures, control logic, standards. These defects, which cause reducing the qualities of a quality design, may not cause errors in the compilation or operation of the software system. However, software design defects reduce the flexibility and reusability of the software. These defects lead to the creation of logic errors in new joints made and the maintenance costs increase too much. In large software systems time and resource management is very important. Although the rate of automation of software development processes increases, the scope and complexity of the projects also increase, so resource problems continue. That is why project managers need accurate and concrete information to make decisions, plan and distribute resources. In this study, a machine learning based model was proposed to identify error-prone classes with structural defects. The first step in creating a learning-based model is to create an appropriate set of data where the existing software classes are labeled as "Clean" or "Buggy". Labeling is a very important step because it affects the model's predictability and test costs. For the labeling of a class, it was observed that the error-prone frequency in previous tests was determined to be "Clean" or "Buggy" in a time window where many versions were found. The idea is that a structurally incorrect class will cause many errors in tests. It is also thought that if an incorrect class is not changed, it can not produce any errors, but on the other hand, a healthy class can also make mistakes in some tests. But the frequency of previous errors can be used to label classes. The threshold value plays a crucial role in this approach because the defect strongly influences the performance of the prediction model and the cost of the correction process. It is known that machine-learning based classification methods perform better if the data sets are balanced with the "Buggy / Clean" ratio. On the other hand, if the number of "Buggy" classes in the training set increases, the model defines many classes as "Buggy" during detection. As a result, the number of classes that need to be tested and the cost of testing increases. In this study, it was attempted to assign the most accurate threshold value to detect the majority of error-prone classes without exceeding the reasonable limit of the number of classes to be tested. When examining the results, it seems that the classifier's learning performance (F-measure) increases as the threshold value produces a balanced data set. However, to create a balanced set of data, half of the classes in a software system must be labeled "Buggy", which is not true for real projects. If a classifier is trained with a balanced set of data, it can identify many classes as "Buggy" and produce many false positives (FPs) that increase the test costs of the project. In order to select an appropriate threshold value, the balance between the classifier's learning performance and the cost of the test process is considered. The second step in building a learning-based model for error prediction is choosing the most appropriate software metrics. The goal here is not only to increase the accuracy of the model, but also to increase its intelligibility. The metrics used by the model should be interpretable for software developers to understand the sources of defects. This information can help rebuild existing classes and prevent future mistakes. In this study, CFS (Corelation Based Feature Selection) and PCA (Principle Component Analysis) methods were used to derive the most appropriate metric subset by removing unnecessary and irrelevant features from complex data sets. CFS evaluates subsets of attributes with an intuitive method. The correlation between the attributes and the predictability of the class label is taken into account. It is expected that the metric subset obtained with CFS will be high in relation to the labels of the classes, and the relevance of the metrics is expected to be low. Thus, CFS solves the multicollinearity problem. In addition to CFS, PCA was used to replace some metrics with more meaningful metrics that could be easily collected and interpreted. When metric simplification is performed, the selectability of metrics must be easy and the metrics must be meaningful for the inferences to be achieved. Metrics such as the number of lines added are not very meaningful metrics in terms of the deductions to be obtained. Because if this value is high it affects the marking of a class as "Buggy"; This metric value will increase as errors occur in the class, and as the value increases, the class will continue to be marked as "Buggy". These types of metrics do not provide project managers with accurate information. While classes are flagged buggy, the main goal is to identify any errors that originate from structural failures. Using PCA, we find the main components of the data set. As we know, metrics with the same principal component provide information about the same thing. Thus, a metric can be replaced by another metric in the same principle component that is easier to summarize and interpret. The third step in modeling is the selection of the classification method. In this study, Random Forest and J48 methods were chosen. Many other studies have shown that these methods are successful in classifying software modules. We have also observed that the performance of the Random Forest method is higher than the decision tree method. The random forest classification method consists of a collection of tree structured classifiers. Fourth, the majority classes have been simplified in order to eliminate data set imbalance. Simplification was done by evaluating the characteristics of the projects. Generally, the collected training data contains far more healthy classes (majority) than the defective ones (minority). As a result of this unbalanced distribution, the performance of machine learning methods for minority classes is low. It is aimed to improve the classification performance by eliminating data set imbalance. To evaluate the proposed error prediction model, we worked on five sets of data produced by Lee et al. The data was collected from Eclipse's open source subprojects. The examined subjects are Mylyn, Team, JDT, Etc that includes data from several small subprojects and All, which is the combination of the four other datasets. Each dataset includes 81 metrics per class. These metrics belong to one of three different groups. The first group consists of historical metrics (HM) which show the changes since the start of the project, for example number of refactoring, number of revisions and number of bug fixes. In the second group, there are code metrics (CM) containing information about the structure of the software, such as the number of lines, coupling, and cohesion. The third group includes micro-interaction metrics (MIM) that contain developer interaction information such as the number of software developers, the number of multi-tasks, and the number of interrupts. The model was evaluated using the tree visualization tool found in Weka's J48 classifier to draw lessons on "buggy" classes. In the Mylyn project, it appears that the "NumInterruptions" metric, which indicates the frequency at which a software developer who is working on a particular task is interrupted at work. There were no errors in the classes where this metric was below a certain value (<7.65). This means that project employees should not be interrupted often while working on a task. Another effective metric is "NumEditEvent". Classes have been labeled "Clean" by the J48 classifier as this metric is zero. Indicates that an error does not occur if a class is not modified. This supports our belief in the labeling of classes. The "RatioCodeUnderstandingEffort" metric was effective in the JDT project. The meaning of this metric is the ratio of time spent browsing to total time spent for this task. This metric is directly related to the understandability of the class. There were no errors in the classes where this metric was below a certain value (<0.835). It means that classes with low complexity produce fewer errors. If we want to reduce the number of errors in a project, we need to pay attention to the complexity and understandability of the classes. In fact, many MIMs and CMs are interrelated. Most of the MIMs are also related to structural defects and show the complexity of the classes and their level of understandability. There is no metric that is effective alone in the Etc project. However, when the "TimeSinceLastTask" value is evaluated together with "NumMultiTask" and "NumInterruptions" values; it was seen that the classes were "Clean" when these metrics were below a certain value. So when employees try to do more than one job at the same time, the "NumMultiTask" value and the "TimeSinceLastTask" value will increase together. Employees in this project should not be given too much work to the developer at the same time. Also, the work of software developers should not be interrupted. The results show that our approach is successful in building learning-based models for error prediction. Our model can identify the buggy classes that produce the majority of the errors in the tests. We have also observed that different models for different software systems have to be created, because each project has its own character. The All, Mylyn, Etc and JDT data sets were measured with a simplified metric cluster. When data sets were measured using the newly obtained metric sets, some drops were observed in the results, but when considering the cost and difficulty of gathering the metrics, this drop seems to be at a reasonable level.