Analyzing diabetes data set using rough clustering theory and establishing a real-time data analytics system for diagnosis of diabetes disease

This thesis aims to develop a comprehensive decision support system by integrating contemporary data science methods, statistical pre-processing techniques, machine learning algorithms, synthetic data generation systems, uncertainty-based mathematical analysis approaches, and real-time data streaming technologies for the early diagnosis of diabetes. Diabetes is one of the most critical chronic diseases in modern medicine, both due to its prevalence on a global scale and the long-term health complications it causes. Detecting the disease with high accuracy in its early stages is of great importance in preventing cardiovascular, neurological, and metabolic complications that may arise in later stages. Therefore, data-driven decision support mechanisms are becoming increasingly important in modern medicine. This thesis presents a multi-layered architecture that encompasses the entire process, from the collection of data to its analysis and the real-time presentation of results to the user, rather than merely creating a machine learning model. Thus, beyond classical classification approaches, a real-time and clinically applicable solution is targeted. The Pima Indians Diabetes data set used in this thesis served as the starting point for the study, as it contains fundamental problems commonly seen in medical data sets, such as missing data, imbalanced class structure, and measurement errors. It was determined that some physiological measurements in the data set contained zero values; since it is biologically impossible for values such as glucose, insulin, or skin thickness to be zero, these data were considered missing or erroneous. Therefore, a systematic data pre-processing process was carried out, the IQR (Interquartile Range) method was used to identify outliers, and extreme values that distorted the distribution were corrected as necessary. As a result of these operations, the statistical integrity of the data set was improved, enabling machine learning models to be trained on a more accurate basis. One of the significant problems with the data set, class imbalance, was addressed using the SMOTE (Synthetic Minority Over-sampling Technique) algorithm. As the low representation of the positive class caused the models to favour the negative class, the minority class was increased within the data space using SMOTE, creating a balanced training set. Furthermore, the CTGAN (Conditional Tabular GAN) method was applied to expand the data set, increasing it to 2000 observations. Thanks to CTGAN's ability to learn complex distributions, the synthetic data was generated in a structure consistent with the original data set, which was an important factor in increasing the model's generalisation ability. These steps created a robust database for the subsequent sections of the study. Following the successful completion of the data pre-processing stage, the study proceeded to the uncertainty analysis phase, which constitutes the second key component of the thesis. Within this scope, an explainability-based knowledge extraction was performed on the data set using Rough Set Theory (RST). RST is a powerful mathematical method, particularly suitable for working with categorical data, known for its capacity to perform reduction operations and generate decision rules in data sets containing uncertainty. Therefore, the continuous variables in the data set were divided into clinically meaningful intervals and converted into categorical form. Following the categorical conversion, RST software developed specifically in C# was used to calculate the reduct, core, positive region, and decision rules. These analyses revealed that no attribute was entirely redundant and that all variables contributed to a certain degree to the diagnosis of diabetes. The fact that the reduce results emerged in this manner demonstrates the multidimensional nature of diabetes and that it is a complex disease that cannot be explained by a single parameter. Furthermore, the decision rules generated by RST, structured as 'If ... then ...', produced an important output that addresses the need for explainability, which machine learning models often lack. These rules provide a significant advantage, particularly in enabling clinical staff to evaluate classification results more easily. Following the uncertainty analysis component, the system established subsequently transitions to a multi-layered software architecture that forms its backbone. This architecture essentially comprises three main components: (1) a C#-based user interface, (2) a real-time data transmission system based on Apache Kafka, and (3) a Java-based analysis layer. This architectural structure offers an approach that fully automates the process from data entry into the system to the recording of analysis results in the database. The user interface developed on the C# side provides a userfriendly platform for the laboratory technician to enter the patient's measurements into the system. However, instead of writing the data directly to the database using classic CRUD operations, this interface transmits the data to Kafka via the Kafka Producer component. Kafka, being a high-performance messaging platform designed for largescale data streams, plays a critical role in the real-time operation of the system. The Kafka Producer component transmits each new patient record in JSON format to the Kafka topic and initiates the data stream. The topic structure ensures that incoming data is processed in a sequential and consistent manner. The Kafka Consumer component, running on the Java side, instantly detects these messages and initiates the analysis process. This structure enables the system to operate in an event-driven manner. Once the Consumer component detects the data, the Java-based analysis layer is activated and machine learning algorithms are automatically run. This process offers a mechanism that is much faster and more scalable than traditional database queries. The machine learning algorithms (Naive Bayes, KNN, KNC, Logistic Regression, AdaBoost, and Ensemble) within the Java-based analysis layer are manually coded and optimised to operate in real-time with data streams. The manual development of these algorithms has provided greater control over performance and allowed the avoidance of the limitations imposed by ready-made libraries. During this analysis phase, the KNN and Ensemble models in particular demonstrated high accuracy performance. The Ensemble model stood out as one of the system's most successful classifiers, producing more stable and accurate results by combining the strengths of different algorithms. Whenever new data arrived via Kafka, these algorithms were automatically run and prediction results were obtained. Once the analysis process is complete, the results are saved to the MSSQL database and made instantly viewable via the user interface. Thus, the entire process from data entry to obtaining results has become fully automated, synchronised, and real-time. This architecture is considered to offer significant advantages, particularly in clinical settings where rapid decision-making is required. The multi-layered, high-performance architecture proposed in this thesis is designed not only to solve a single classification problem, but also to offer a broader perspective on how modern health informatics systems should be structured. This architecture presents a structure that optimises every stage of the process, from the point at which data is produced to the delivery of analysis results to the user. The data flow, starting from the C#-based user interface, reaches the Java-based analysis layer via the Kafka system, which enables real-time data transmission. Once the analysis is complete, the results are stored in the MSSQL database and returned to the user. Within this flow, data is not unnecessarily delayed or slowed down at any stage. Therefore, the system exhibits a structure suitable for clinical applications in terms of both speed and reliability. Each machine learning algorithm used in the analysis layer has the capacity to model different data behaviours. While the pattern-based approach of the KNN algorithm can successfully distinguish between various subgroups of health data, Logistic Regression can produce balanced results in areas where linear relationships prevail. The AdaBoost algorithm's ability to combine weak classifiers to create a more powerful model ensures that complex areas in the data set are resolved more effectively. The main reason ensemble models perform better than others is that they combine the strengths of multiple algorithms to provide more stable prediction power. This is particularly compatible with the heterogeneous nature of health data, as health data often contains complex relationships that a single model would struggle to explain. The decision rules provided by RST have made a significant contribution to the issue of interpretability, an area where machine learning algorithms often fall short. The rules generated by RST clearly show which category combinations in the data set increase the risk of diabetes. For example, it can be seen that high risk emerges when specific BMI values, specific glucose ranges, or specific age groups are evaluated together. Such rules are valuable in explaining why the model produced a specific result and stand out as an element that increases both transparency and user confidence in clinical decision support processes. One of the key strengths of this real-time end-to-end system is that all data entered by the laboratory technician is processed during the analysis process without requiring any manual intervention. Thanks to its Kafka-based event-triggered architecture, the system initiates the analysis process as soon as data is entered, produces results without delaying the user, and records these results in the database for presentation to the user. This architecture saves time, particularly in busy clinical environments, and minimises human error. Furthermore, the system's modular structure allows for the easy addition of new algorithms, the expansion of the workflow, and the updating of the database schema according to needs. Therefore, the developed system is not only a tool for diabetes diagnosis but also serves as an adaptable platform for different disease types. In conclusion, this thesis presents a modern approach to diabetes diagnosis in terms of accuracy, speed, explainability, and real-time data processing capabilities. The developed architecture integrates numerous technologies under a single umbrella, from data pre-processing to synthetic data generation, uncertainty analysis methods to real-time machine learning, thereby producing an end-to-end solution. This structure serves as both a robust reference and a high-potential starting point for systems to be developed in the field of health informatics in the future. The system offers a practical decision support mechanism that can be used in situations requiring rapid risk assessment, particularly in healthcare institutions, and assists clinical staff in making more accurate and timely decisions