LLM-based automatic analysis system for imbalanced classification problems

Objective: The complex and heterogeneous nature of healthcare data introduces the problem of class imbalance, which directly impacts the performance of machine learning algorithms. The aim of this study is to develop an integrated and autonomous system that automatically selects the optimal balancing method for class imbalance in healthcare data and interprets analysis results with the support of Large Language Models (LLMs). Material and Methods: Three different clinical datasets with extreme imbalance ratios of up to 1:100 were employed in this study. The developed autonomous analysis pipeline manages various balancing strategies, such as SMOTE derivatives, CTGAN, and Diffusion Models (TabDDPM), through an LLM-based loop. Additionally, a novel outlier management mechanism termed "Sigma Guardrails" was integrated into the system to preserve rare but clinically valuable cases within the dataset. Model performance was evaluated using multidimensional metrics including F1-Score, MCC, ROC-AUC, and Brier Score. Results: The findings demonstrate that the autonomous system can be successful even under conditions of extreme imbalance. Furthermore, it was determined that through the Sigma Guardrails protocol, some statistical outliers in the data were deemed clinically significant and preserved within the dataset. Low Brier scores indicated the high clinical reliability of the probability estimates generated by the system. Conclusion: In conclusion, the developed LLM-powered autonomous pipeline offers an effective solution for clinical decision support processes by minimizing technical complexity and accounting for false negative costs. Keywords: SMOTE, Large Language Models, Machine Learning, Python, Class Imbalance Problem