Analysis of cancer dataset with statistical learning

Cancer continues to pose a significant global health challenge, underscoring the criticality of early and accurate diagnosis for enhancing treatment outcomes and patient well-being. The classification of cancer types assumes a pivotal role in tailoring treatment plans, minimizing unnecessary procedures, and optimizing therapeutic success. This thesis presents an extensive analysis of statistical learning algorithms and machine learning (ML) algorithms on breast cancer, lung cancer, and prostate cancer datasets. The primary objective was to evaluate the algorithms' performance in distinguishing between benign and malignant samples across diverse cancer types. To ensure robust and reliable results, a comprehensive steps of preprocessing techniques was implemented, encompassing data cleaning to address null values and duplicate records, data scaling for feature normalization, random over-sampling to tackle class imbalance, and an 80:20 data splitting ratio for training and testing. Additionally, cross-validation was employed to assess model generalization and robustness. The paramount importance of accurately diagnosing cancer types lies in its potential to significantly impact patient outcomes and guide treatment strategies. The results showcased impressive accuracies ranging from 95.8% using ridge logistic regression to 97.2% using lasso logistic regression for breast cancer. Similarly, ML algorithms, such as Decision Tree, SVM, Random Forest, and XGBoost, achieved accuracies between 93% using random forest to 98.6% using XGBoost for breast cancer. Additionally, lung cancer statistical learning algorithms demonstrated accuracies between 93.75% using Ridge regression to 96.87% using Lasso regression, while ML algorithms achieved accuracies from 95.83% using Decision tree to 98.95% using Random forest. For prostate cancer, statistical learning algorithms achieved accuracies between 74.11% using ElasticNet regression to 77.64% using Lasso regression, and ML algorithms achieved accuracies ranging from 63.53% using Decision tree to 75.29% using SVM. These findings underscore the effectiveness of both statistical learning and ML algorithms in cancer classification, affirming their potential applicability in real-world scenarios to advance cancer detection and diagnosis.