Meta-analysis of clustering algorithms for big data

One of the many impacts of rapid technological growth is the explosion of data. As data grows more complex, knowledge mining approaches that use clustering to extract the most meaningful insights from larger data sets are becoming increasingly popular. One of the knowledge mining approaches used as data complexity increases is clustering. Clustering is an unsupervised learning method that can group data based on similarity features and has been widely implemented in various fields. This research conducted a meta-analysis of six clustering algorithms K-Means, DBSCAN, Mean Shift, Agglomerative Clustering using a real dataset of more than 22,000 international conference participants. The dataset was processed through feature selection, cleaning, and scoring based on six key attributes. Evaluation using Silhouette Score, Davies-Bouldin Index (DBI), and Dunn Index showed that Agglomerative Clustering gave the best results (Silhouette: 0.2645, DBI: 1.5168, Dunn: 0.533), despite producing 17 clusters and consuming 5401.72 MiB of memory. Mean Shift and DEC perform quite well with equally low Silhouette Scores (-0.055) and DBI (-2.291), but very low Dunn's Index values. K-Means is more efficient in terms of time and memory but has lower cluster quality (Silhouette: 0.04319, Dunn: 0.2617). DBSCAN, although efficient, produces high noise (258 points) and the worst DBI (9,969). Spectral Clustering had the poorest performance overall (Silhouette: -0.04485, DBI: 6.52696). From this study it can be concluded that Agglomerative Clustering produces the best cluster quality, but still has the disadvantages of memory efficiency and excessive number of clusters. K-Means and DBSCAN excel in computational efficiency, but are less optimal in cluster quality. Spectral Clustering showed the worst performance with unrepresentative cluster separation. This research concludes that in the context of big data clustering cannot only be seen from one evaluation metric and the best approach is to combine internal evaluation matrices to get more optimal results.