Analysis of spectre attacks and development of AI-based detection mechanism

Speculative execution and cache mechanisms, used to achieve high performance in today's processors, also pave the way for hardware-based vulnerabilities. These microarchitectural features allow attackers to access confidential data by manipulating the processor's predictive execution behavior, leading to attacks like Spectre. Such attacks pose a serious threat not only at the hardware level but also to the integrity of software-based systems. In this thesis, we address the detection of Spectre attacks, which lead to serious vulnerabilities in modern processors, using two different methods. In the first method, the Spectre V1 attack was implemented on the GEM5 simulator and data was collected from hardware performance counters using 519.LBM from the SPEC CPU2017 benchmarks. This multi featured microarchitectural data was compiled into a dataset containing attack and normal scenarios, and attacks were detected using LSTM, CNN, and SVM models. Experimental results demonstrate that machine learning-based and deep learning-based models can distinguish Spectre attacks with high accuracy. In the second method, it is argued that detecting cache side-channel attacks can contribute to indirectly preventing Spectre attacks. For this purpose, a dataset generated using the Prime+Probe method on an Intel processor was trained with RF, DT, CNN, and LSTM models, as well as their hybrids. Comparative analyses demonstrate that deep learning-based models, in particular, provide high accuracy in attack detection. This study presents a holistic approach for both direct detection of Spectre attacks and indirect prevention through cache side-channel analysis, demonstrating the effectiveness of HPC-based machine learning and deep learning methods in enhancing processor security.