Development of artificial intelligence based modulation recognition method in next generation communication systems

The application of deep learning (DL) and machine learning (ML) techniques for automatic modulation recognition (AMR) is pivotal in advancing next-generation communication systems. Artificial intelligence (AI) is poised to play a crucial role in next-generation communication technologies. As AI continues to advance, systems such as cognitive radio (CR) will have the opportunity to operate with increased efficiency and effectiveness. The integration of AI into these systems will enable more intelligent, adaptive, and dynamic management of communication networks, enhancing their overall performance and reliability. This thesis investigates the effectiveness of various convolutional neural network (CNN)-based models in classifying modulation schemes for different system models across different signal-to-noise ratio (SNR) conditions. Datasets produced in the IQ and rθ diagram planes, represented as numbers or images in various formats, are utilized to feed the classification models. This approach allows for the assessment of how the content of these datasets impacts the performance of the classification models. Furthermore, innovative contributions to the literature are made by incorporating various index modulation (IM) techniques into classical modulation types within the scope of the classification problem. The classical modulation techniques employed include binary PSK (BPSK), quadrature PSK (QPSK), 8PSK, 16PSK, 32PSK, and 64PSK from the phase shift keying (PSK) modulation family, as well as QAM16, QAM64, and QAM256 from the quadrature amplitude modulation (QAM) family. The classified IM techniques include spatial modulation (SM), quadrature SM (QSM), and generalized SM (GSM). Utilizing a unique dataset generated on the rθ diagram plane, the study emphasizes hyperparameter optimization, transfer learning, and the impact of additive white Gaussian noise (AWGN) channels. The effects of noise have been extensively examined in various studies by selecting low, medium, and high SNR values or by analyzing a range of SNR values that encompass these regions. Key findings highlight several important points, such as the superior performance of certain models over others, the effectiveness of CNN models in distinguishing high-order modulation schemes, and the enhanced accuracy achieved through the rθ transformation from the IQ diagram plane, particularly under low SNR conditions. The studies collectively demonstrate that DL techniques, when properly optimized, significantly outperform traditional approaches in modulation recognition tasks, thereby underscoring their potential in dynamic and autonomous communication environments. Finally, the modulation classification problem was evaluated from a novel perspective within the context of the data wisdom, thereby introducing a new dimension to data analytics for addressing this issue.