Advanced modulation techniques for next-generation wireless communications

beyond necessitates innovative approaches to enhance spectrum efficiency, reduce complexity, and improve reliability. This thesis investigates the combination of index modulation (IM) and reconfigurable intelligent surfaces (RIS) with technologies such as orthogonal frequency division multiplexing (OFDM) and non-orthogonal multiple access (NOMA) to improve bit error rate (BER), spectral efficiency, and system reliability, while ensuring low complexity for practical implementation. In Chapter 2, a coordinate interleaved OFDM with in-phase/quadrature IM (CI-OFDM-IQIM) scheme is proposed to integrate OFDM with IM, achieving enhanced error performance and spectrum efficiency by reducing decoding complexity and transmitting real and imaginary components through subblock clustering. Chapter 3 introduces the RIS-aided enhanced receive spatial modulation (RIS-ERSM) scheme, which enhances spectrum efficiency and reliability by utilizing receive antenna indices in a clever manner and $M$-ary modulation, while Chapter 4 presents the RIS code IM with quadrature RSM (RIS-CIM-QRSM) scheme that integrates QRSM with CIM to improve error performance and energy efficiency. Chapter 5 proposes two hybrid-RIS-enabled enhanced reflection modulation (Hyb-ERM) schemes that enhance BER performance and spectrum efficiency, while Chapter 6 introduces the enhanced over-the-air RIS-IM (E-OTA-RIS-IM) system, leveraging RIS partitioning for OTA transmission of additional IM bits, leading to improved spectral efficiency. Finally, Chapter 7 presents a novel downlink NOMA system with passive RIS that eliminates successive interference cancellation (SIC) to reduce complexity and improve efficiency. This thesis demonstrates the potential of these innovative communication schemes to meet the stringent requirements of 6G and beyond, including higher spectral efficiency, low complexity, and enhanced system reliability.