Musical genre classification application using both acoustic and visual features

In this thesis, the application of musical genre classification has been improved by using both visual and acoustical features of the voice. Four different types of music were chosen as the most popular types of music. SIFT and SURF image descriptors are used in the design of visual features. Mel Frequency Cepstral Coefficients are used as the acoustic features. Support Vector Machine and k-nn classifier are used also as classifiers. With the development of technology, people are increasingly reliant on automatic classification devices or software. With the development of the music market, however, there is growing interest in the application of automatic musical genre classification, and the use of such systems is increasing day by day. Therefore, it is aimed to eliminate this necessity by developing the application of music genre classification. Music genres have certain characteristics in themselves. Each unique acoustic feature is distinguished from the other genres. To classify genres, people first need to know these genres. They need to listen many songs about that genre to learn it. So, they can classify them. Digitally, it is in same manner. We should train classifiers with using many songs for each genres. Then, the classifier can learn the genres and classify the test songs easily. It is not possible to classify all songs hundred percent. Because, some songs can contain not just one genre's features. They can have many genres' features. So, when classifiying, the classifier can give different results. Acoustic features of sound consist of 3 main classes. These are Timbral features, Rythmic features and Tune features. Because of combining visual features, only the MFCC property belonging to the class of the timbral features was used here. MFCC is the most used and most effective feature in sound processing. Especially in applications such as voice recognition and speaker recognition, confusion arises. It has been also used in the Music Genre Classification applications up to now. The MFCC Works in a similar way to the perception of the human ear. According to the experiments performed, the human ear perceives the sound linearly in the first 1000Hz and logarithmically in the frequencies more than 1000Hz. Inspired by this, MFCC features have been obtained. In the visual feature extraction part, first the spectrogram is obtained from the music files. Then, image processing techniques are applied to this spectrogram image to obtain visual features. One of the image descriptors used in visual features is SIFT. Image descriptors allow you to extract properties that belong to an image. The SIFT image descriptor works independently of both the image rotation and the scale when extracting features from the image. In other words, any change that may ocur in image rotation or scale does not affect the performance of the SIFT image descriptor in any way. The SURF algorithm is also used as another image descriptor. SURF is the accelerated version of the SIFT algorithm. Therefore, SURF works independently of both image rotation and scale. After the image descriptors are obtained, the bag of feature histogram is created using the vectors that come out of these descriptors. The bag of feature method used in image classification especially in recent years. The vectors obtained from the image descriptors are classified according to their similarity using the k-means method and their histograms are taken. These histograms are then used in classifier training stage. Support Vector Machine and k-nn classifier are used as classifier. The application of k-nn classifier is a sort of classification that is simple. This classifier differs from the other classifiers in that it does not need the training phase. Training data is used directly by this classifier during classification. The support vector machine is a supervised learning type of classifiers. It is the most used method as classifier in image and sound processing field. This classifier performs the classification process by finding the best hyperplane to distinguish the different classes frorm each other. The basic element used in the system realization is Raspberry pi 2 produced by Raspberry pi foundation. Raspberry pi is a card with Linux Raspbian operating system. So we can think of it as a computer with Linux installed. Therefore, it is possible to develop software by installing any computer software program(c, c++, python…). This work has been developed using python 2. Raspberry pi 2 has a 4-core ARMV7 Cortex 900MHZ processor, 1GB of RAM and an SD card slot. The conducted work has shown that the algorithm can classify four different genres successfully even with using only visual features. When used just visual features, the algorithm can classify Metal, Pop and Classic musics succesfully. However, Jaz music classification performance is lower than other types with using visual features. The acoustic features show that the classification performance of all 4 types are similar to each other. Without considering Jaz music performance, the visual features are more succesfull than acoustic feautres. For the Jaz music, acoustic features are more succesful than visual features. In the first chapter, music and music genres and their features with distinctive properties are discussed and obtaining spectrogram is mentioned. In the second chapter, feature detectors and image descriptor are mentioned. Key points are found by feature detectors on the spectrogram and features are detected by image descriptors. SIFT(Scale-Invariant Feature Transform) and SURF(Speeded-Up Robust Features) image descriptors are mentioned. In the third section, the vectors obtained by the image descriptors are group based on their similarities and the histogram is obtained based on that group. The bag of features are obtained by using histogram. In the fourth section, acoustic properties of the sound and Mel Frequency Cepstral Coeficient are mentioned. Four diffrent number of MFCC coefficients are used. 9, 11, 13 and 15 MFCC coefficients are used and their performance are compared in the conclusion part. In the fifth section, Support Vector Machine and knn are mentioned, which are used to classify both visual and sound features. Both linear and nonlinear SVM classifying algorithm discussed in that section. Also, selecting the appropriate 'k' value is mentioned. In the sixth section, implementation of the system and results are shown.