Automatic audio based media monitoring

Televisions are now a part of everyday life. Almost every house has at least one, in most cases more than one TV. The most popular activity of people doing at home is to watch TV in their leisure times. It can be argued that in many countries this is the case, and in Turkey, it is even worse and it can be claimed that TV viewing habit is kind of an addiction. Turkey is one of the countries with the highest number of TV viewing times with an average of 4 hours a day after the US. The fact that television is so in our lives, the advertising industry is working on this channel in great measure, and it is still the most invested media channel in all the circles. Although trends with the evolving technology are beginning to shift towards mobile and internet, TV ads for advertisers will continue to be indispensable. In a study conducted by PwC, projections of 2016-2020, television advertising expenditures will rise by \$74.7 billion, exceeding expenditures for newspapers, magazines, brochures, internet, electronic mail, social media and radio advertising. TV commercials are more effective than other channels in terms of influencing consumers and affecting sales figures, raising brand value. Because advertisers put so much money on TV advertising, one must measure to determine how effective the resulting advertising campaign is. That is why marketers want to evaluate the success of their advertising campaigns on television from a variety of perspectives such as the number of tracks, reach of target audience etc. It is also a very important issue that should be measured whether the advertisements are published as stated in the agreements with the publishing organizations. For example, parties can agree on an ad serving six times a day, with at least two of them being between 6 and 10 pm. In this case, it is necessary to report the broadcast and the hours at which the advertisements are aired. Firms often go out of their way to deal with staff at their own discretion, media follow-up centers or advertising agencies. A technical analysis of an advertising campaign may include the following: the time and frequency of the ad's release, the number of campaigns, spot number and duration of the first time it is published, the rate of return, the number of tracks, and most preferred channels. Artists who produce copyrighted works of art in the visual and auditory media, such as artists, musicians, speakers, do not want to use their contents without permission. Because of the violation of rights arising from unauthorized use, copyright holders can claim damages from publishers. So, for example, music companies follow the situation of unauthorized playing of their songs on the radio. Radio or television broadcast monitoring typically requires human effort. Advertising agencies, media research companies or even advertisers themselves use human resource to solve this problem. The processes of recording, monitoring, reading, compiling and reporting many publications in written, audiovisual media are not possible without human effort. People can easily make mistakes such as fatigue, carelessness, and may miss some results. It is possible to automate processes by taking advantage of audio-based search and matching techniques which can make tracking of ads easier, faster, cheaper and more reliable. The advertisement tracking system exploits the state of the art audio fingerprinting and advertisement detection technologies with a robust infrastructure, resulting a highly available system offering proper recording, storage and indexing of broadcasts in real-time. Despite there are companies offering similar services on a global scale, there is no such product in Turkey. In this respect such a product can bring innovation to the national market and to gain share in the market thanks to our technical superiority. Today, particularly in the field of audio processing, the products are quite successful. Smartphones, which we all use, have applications that recognize songs or sounds. Even under heavy noise, the applications on our mobile phones can easily listen to a piece of music played in a crowded café and find out what song is in seconds. It is possible to apply these developments to the advertising tracking and broadcasting monitoring field. Observing transmission channels for copyrighted content, which is already already in practice, can achieve more precise and faster results with new developments in the field of audio processing. Most of the audio based query-by-sample system relies on the strength of audio fingerprinting technology. An audio fingerprint can be defined as a robust, small signature that identifies the audio from its perceptual attributes. A good audio fingerprint should have a high entropy, take up not much space in storage and the RAM, have a fair computational cost, should be robust for noisy environments and can be indexable. Query-by-sample searching with auditory cues produce results with fairly satisfactory accuracy. However, it is claimed that techniques in image processing are more complex than sound processing. Thus, in the developed system, although the television is more of a visual media, it is sufficient to search by audio. A benefit of using only audio data is that it will make possible to run the same application for radio advertisements tracking. An audio fingerprint is preferably small part of data representing perceptual properties of the given sound. A good fingerprint should have high entropy, high noise tolerance, small footprint and low computational power. When extracting audio fingerprint, these steps are followed: First, the audio signal is preprocessed. The appropriate frame size and overlap ratio are then determined. Later, which frame function to use (Hanning, etc.) is decided. Frequency-Time domain transformation is performed (Short Time Fourier Transform). Then, feature extraction is performed (spectral analysis). Finally the fingerprint is modeled, and if the entropy can be achieved in sufficient quantities, the resulting data is reduced (bit reduction). In our study, it is aimed to develop a system which detects and identifies commercials airing on television broadcasts on the fly using audio signal processing techniques. Thus, advertisers will be able to track and report their advertisements on television channels. A robust infrastructure with high availability has been developed that allows proper recording, storage and indexing of broadcasts. Audio fingerprinting and matching techniques have been studied and algorithms that provide adequate speed and accuracy have been adapted to be used in real-time streams. The proposed system has been shown to be working effectively on the actual data comparing to the existing methods. In Chapter 2, studies on monitoring TV and radio broadcasts are examined and a brief literature review is given. In Chapter 3, the concept of audio fingerprinting is described. The proposed system is given in Chapter 4 in detail. In Chapter 5, experiments with different methods are examined on the actual data set. In Section 6, the test results are evaluated and then discussed.