Automatically generating regular expressions for data extraction from web pages using genetic algorithms

In this thesis study, an automatic regular expression generating model based on genetic algorithms was designed to extract relevant images from web pages. After the data shared on the web is extracted, it is used in many areas such as text mining, data mining, image analysis, fake news analysis. Extracting data shared on web pages is called web scraping. However, this data extraction process, which is usually carried out manually by web scraping programs or manually, causes more errors and can be more time consuming. In the literature, it is generally seen that studies on extracting text data such as news headlines, main content and tables are more intense. In this thesis, a method has been developed to extract image data of the actual news content on web pages. Unlike other methods, a regular expression is automatically generated in the model designed to access the relevant images. Regular expressions are difficult to write and require experience, making writing them by hand difficult. For this reason, by examining the HTML text of one or more relevant image data, a regular expression that allows access to the relevant images is automatically created. In the study carried out using genetic algorithms, it was aimed to produce a regular expression that provides access to the most relevant image according to the principle of elimination of the bad and survival of the best. The data set used in the study contains HTML codes of image data collected from 200 websites from 58 different countries, and there are 100 pages for each site. There are 22,682 relevant image data in the data set, which contains a total of 635,015 relevant and irrelevant image data. Each image data is encoded with 30 features. Three of these features were used in the study. In the findings obtained in the study, as a result of the tests carried out with the model designed using the Python programming language, it was observed that the rate of accessing the relevant images for 200 websites was 98%.