Web scraping in ecommerce and designing a blocking prevention method for web scraping

The development of new technologies in the modern world has great consequences for the development of numerous areas in Internet and in big data environment. With this affect e-commerce has become a great online source that is widely used and has huge amount of big data, which can be analysed for future decisions. Web scraping technology is used to scrape the data from online platforms. In this study, a web scraper was created. One of the most popular and leading e-commerce web site in Turkey was taken for testing the web scraper. With the help of web scraping, you can track business processes, customer behaviour, their preferences and demand for the goods. However, online platforms have a blocking mechanism that blocks web scraping programs. That is why, the created scraper also faced blocking. Therefore, contribution to this research work was creating a method to prevent blocking, which consists of three algorithms and called "Hybrid humanoid scraper". The first algorithm was introduced, which was not blocked, but did not have a review part. After that, the second low and high frequency algorithm was introduced with a review part. By using the low frequency, the web scraper retrieved data successfully. However, with high frequency scraper was blocked. Finally, it was possible to use a high frequency by adding the third algorithm and bypass the blocking of web scraping by changing IP addresses on the e-commerce platform. By using web scraping and algorithms to prevent web scraping blocking on the web platform, ethical and legal standards were not violated.