Examination of Python libraries used in web scraping

Data has become a crucial asset for organizations today, with the internet being the primary source of data. Web scraping allows for the automatic extraction of structured data from HTML documents. This data can be used for various purposes such as collecting product and service information, price comparison, obtaining contact details, tracking news and blogs, and analyzing customer feedback. The web scraping process refers to the automatic data transfer between a web page and a web robot. In this thesis, Python libraries used in web scraping, such as BeautifulSoup, Selenium, and Scrapy, have been examined in detail, and their suitability for different types of websites and purposes has been determined. Web scraping is a technique for automatically collecting data from websites and is currently used in various fields such as data analysis, machine learning, and data mining. HTML is used to define the structure of web pages, while CSS is used to define style properties. Python is a high-level programming language used in many fields, including web development, data science, and artificial intelligence. BeautifulSoup, Selenium, and Scrapy are the main Python libraries used in web scraping. Using the specified libraries (BeautifulSoup, Selenium, Scrapy), data scraping codes were written for various websites (real estate listing site, news site, online market, and e-commerce site), and their execution times were measured. Measurements were conducted separately on Windows, Linux, and cloud server systems. Codes written in Scrapy may contain more lines compared to other libraries but offer a more structured approach. BeautifulSoup provides quick results for simple and static sites, while Selenium and Scrapy are more effective for dynamic content. Operating systems do not significantly affect the execution times of the codes, but the codes on the cloud server ran slightly slower. BeautifulSoup is a simple and user-friendly option for beginners in data scraping, suitable for static or minimally dynamic sites. Selenium is a powerful tool for extracting data from sites with dynamic content and heavy JavaScript, capable of simulating user interactions. Scrapy, on the other hand, is a high-performance tool for large-scale projects and complex sites. Selecting the right library for scientific studies and projects is crucial for the efficiency and success of web scraping techniques. This thesis serves as a guide for scientific studies conducted using web scraping and acts as a reference for future research.