Evolving strategies for web crawler

With the rapid growth of Internet and Internet-based information, it becomes the largest and publicly accessible data source in the world. Every day millions of information available so to achieve information becomes harder. To get the correct information trusted web sites and search engines are used. Trusted web sites have links between themselves, and users can reach correct and relevant information. Search engines are using crawler to follow links between pages. The context available to such crawlers can guide the navigation of links with the goal of efficiently locating highly relevant target pages. Crawler takes seed pages from search engines and follows these links using multi-agents. After first search, the results are inserted to database and they are used for seed pages for another search. The aim is to get access more reliable information using more seed pages in a short time.