|
|
|
|
|
Web CrawlerSee WebCrawler for the specific search engine of that name.
A web crawler (also known as web spider) is a program which browses the World Wide Web in a methodical, automated manner. A web icular site over a period of time, because they access many more pages than the normal (human) user and therefore can make the site appear slow to the other users if they access the same site repeatedly. For similar reasons, web crawlers are supposed to obey the robots.txt protocol, with which web site owners can indicate which pages should not be spidered. The procedure of following links and not submitting queries to databases causes much content to be ignored: the deep web. See also: Google, PageRank, Data mining External links
|
 |
| |
|
|