web crawler

See WebCrawler for the specific search engine of that name.

A web crawler (also known as web spider) is a program which browses the World Wide Web in a methodical, automated manner. A web icular site over a period of time, because they access many more pages than the normal (human) user and therefore can make the site appear slow to the other users if they access the same site repeatedly. For similar reasons, web crawlers are supposed to obey the robots.txt protocol, with which web site owners can indicate which pages should not be spidered. The procedure of following links and not submitting queries to databases causes much content to be ignored: the deep web. See also: Google, PageRank, Data mining

External links

InternetAdSales.com: Robots, Spiders, Crawlers and HTTP_User_Agents - Comprehensive listing of common web crawlers
Google Dance Tool - A tool to help webmasters determine when Google's webcrawler is crawling the web

<< Previous

Word Browser

Next >>

worldcon
writing
winter solstice
whitehorse, yukon
white house
warren g. harding
whist
william withering
white elephant
battle of monte cassino
double cross system

codename garbo
codename zig zag
tricycle (world war ii)
battle of kursk
wargaming
washington
canellales
wearable computer
wilhelm von humboldt
world war i casualties
world war i reparations

world series of poker
william barnes
woodworking
william gibson (novelist)
wings of desire
wireless valley
wild strawberries (movie)
wavelength
wachowski brothers
wisconsin
summerfest

werner heisenberg
wireless telegraphy
william shockley
walter brattain
world music
wu tang clan
werner herzog
world wide web
wwf
wireless lan
wireless personal area network