Spiders, robots or crawlers are software programs that follows or crawls web links throughout the Internet, grabbing content from sites and adding them for indexing. They are programmed to follow web links from one web page to another and from one website to another. The more search engine spiders find links to your site, the more they will visit for indexing. For example, Google relies on its’ spiders to create a huge index of site and page listings.