Crawlers (or bots) are used to assemble information out there on the web. By using web site navigation menus, and reading internal and exterior hyperlinks, the bots begin to understand the context of a web page. Of course, the words, photographs, and other information on pages also help search engines like google and yahoo perceive what they’re a