Post by billywheathers on Mar 16, 2018 14:10:55 GMT
Search engines have two major functions: crawling and building an index, and providing search users with a ranked list of the websites they've determined are the most relevant.
Crawling and Indexing
Imagine the World Wide Web as a network of stops in a big city subway system.
Each stop is a unique document (usually a web page, but sometimes a PDF, JPG, or other file). The search engines need a way to “crawl” the entire city and find all the stops along the way, so they use the best path available—links.
Crawling and Indexing
Crawling and indexing the billions of documents, pages, files, news, videos, and media on the World Wide Web.
Providing Answers
Providing answers to user queries, most frequently through lists of relevant pages that they've retrieved and ranked for relevancy.
The link structure of the web serves to bind all of the pages together.
Links allow the search engines' automated robots, called "crawlers" or "spiders," to reach the many billions of interconnected documents on the web.
Once the engines find these pages, they decipher the code from them and store selected pieces in massive databases, to be recalled later when needed for a search query. To accomplish the monumental task of holding billions of pages that can be accessed in a fraction of a second, the search engine companies have constructed datacenters all over the world.
These monstrous storage facilities hold thousands of machines processing large quantities of information very quickly. When a person performs a search at any of the major engines, they demand results instantaneously; even a one- or two-second delay can cause dissatisfaction, so the engines work hard to provide answers as fast as possible.