Continuing with our Search Engine Optimization (SEO) discussion, it’s necessary to understand just how search engines see our WordPress sites. While the computation and algorithms are complicated and proprietary to each search engine, an understanding of the basics gives us the tools needed to optimize our sites.
A brief introduction to web crawlers
Hopefully, I’m not ruining any fantasies here, but there isn’t a room full of people at Google looking through web pages and indexing the contents of different sites. That would require way too many people and cost way too much. Instead, large search engines generally implement web crawlers, programs which quickly read web page data. This process called crawling or spidering generally performs the following actions:
- Read the prominent* page data on the current page
- Read all hyper-links to other web pages and add them to a list of future pages to crawl
- Move on to the next web page
* What a particular crawler deems to be “prominent” data depends entirely on the purpose of the crawler Continue reading