If you’re like me, you’ve noticed how effective Google has become over the years of producing increasingly relevant results for whatever search terms are input. We’ve all heard statements about the “algorithm” that often times seems to be a creepily accurate magic rather than powerful tech science, but that in itself is a testament to the power of the tool. Search engines can use a number of techniques to produce the “best” results from search queries, and Google’s (and Bing, by extension) ranking factors are completely powered by this algorithm-based system. This process, while far more complex than can be completely described here, essentially breaks down into four distinct parts: Crawling, Indexing, Retrieval, and Ranking.
Crawling and Indexing
In order for Google to recognize what is on a website, it must use a series of robots and other tools called “crawlers or spiders” to report back what is on it. The search engine recognizes through a number of predetermined factors what dictates a relevant and high quality site, sends the bots to analyze the content, link structure, images, and anything else it deems relevant to the topic at hand. To see an example, check out the image below on the top ranked search for guides to help in playing the popular online video game “World of Warcraft” (actual search term – “World of Warcraft guides”):
Google’s bots have crawled various portions of this website to deem it to be the most relevant result for the above search string. However, it likely deemed the areas in red to be the most important on the page. The links at the top, various sections across the page that actually include the exact search terms, and links to the content most commonly accessed.
After Google’s spiders scrape the info it needs from the site’s pages, it will use this info and others from the results in similar search terms to create an indexof these terms. Essentially, this is a catalog of everything it could gather and the levels they relate to each other. The page structures, sitemaps, anchor text, and ad blocks are all collected and held in databases for the monumental tasks of retrieval and ranking.
Retrieval and Ranking
According to Moz, ranking “refers to the process search engines use to determine where a particular piece of content should appear” on search results. Let’s take our World of Warcraft guides example from earlier. When entering this string into the search bar and progressing to the results page, the retrieval process is actively being utilized. See below for the results:
We see above that the search has retrieved information from a number of relevant sites (some are from the same main site with various relevant pages) and even a few YouTube videos related to the same search. The order in which this all appears, influenced by the relative authority of the site, popularity, citation analysis, and trust the search engine has in it all dictate the position on the page that the results show. This is the ranking process in action, actively utilizing the semantic map it has created.
Of course, the process is far more complex than even the best SEO tools are able to completely understand. On top of this, Google’s algorithm is constantly changing and becoming more powerful, thus putting any assistance or active work a full step behind where Google may be on any given day. However, understanding the process in which this all occurs can allow Marketing and Development teams an edge on the competition in the constant pursuit of being at the top of that search results page.