Search engine indexing is the process that sites such as Google use to organize the internet. Whether selling products or promoting content, attracting visitors to your website is critical for online success. SEO (Search Engine Optimization) has been a buzzword for years and has led to a wide variety of theories floating around the web which try to explain how to rank first in a search on Google or Bing. While the precise details of Google’s search algorithms are not in the public domain, there are many things that we do know about the function of search engines which can act as a guide to tailor your website in a way that will provide the best opportunity for success.

What are crawling and indexing?

Every time that you type some words into the search bar on Google, a complex algorithm attempts to quickly provide the most relevant content from the internet. When the search is executed, however, the engine does not scour the entire internet in real time. Rather, Google, Bing or another search engine query their previously created index of the internet. The index is created and constantly updated by web crawling algorithms which are also known as spiders. Googlebot, the Google crawler, is an automated robot which is constantly navigating the web, evaluating content and updating the Google index. Your ability to be found, therefore, depends heavily on what the web spiders see.


The connected nature of the internet, tied together by links, allows web crawlers to navigate through a significant portion of the internet. The process starts with a set of high quality sites. Next, the crawler will navigate the links contained on the site, then navigate the links on those sites and so on. Through this procedure, sites are added to the index. In this knowledge lies the answer to a common question: How do I submit to the Google index? Submission is accomplished by obtaining links to your site from other reputable sites. Note that it is also possible to submit your URL manually to Google.

This diagram of the indexing process shows how search engines organize the internet during indexing

Search Engine Indexing Process


Search engine indexing does not just create a list of URLs, but also evaluates site content and gathers many pieces of information from the site that can be used to serve relevant content when a search is executed. A major piece of this data collection is a list of relevant terms or keywords which are used on the page. A successful SEO Strategy, therefore, depends on selecting and using keywords which are relevant to searches that your target audience may execute. Another key piece of information that is gathered by the search engines is a map of links to and from the site.

What makes a site rank higher during search?

In order to rank highly during a web search, a site has to be relevant to the search phrase and important within the online community. Relevance is evaluated primarily through keywords and keyword combinations, which is why identification and use of relevant keywords is critical to SEO success. Historically, web practitioners have attempted to game the system by including keywords thousands of times in a hidden location on the page to increase ranking. The search algorithms have been developed to detect this practice, known as spamming, and to rank these sites lower. Keywords must, therefore, be used only a reasonable amount of times. The words and phrases are analyzed and a “semantic map” is constructed which identifies relationships between the different terms to improve the ability to match a site with a search phrase. For this reason, using a variety of related keywords can help to rank for relevant searches.

Because the internet is likely to contain many sites that are all relevant to a particular topic, the importance of the sites is also evaluated. In academic literature, being cited by many peers increases credibility because each citation acts as an implied endorsement. In the same way, having many links to your site will increase the importance and you will rank higher. The quality of the linking websites is also evaluated, with better links having more influence. The nature of the link itself, including the anchor text, is also considered to determine relevance. Therefore, engaging with the online community in your field to gain exposure links increases the likelihood of ranking higher during search. There are many other factors which are also taken into account in search engine ranking. Some of the top factors have been identified through an expert survey and correlation study conducted by MOZ.

Other factors that affect search engine indexing

A search engine does not yet see everything that can bee seen by a human. In determining the design of your site, it is important to remember that there are certain attributes that are hidden to the crawler. While some screening of images is done, the actual content of the image and any text contained on the image are not currently taken into account. It is therefore important to populate the titles, attributes and descriptions of images to be included during the indexing process. This same principle also holds true for other forms of media, such as video or audio clips.

Content which is not displayed directly on the page, but rather through a program may also be ignored by the search engine during the indexing process. This includes Flash, JAVA, AJAX and embedded plug ins. If your site contains these advanced features, it is important that your SEO strategy and keywords are not buried within the program where the search engine spider will not locate them. Also, frames which are used to display another websites content may also be ignored.

While search engines are incredibly complex, they perform 2 main roles, indexing and ranking. By understanding that keywords and links are two of the main pieces of data that are captured during the indexing process, content and SEO strategies can be tailored to ensure that your site is both relevant to searches that are performed by your target audience and important in the online community. Successful implementation of this will yield significant benefits in achieving your online goals.