Co-occurrence—Google’s secret agent… for good?

By Miles Petty

Co-occurrence is the secret agent of internet search engines. It is so good at what it does that it’s hard to track down on Google. It is the master of ranking content above spam, the reason that brand value on the internet matters, and the way your search results give you the content you want rather than ranking the sites that manipulate the internet the best.

So, what is co-occurrence?

The term “co-occurrence analysis” is a little academic—in fact, the links on the first page of Google all focus the academic side of the term, and it takes a few tries to get at how Google itself uses it. In general terms, it is taking a collection of data and seeing when that data pairs—for example, how many times a brush and shampoo were purchased together at a drug store—and then data mining results. While it can have an impact on where retail stores place displays and when they run sales, it likely makes its biggest impact on your daily life in the influence of content in internet search.

How search results work is a complicated mix of factors such as links, keywords, clicks, and content. For much of the early years of internet search, users were able to game the system and push their sites to the top rankings by stuffing keywords all over a page and adding links from trash websites. To combat this, search engines started punishing sites with spammy links and—using co-occurrence—took steps to get better at finding brand niches online and sorting “neighborhoods” of links and relevant search terms.

How co-occurrence works in search

A typical example of co-occurrence is women’s shoes—a site may not ever have the exact words “women’s size 9 hiking boot” or even “women’s shoes on sale.” Nevertheless, someone looking to buy women’s shoes who searches these terms might be directed to that site anyway because the site has women’s shoes and uses other terms that the search engine knows refer to online shopping, like cart, shipping, and price.

Perhaps an easier way to imagine how co-occurrence works is the “people you may know” function that pops up on your Facebook feed. Underneath the profile picture of the person Facebook is suggesting you friend, the site will show how many friends you have in common. Facebook can see 1) that you are not connected, and 2) that both of you are connected to many of the same people. Having both your names show up on the friends lists of so many of the same people suggests that there is a good chance you know each other.

In the same way, search engines take words that are not connected directly or might not be considered keywords, but are often used in similar content, and associate them. For example, someone might search for “love poems” on their anniversary. While having the exact keyword search term on a site will help its rankings, simply making sure every paragraph says “love poems” will not be enough to move it to the top. Instead, names of famous poets, similar words in proximity to the keyword search term such as “passion” or “romance,” and less connected words such as “sonnet” or “verse” might be recognized by the search engine for their co-occurrence.

Co-occurrence, co-citation, and web content

To be fair, the Facebook “people you may know” example may be a better explanation for co-occurrence’s sister function, co-citation, which credits sites that do not share links, but who have common links. For example, if ESPN and Sports Illustrated do not link to each other, but the sports website The Ringer links to them both, the search engine will know they are related.

Both co-occurrence and co-citation allow a search engine to give a deeper dive into what words, ideas, and behaviors are related and increases the value of website content. Imagine it like examining website content in its natural habitat and putting similar species together, while not rewarding the wolf who has put on sheep’s clothing.

Of course, this leaves the question that if co-occurrence is so great, how important are keywords? While it may seem that keywords have been supplanted in the hierarchy of SEO tools, that hypothesis is either inaccurate or only partially right. Keyword stuffing is no longer effective, and the greatest keyword plan might not be enough to get the top Google ranking. But having no keyword strategy can also doom a site to the hinterlands of non-page one results. While search engines are trying to lump similar species together, they are doing so within the confines of the zoo that is the world wide web: a wolf won’t make it into the sheep’s pen, but putting nametags on the sheep can still be really helpful. Researching and utilizing keywords can put your site in the right niche and will still help push it up the rankings.

Competitors for good and Brand Niche

In fact, knowing what your competitors are doing and using similar keywords as they are will be a boon to your presence on search. One counter-intuitive facet of search today is how positive mentions of competitors can increase your rankings. The benefit of associating with brands in your niche is that you will increase your own brand presence and the way search engines view you. Positive mentions will give your brand identity and help establish your prominence in the industry. You don’t have to be too nice, but it does pay to be a good web citizen.

Search engines use co-occurrence not only in discovering relationships between sites as they crawl the web, but also in the terms users employ in search. When searching for something specific, a user will often search multiple variations on the same thing—going from a broad term like “women’s shoes” to a specific term like “women’s size 9 comfortable pumps”. The words used in each of the different searches will be used by the search engine to establish relationships in content going forward.

The ultimate goal of the search engines is to keep (or gain) users, and to do so they must keep their users happy. It is in the search engines’ best interest to funnel users to the information they want as directly as possible. Establishing a good brand with trusted results and good content is the best way to make sure your site is found. Thinking through keywords, knowing what successful competitors are doing, and establishing yourself in a brand niche are all bonus steps that will give you the edge in a crowded search world.