Google uses human raters to help train its algorithm

It’s no secret that Google uses a team of “Human Raters” to establish a baseline for Page Quality (PQ) on the web to help train its algorithms. The video below from Google’s Head of Web Spam, Matt Cutts, explains what human raters are there to do (and how):


There are a few takeaways from this: first, the human raters are used to test changes to the algorithms (hence their algorithm training role and I use plural as semantic search is made up of some multiple algorithms feeding off 200+ signals), second even they get it wrong at times, but that is of instructional value to the Google search team, third, by seeing what the human raters are asked to rate (and how) we can see the direction Google search is heading towards even if we do not know the details (a little like metadata).

Google has a manual it gives to its human raters and inevitably it gets leaked from time to time. Last time this happened was in 2012 and the manual was 120+ pages long. This time round Ana Hoffman broke the news leading to Jennifer Slegg’s blogpost about it. The manual has grown to 160 pages now and it’s been completely re-written so some things have changed.

In order to understand how Google uses the data it receives from its human raters check out this short video:


Factors That Are Important to Search

The 2012 guide had several factors that were important when it came to ranking a web page:

• User intent and page utility
• Location
• Language
• Query interpretation
• Specificity of the query and landing page
• Timeliness of the informational need of the query to evaluate whether recent content is necessary
• The language of video content vs. query and landing page content language

Two years on the critical factors raters look at have shifted to what Google calls its E-A-T filter:

• Expertise
• Authority
• Trustworthiness

In a direct nod to value the Google PQ Guidelines state that:

“As long as the page is created to help users, we will not consider one particular purpose or type of page to be higher quality than another. For example, encyclopedia pages are not necessarily higher quality than humor pages.”

Utility, Page Quality and Locale

A number of things stand out from the document. First the easy one: Page Quality (PQ) is search query independent. This means the raters (and by association Google’s algorithm) look at landing page quality to see how easy it is to navigate, extract the information the searcher needs, quickly and also find additional links that help add breadth and depth to the result. Second, locale is always present. Every query has a locale even if it is location-agnostic. When locale (task language and IP match location we get hyper-contextualized results).

Google Quality Rater Guidelines snippet

Now the harder bit: Utility scoring applies to the query-dependent results delivered on a search. This determines how well a page (or a website) that appear on search match the query intent. A website that scores well then is one that provides a great end-user experience plus value in terms of its content.

The hardest bit still: A great end-user experience and value require constant attention to detail. Checkbox website creation where a page needed to be up and some content needed to be on it are now really useless practices wasting time and effort.

Just like in the real world everything exists for a reason so in the digital one everything that now is placed on a website has to have a really carefully thought-through reason that defines the What, Why How rule that’s core to semantic search.

At times Google’s Rater Guidelines read like crib notes for my book, Google Semantic Search I’d like to lay claim to my insightfulness at second-guessing what was coming, but really all I did was read up on the research and the join the dots. Semantic search is all about identity and intent. Uniqueness and value. Although there are technicalities that need to be taken into account, ultimately they are there to help achieve these things, rather than subvert them. That is the truly disruptive paradigm shift that semantic search has made possible.

External Links

Google’s Patent for scoring constructed responses and methods for training, monitoring, and evaluating human rater's scoring of constructed responses
Dejan SEO’s Real Websites of the Examples used in Google’s Rater Guidelines
Google Search Quality Raters
Field Experiment on Reputation at Google Answers
An Interview With a Google Search Quality Rater