In this article, you will learn who search quality evaluators are, what they do and what they don't do, how to become one and how search quality evaluation can help you boost your traditional SEO and Content Marketing strategy.
What is a search quality evaluator?
Search quality evaluators are home-based workers with whom Google contracts to provide them feedback. These evaluators help Google confirm that the proposed changes to its algorithms will return results that are more relevant and of good quality.
Search quality evaluators (aka search engine evaluators or Google raters) come from all sorts of backgrounds and are located in more than 40 different countries across the globe.
There are literally hundreds of tasks involved in search quality evaluation, all of which have one thing in common: to assess the search engine's capacity to improve the results and experience they return to the user.
Some examples of tasks performed by evaluators include:
compare two sets of search results and check which one is better, why it is better and by how much
rate how much an automated voice sounds natural or not
research and classify to which category a certain business belongs
create queries that tell a cell phone to do a specific action
check the utility of completions and related queries
rate the helpfulness of knowledge graph panels and other types of special results
etc.
What is the core of the search quality evaluator job?
Although there are many different types of tasks performed by evaluators, the following two are the most common and important ones—the basis of the evaluator job:
Rate the quality of
web pages
Evaluators determine to what extent a web page meets each of the quality attributes defined on Google's Rater Guidelines.
Rate the relevance
of search results
Evaluators determine to what extent a search result meets the needs of the user, as described on Google's Rater Guidelines.
What are Google Rater Guidelines?
The bible for search quality evaluators and a gold mine of information for SEO and content strategists.
The Search Quality Evaluator Guidelines (download PDF here) is the document provided by Google that explains to raters how to evaluate the quality of web pages and the relevance of search results.
This document is usually updated once or twice a year, and its latest version is 167 pages long.
Google used to keep these guidelines confidential, but after they were leaked numerous times, they decided to make the document publicly available to everyone.
Everytime a new update is released, this event makes the headlines of all major search engine news websites. And it hit the headlines because of the importance that Google itself places on that document.
Look at this example, where Google's Danny Sulivan suggests that web publishers take some time to read the Raters Guidelines.
On this other example, Ben Gomes, Google Head of Search, states in an interview to CNBC: "You can view the rater guidelines as where we want the search algorithm to go. They don’t tell you how the algorithm is ranking results, but they fundamentally show what the algorithm should do."
More recently, in an article on "what webmasters should know about Google's core updates", Google stated the following: "We understand those who do less well after a core update change may still feel they need to do something. We suggest focusing on ensuring you’re offering the best content you can. That’s what our algorithms seek to reward. (...) If you understand how raters learn to assess good content, that might help you improve your own content. In turn, you might perhaps do better in Search."
If all the big shots at Google Search are suggesting online publishers to read their rater guidelines, then why not?
Are search quality evaluators Google employees?
Well, they wish.
Search quality raters are hired by third-party companies that provide search engine evaluation services for Google. Although the evaluation methodology and instructions is defined by Google, the workforce hiring and management process is responsibility of these companies.
The two most popular ones are Appen and Lionbridge.
The position type is usually independent contractor, which means raters perform work for these companies as a nonemployee.
It is curious that even though Google considers search quality tests as critical to them, at the same time they handle many strategic decisions regarding raters to these external companies whose less than optimal processes and controversial actions are often times questioned by the rating community.
In 2018, for example, Appen cut the pay rate for new raters by over 50% in some countries—right after they acquired Leapforce. While Google likes diversity among their pool of raters (i.e. people from different backgrounds), such bold move from a third-party compromises the diversity of the rater pool as the job suddenly became much less attractive to many people.
How does search quality evaluation relate to SEO?
In the end, it is as if they were made for each other.
SEO is dynamic and techniques that worked well in the past may even hurt your position in the rankings now. Some people say SEO is dead because they relate it with cheating practices that these professionals would apply to rank well by somehow deceiving the search engine's algorithms.
Well, as search engines become smarter over time, it is correct to say this type of optimization is definitely about to die. And that's good because it makes room for another type of SEO, more in line with the idea of educating people about what constitutes a content that is relevant and of high quality.
Another reason why this modern SEO is gaining traction is that most platforms are now what they claim “optimized for SEO”, thus the optimization efforts are getting geared to an approach that is not as technical as it used to be in the past. Instead, webmasters and publishers are now more concerned with a qualitative approach that is focused more on the content than on the technicalities.
Strangely enough, very few professionals in the industry really master the criteria of what makes a result relevant for a query in terms of content. If they only knew the nitty-gritty of search quality evaluation.
