Can’t believe they actually hired an SEO…
To improve its search results, Google contracts 12,000 search quality raters around the world.
Quality Raters evaluate search results and provide feedback to Google on how to improve them. They strictly rate top search results using Google’s Quality Rater guidelines that are 170 pages long and frequently updated.
For many who work in Search Engine Optimization (SEO), the Quality Rater guidelines are an important roadmap of what Google wants to rank—and demote—within its search results. The SEO industry often uses the guidelines for hints about actual ranking factors for an otherwise secret algorithm, and Google directs folks to the guidelines for best practices in improving their websites.
So I had to ask: Could I actually get a job as a Google Quality Rater?
Many might consider my chances slim: I work as a full-time SEO professional helping businesses rank higher in Google, and I’m also a friendly but occasional critic of some of Google’s business practices.
Not only did they hire me as an official rater, but I also learned a lot about the job that would help with working more successfully as an SEO.
In retrospect, it makes sense that they wouldn’t care about SEOs working as Quality Raters (though I doubt they were actually aware.) Working as a Quality Rater teaches you a very holistic way of how Google wants to rank web pages without any shortcuts. I suspect, with a few exceptions, Google would love more SEOs to go through this process of rating search results and websites.
In fact, I’m still officially employed. But first…
A Note on Confidentiality and NDAs
While Google doesn’t magically reveal its algorithm to evaluators, everyone must sign strict confidentiality agreements before working. They take these agreements very seriously.
This means there are certain things I’m not allowed to discuss. (To be fair, most—but not all—of these things are pretty boring.)
According to the non-disclosure agreement, I am only allowed to talk about:
- Things that are already common knowledge in our industry
- Knowledge-based on my own skills and experience
Because the Rater Guidelines themselves fall under “common knowledge,” I am certainly free to discuss specific concepts explained in the guidelines, along with “my own skills and experience” as an SEO. (As with anyone I work with, under no circ*mstances will I violate the NDA.)
How the Quality Rater Process Works
First things first: you don’t actually work for Google.
Quality Raters work as contractors for one of several companies Google hires to manage the process. The list often changes, but a few of the companies known to hire quality raters are Appen, Lionbridge, and Teemwork.ai.
Quality Raters are part-time contractors. In the United States, this means pay of around $15/hour and virtually no benefits.
No, you will not get rich.
What you do get—especially as an SEO—is tons of training to help you evaluate search results and web pages the way Google wants to see them.
To qualify for the job, you must meet certain minimum requirements, e.g., high school education, language skills, a computer, etc.
You also have to pass a series of tests. Because the tests are based on the Quality Rater guidelines, I wanted to see if I could pass without studying.
Nope. I failed the first time I tried.
The tests are challenging, and most people take a week or more to finish. Many never pass. Fortunately, they provide lots of study material and practice tests beyond the Rater guidelines themselves. This also is the most valuable part of the education.
In all, I spent around two weeks studying and passing all the tests in fits and starts before they officially extended a job offer.
This was honestly the most valuable part of the experience, as it taught me the most about how Quality Raters work and how Google views websites and search results.
What SEOs Can Learn Working As A Quality Rater
Let’s get this out of the way: Google makes it clear that the raters themselves do not directly impact the ranking of any one website. (Special emphasis on the word “directly.”)
Google has publicly revealed at least three ways they use Quality Rater data:
- To evaluate and test the quality of algorithm changes
- To give their algorithms “positive and negative examples of search results,” e.g., machine learning
- Evaluate and train experimental results, such as Google’s new AI-powered search
As a Quality Rater, you don’t get insight into how Google uses the data, but you do get to see the raw data itself – simply because you help produce it!
While we won’t get into specifics of the rating process here beyond what Google has told us, the Rater guidelines themselves provide a lot of hints as to what Quality Raters focus on.
While I can’t publicly share the tools used, I made a Google Page Quality Scorecard that you can use to help rate your own web pages using Google’s rating guidelines.
1. Page Quality
The Rater guidelines spend nearly 70 pages on Page Quality. A few of the factors that go into Page Quality considerations include:
- Website/Author Reputation – with extensive instructions on how to research both
- Evaluating Your Money or Your Life (YMYL) Topics
- Experience, Expertise, Authoritativeness, and Trust (E-E-A-T)
- Spam
- Overall Content Quality
As SEOs, we often fall into ‘mind traps” when looking at web pages based on our knowledge of Google. For example, we might see keyword-optimized headings and think, “This page could rank highly,” when in fact, the average user might see a mediocre page at best with lots of spammy ads.
In a way, the Quality Rater guidelines help you to see web pages as a regular user.
2. Needs Met
The Rater guidelines define Needs Met as “focus on user needs and think about how helpful and satisfying the result is for the users.”
In other words, independent of Page Quality, how well does the webpage or search result satisfy user intent?
You can see from Google’s image above Needs Met is highly dependent on if users would want “to see additional results.” Pages that require the fewest clicks from users score the highest on the “Needs Met” scale.
Note that Needs met is separate from Page Quality. This means it’s totally possible to have a high-quality webpage that nonetheless scores low on a Needs met scale.
3. Other Rating Factors
The Rater guidelines explain several other areas that quality raters are expected to evaluate. These include:
1. User Intent: “When a person types or speaks a query, he or she is trying to accomplish something.” It’s fair to say that if you want to evaluate search results, it helps to firmly grasp what the user wishes to accomplish.
User intent can vary based on keywords used, location, device, and language. Queries can also have multiple intents, so it’s important to consider every angle when looking at search results.
2. Website/Creator Reputation: It seems Google cares deeply about reputation. The rater guidelines go into depth explaining how raters can carry out reputation research, look for reviews, etc.
3. Your Money or Your Life (YMYL) Topics: Most SEOs are familiar with YMYL. While many assume YMYL applies to any type of financial or health-related search query, the Quality guidelines make it clear it only applies to queries with the potential to do harm.
For example, a query about “shopping for pencils” is highly unlikely to cause harm and, therefore, wouldn’t be considered YMYL. On the other hand, shopping for a mortgage could have huge financial implications, so it would definitely fall under YMYL consideration.
4. E-E-A-T: While occasionally controversial in SEO, Google spends an inordinate amount of time emphasizing the importance of Experience, Expertise, Authoritativeness, and Trust (E-E-A-T) to Quality Raters.
In fact, Google made major changes with the addition of “Experience” to the Quality Rater guidelines over the past year. Many believe this is related to Google’s Helpful Content and Product Review updates.
5. Spam: Seems simple, but Google guidelines emphasize the importance of evaluators identifying spam pages. In the age of AI-generated content, Google’s spam policies take on a new light. Types of spam Google expects raters to identify include:
- Scraped content with little/no added value
- Thin affiliate page
- Spammy automatically-generated content
Should You Work As A Google Quality Rater?
If I’m being honest, working as a Quality Rater has been some of the best free/paid SEO training I’ve ever undertaken.
Should you do it yourself? Or have someone on your team go through the training process?
On one hand, we can bet Google and its affiliated companies aren’t excited about a bunch of SEOs applying to work simply to quit a few weeks later.
(Due to the hourly commitment, the job really isn’t sustainable after long if you have another job – especially given the low pay.)
That said, the training is fantastic from a content quality and ranking POV. While it won’t help you learn technical SEO, you rarely see this kind of expertise demonstrated in agencies/consultants in regard to content, user intent, and web page quality understanding.
Many agencies already train SEOs on Google’s Quality Rater guidelines. For those really dedicated to the craft of SEO, it may be worth it to see if you—or someone on your team—can actually do the job.
For what it’s worth, I’m still officially employed as a Rater.
For now.
