Google Explains How it Measures Search Quality - uSERP (2024)

Elizabeth Tucker, a 15-year former data scientist at Google and now Director of Product Management, appeared on Google’s Search Off the Record podcast, where John Mueller (Google Search Advocate) and Lizzi Sassman asked her how Google measures search quality.

Tucker’s revelations on whether Google search quality is improving and how it quantifies search quality and adapts to new user behavior could give marketers the edge they need to stay ahead.

Keynotes

Some main points Tucker spoke about:

• “How making one search much better could destroy 50 more.”
• “Google uses multiple methods to measure search quality, including behavioral analysis and human evaluators.”
• “As search quality improves, users tend to make more complex queries.”
• “Keyword searches have gotten longer.”
• “Prepositions are still a problem for Google.”
• “Not everything important in search quality is measurable, and not everything that is measurable is important.”

Google Search Off the Record Podcast

Google’s multifaceted measurement approach

Tucker began by explaining that it’s hard to determine if search quality is improving and that websites are constantly changing.

Tucker explained:

• “It’s really hard, we use a lot of metrics where we sample queries and have human evaluators go through and evaluate the results for things like relevance.”
• “I think one of the challenges that we face is understanding how Google is performing overall. But what we’ve generally found, in Search Quality, is that you can make one search much better and meanwhile destroy 50 more.”

John Mueller asked Tucker how Google chooses what to show versus what it shows less off.

Elizabeth spoke about signals and systems:

• “We have many signals, many systems that all need to work together to produce a great search result page. Some of the systems are by their nature denotative, and webspam would be a great example of this.”
• “We use a lot of metrics where we sample queries and have human evaluators go through and evaluate the results for things like relevance.”

“If we just stood still, Search would get worse. People’s needs are constantly changing. Our work is almost never done.” Tucker said.

Kids search differently than adults

Mueller brought the conversation to the use of keywords, saying:

“Google was very focused on keywords, and that’s kind of how I still search. But, when I see how my kids search, they write like full questions and they expect answers.”

Tucker agreed:

• “kids have grown up in a world where they expect to be able to ask Google virtually anything in any way and get an accurate answer. And I think that speaks to all of the work that we have done to make that possible.”
• “Search behavior has changed, as I’ve had more confidence that Search could actually kind of handled that question.”

Tucker revealed that search quality improves when user queries become more complex and Google’s team navigates a constantly changing landscape.

Tucker finished with:

• “The better we’re able to do this, the more interesting and difficult searches people will do.”

Four words used to be a long keyword search

Tucker delved a little deeper into keyword search and how Google used to consider four words as long, but now the average search is 10 to 20 in length, making Google’s job even more challenging.

Tucker referred to the old days:

• “When I started at Google, if a search had more than four words, we considered it long. I mean, nowadays I routinely see long searches that can be 10-20 words or more. When we have those longer searches, understanding what words are important becomes challenging.”

She then spoke about snippets:

• “When featured snippets first came about, it was motivated by this kind of problem. How can we match the answer, not just keyword match on the words in the question?”
• “We wanted pages that would provide the answer, not just match the words.”

Prepositions and not still a problem

Lizzi asked Tucker if Google had a problem with prepositions and the adverb “not” because she’d noticed searches using them don’t work well.

Tucker explained why Google has a hard time with the word not:

• “It’s really hard to know when not means that you don’t want the word there or when it has a different kind of semantic meaning.

And why Google has a problem with prepositions:

• Prepositions, in general, are another hard one, and one of the really big exciting breakthroughs was the BERT paper and transformer-based machine learning models.”
• “Google used to be very keyword-focused. If you just put together some words with prepositions, we were likely to go wrong!”
• “Prepositions are very difficult or used to be for our systems.”

Finishing with an honest evaluation:

• “I would not say this is a solved problem. We’re still working on it.”

Not everything important is measurable, and not everything that is measurable is important

Towards the end of the podcast, Tucker discussed how it used to be easy for Google to determine whether an update was helpful, but that’s no longer the case, and it’s a hard problem.

Tucker said:

• “Not only do we have to look at different aspects—are results topically relevant, are they accurate, are they trustworthy, are they fresh?”
• “There are many searches that are relatively easy to get right. But there are lots of unusual searches, difficult searches. There are searches where someone’s health or well-being may be at risk.”
• “It’s the most incredible technical challenge and understanding if we’re getting it right, where we’re getting it, right, where it needs focus out of those billions of queries that come in every day. Man, is that a hard problem?”
  

EEAT expertise, authoritativeness, and trustworthiness

John threw some questions SEO professionals are asking regarding E-E-A-T from Google’s Search Quality Rater Guidelines.

Some questions from the SEO world are:

• “Is this a ranking factor? Do I have to kind of apply E-E-A-T to my pages? How would you kind of look at that?”

Elizabeth’s reply:

• “It’s not the only thing to think about when they’re doing page quality rating, but it is a very important aspect. So there is no E-A-T or EE-A-T rankings.”

“But this really is for people to remember. It’s a shorthand, something that should always be a consideration, although different types of results arguably need different levels of E-E-A-T.”

Looking forward

Google will continue to refine and update its search quality measurements to keep pace with how people search.

Tucker finished the podcast with a valid point:

• “Technology is constantly changing, websites are constantly changing. If we just stood still, search would get worse.”

SEO professionals gave Tucker the thumbs-up

After the podcast, Google’s John Mueller said on LinkedIn, “I learn something every time I chat with Elizabeth.“

Glenn Gabe also gave his opinion on X:

If you want an inside look at how Google Search works, we agree with Glenn and recommend listening to Google Search Off the Record Podcast.