Rigorous testing
Our goal is always to provide you the most useful and relevant information.
Any changes we make to Search are always to improve the usefulness of results you see. That's why we never accept payment from anyone to be included in search results.
Testing
Search has changed over the years to meet the evolving needs and expectations of the people who use Google. From innovations like the Knowledge Graph, to updates to our ranking algorithms that ensure we’re continuing to highlight relevant content, our goal is always to improve the usefulness of your results.
We put all possible changes to Search through rigorous testing and evaluation. If that testing shows a change brings less useful results, we won’t launch it.
Search experiments: from ideas to launch
Our engineers have many ideas for ways to make your results more useful. But we don’t go on a hunch or an expert opinion. We rely on extensive testing and have a rigorous evaluation process to analyze metrics and decide whether to implement a proposed change. Data from these evaluations and experiments go through a thorough review by experienced engineers and search analysts, as well as other legal and privacy experts, who then determine if the change is approved to launch. In 2019 we ran over 464,065 experiments, with trained external Search Raters and live tests resulting in more than 3620 improvements to Search.
We gather data in multiple ways. Last year, we ran:
-
Search quality tests
383,605 Search quality tests
We work with external Search Quality Raters to measure the quality of search results on an ongoing basis. Raters assess how well a website gives people who click on it what they are looking for, and evaluate the quality of results based on the expertise, authoritativeness and trustworthiness of the content. These ratings do not directly impact ranking, but they do help us benchmark the quality of our results and make sure these meet a high bar all around the world.
To ensure a consistent approach, we publish Search Quality Rater Guidelines to give these Raters guidance and examples for appropriate ratings. Take a look at these guidelines here. You will see that while evaluating the quality of results might sound simple, there are many tricky cases to think through, so this feedback is critical to ensuring we maintain high quality results for users.
-
Side-by-side experiments
62,937 Side-by-side experiments
Search isn’t static. We’re constantly improving our algorithms to return better results and Search Quality Raters play an important role in the launch process. In a side-by-side experiment, we show Raters two different sets of search results: one with the proposed change already implemented and one without. We ask them which results they prefer and why.
-
Live traffic experiments
17,523 Live traffic experiments
In addition to the Search quality tests, we conduct live traffic experiments to see how real people interact with a feature, before launching it to everyone. We enable the feature in question to just a small percentage of people, usually starting at 0.1%. After we collect enough data, we compare the experiment group to a control group that did not have the feature enabled. We look at a very long list of metrics, such as what people click on, how many queries were done, were queries abandoned, how long did it take for people to click on a result, and so on. We use these results to measure whether engagement with the new feature is positive, to ensure that the changes we make are increasing the relevance and usefulness of our results for everyone.
-
Launches
3,620 Launches
Finally, every single proposed change to Search goes through a review by our most experienced engineers and data scientists, who carefully review the data from all the different experiments to decide if the change is approved to launch. Of the proposed changes this past year, many never went live, because unless we can show a change actually makes things better for people, we don’t launch it.
We sell advertising, not search results
We have always believed it is important for everyone to have access to the best information, for free. And it’s ads that enable Google to provide a Search engine that is free — one that works just as well for anyone, no matter your level of education or income.
When you use Google Search, ads may appear with your search results. We think it’s important to be transparent about the difference between paid and organic results, which is why ads are clearly labeled, so they are easy to distinguish from the rest of the page.
All our advertising and sponsored content is clearly labeled (e.g. "Ad"; "Sponsored") so that they are easily distinguishable from Search results.
We work hard to display ads only when they are relevant to the task you are trying to achieve on Google. We only charge advertisers when users interact with the ad, so our interest is in showing only useful ads. In fact, we frequently run no ads at all.
Google’s commercial relationships have no impact on algorithmic Search changes, and partner advertisers do not receive special treatment in resolving organic search issues or requests. We make sure these issues are handled based on the importance and impact to users, and not due to a financial relationship with Google.