On July 28th, Google announced in a post on G+ that they are changing how Quality Score is calculated for new keywords.
The most significant part of the change is that keywords without traffic will now, by default, receive a score of 6, and Average for the three components of Quality Score. Once keywords gather enough impressions, scores will update about a day or so later.
By announcing this only on Google+, they are clearly playing down the importance of the change. I was once on the team that helped launch new QS changes so I understand why they’re trying to minimize the number of advertisers who get unnecessarily worked up over a change that may be small. But now that we’ve gotten some data for our own accounts with Optmyzr’s Historical Quality Score Tracker (which by coincidence we revamped at the same time as Google made its change), let’s take a closer look to see if this is in fact just a small change.
How Quality Score Calculations Have Changed
Before the change made by Google, new keywords would immediately start off with a quality score based on Google’s expectations about future performance. How Google came up with this initial assessment was always a bit of a mystery to most advertisers. While the machine learning algorithms that handle this initial calculation of QS may be complex, they’re doing something very simple: predicting the CTR of a keyword and its ad for any of the auctions that trigger it using closely related historical performance data.
The 3 Factors Used To Decide Starting QS
First, no matter how good you believe you are at coming up with keywords nobody else has ever thought of, Google almost always has data about it from other advertisers, and this systemwide keyword performance is one of the main factors used to predict the performance of new keywords.
The second factor is your account level quality score: in simple terms this is Google’s expectation of whether you are an average, worse-than-average or better-than-average advertiser. By looking at whether your other keywords have usually had good or bad CTRs, they can assign an account level quality score.
Hal Varian, Google’s Chief Economist, says that there is no such thing as account level quality score, and while I believe that to be true from the perspective that it’s not a metric they can look up and share with advertisers, I firmly believe account QS is a concept that matters. Any machine learning system worth its weight would certainly consider historical performance of the advertiser as a strong signal to predict future performance, and that is basically what account level quality score is.
Here’s an example of how the first two factors would interact: if you are adding new keywords that have historically performed poorly across AdWords, but as an advertiser you are typically better than average, these two factors could offset one another and your starting QS would have been about average.
The third factor used to help determine the starting quality score was the ad text. That itself was broken into two components, the ad and the visible URL. Just like with keywords, Google often already had some historical performance data at the visible URL level that it could use to decide if your new keyword was likely to perform better or worse than average.
How Quality Score Is Calculated After The Change
With the recent update to QS, Google no longer predicts quality score for brand-new keywords. Instead, all of the keywords now get a starting QS of six. This is a good thing for low-quality advertisers who used to have to pay a higher initial CPC to be able to participate in the auction with brand-new keywords. It’s not such a good thing for high-quality advertisers who used to benefit from a lower starting CPC for new keywords because Google assumed that these would perform equally well as the other keywords in the account.
Let me explain that… because ad rank is determined by a combination of maximum CPC, QS, and ad extension performance, starting off every advertiser at the same quality score has evened the playing field: only CPC and ad extension performance are the variables that the advertiser has some control over. An advertiser who used to get a bit of a discount because they always had super relevant ads will no longer receive this QS discount when adding new keywords.
Quality Scores Are Moving Closer to 6
Our Historical Quality Score Tracker tool was specifically built to address the issue that advertisers would often be left in the dark about how their QS had moved, especially after big changes in account structure, or after Google algorithm updates. Because we are running this tool on some of our own accounts, we were clearly able to see the impact of this recent update by Google.
You can clearly see that the QS in this account changes on July 27, one day before Google’s announcement.
You can also see there is a big increase in the number of active keywords that have a QS of 6 and a big drop in keywords with other QS values.
What often surprises me is how few keywords get impressions on Google Search where the keyword is an exact match to the query on a daily basis. Even in accounts with tens of thousands of keywords, our QS tool shows that there are usually only a couple thousand keywords that get impressions that matter for calculating the QS you see in the account. Remember that QS is based on data from Google Search, where the query matches the keyword exactly. When there is not an exact match, the data is only used to set the real-time QS at auction time, but not to calculate the 1-10 QS number you see in your account.
Existing Keywords Are Shifting To QS 6
One thing that may not have been obvious in Google’s announcement is that many existing keywords would also move from their current QS to 6 because they don’t yet have enough data to get their own more permanent QS number. Don’t be fooled by the statement “Once keywords gather enough impressions, scores will update about a day or so later.” This means that it can take days, even weeks before keywords get enough data. Then once they do, the next QS update cycle will include the new QS.
The Financial Impact Of This QS Change
For accounts whose overall account QS declined because they had new keywords that were at a QS higher than 6, the costs went up. This illustrates my point that good advertisers no longer receive a break on the starting CPC thanks to their good history. For accounts that were already closer to a QS 6 before this change, we saw costs decline a bit.
When Is A Keyword No Longer New
I thought it would be very interesting to evaluate how many impressions Google needs before deciding the quality score of a keyword, i.e. when the value shifts from 6 to its more permanent value. To do this I looked at what percentage of keywords had a QS of 6 after 1, 10, 100, 200, 500 and 1000 impressions (all-time).
What I found was that the number of keywords shifting out of the QS 6 bucket starts to decrease as the number of impressions rises. While there are decreases no smaller than 5% when going from 1 to 10 and 10 to 100 impressions, after that the decreases are closer to 1%. In other words, after 100 impressions, not that many keywords change QS from a 6. That may indicate the 6 may be the actual expected value.
Google’s machine learning systems are very good at drawing conclusions rapidly based on similar data so I don’t believe there to be a single number of impressions after which Google sets your QS. What I think my data illustrates is an approximate upper bound of how much data they need. In other words, they may know much more quickly, but in the worst case, they would need around 100 impressions to figure out QS.
Conclusion
In our observations of the July 2015 Quality Score change, there has been a dramatic shift of existing low-data keywords towards a QS of 6. As a result, costs have changed for many advertisers. Those who benefited from CPC discounts for new keywords because they had good account Quality Scores receive less of a CPC discount.
