Quality Score Issues? Make Sure Adsbot Can See Your Landing Page
Some days I look at my inexhaustible list of things to do and I’m amazed by how often the word “Google” shows up… check Google Mail, peek in on my Google Analytics, review my tabs in iGoogle, evaluate my newest client in Google Trends, check on budget use in Google Adwords, and you get the idea. Amidst all this avalanche of information, however, are some interesting tools, when one actually digs around and checks out what all they are capable of.
Google Webmaster Tools – Not Just For Webmasters
Google Webmaster Tools seems to be overlooked by the PPC community. Organic optimizers love it, but the PPC analysts kinda go “meh.” My question is why? Don’t you guys realize that there’s a practical use for everything Google produces? We just have to figure out what it is. Let’s take a look at a simple, basic “tool” that all websites should implement – the robots.txt file. Google didn’t invent the file, but they do give you excellent resources for designing and testing it. SEO techs work so much with robots.txt files that it’s almost passe. Every site has one, and if it doesn’t, the SEO tech will put it there. But as a PPC analyst, do you really understand what power this file holds over you?
The robots.txt file is the file that tells all web bots where to go and where not to go, including Adsbot-Google. Do you realize what disaster awaits you if your PPC account’s landing pages for some reason are disallowed by the robots.txt file?
Landing Pages and Quality Score
Case in point: the incurable poor landing page Quality Score. In the last couple of months, Google Adwords was nice enough to give us a way to see part of the landing page component score in the assessment of our quality scores. Ok, that in itself was a gift because you just try to explain to a site owner that it’s their landing page with 48 thumbnails and slow load time that’s causing their poor QS – they don’t buy it.* So that was fantastic – now there’s something the PPC analyst can point at and go “see, I toldja so.”
But what do you do when their web guys have changed the page – speeded it up, improved the relevance, and you still have a score of 2/10? Maybe some webmaster sleuthing will uncover a blockage. In one example, a company was undergoing SEO work the same time as they were having their Adwords account assessed and optimized. These two projects were being handled by totally different consultants who were not, at first, aware of the other’s role in working with this web site.
The new landing pages being implemented were developed by in-house web guys, and put into a new directory, named with a keyword-appropriate phrase. Ads were retargeted and the account was monitored. After four weeks, there still was no landing page QS change. Oddly though, the PPC analyst noticed a change in something else. Routine SpyFu checks indicated that this web site’s organic search relevance was improving, and more of his site was now indexed. The PPC analyst asked the site owner if they had hired an SEO analyst. “But of course… glad you noticed.” The PPC analyst had an idea and asked for a copy of the robots.txt file.
A week later there was still no robots.txt file in the PPC analyst’s email so it was time to be creative.
Google Webmaster Tools – Not Just For Webmasters
Oh I already said that, didn’t I? Well, it’s true. Just because they’re called Webmasters Tools doesn’t mean other people aren’t allowed to use them. If you’re a professional PPC analyst, you need to fully understand the power of the Google Login. Hopefully you already understand the power of integrating Adwords with Analytics. It’s time to use Google.com/webmasters/tools – if your PPC client hasn’t already got this set up, set it up for them, or ask them to have their web staff or SEO set it up. In our case, the SEO analyst had indeed already established the Google Webmaster account.
Once you are at the dashboard, it’s a simple matter of a few clicks to check on a possible roadbloack that could be preventing your landing page Quality Score from being re-evaluated. On the left is a nav menu, and at the bottom is a link that says “Tools.” When you click that link, the first heading at the top left is usually “Analyze Robots.txt.” This is where you want to go. If the site has a robots.txt file in use, its contents will be displayed here.
User-Agent: * Disallow: /wp-includes Disallow: /wp-admin Disallow: /projectshare Allow: / Sitemap: http://metricvoodoo.com/sitemap.xml.gz
Now, you’re probably saying, “yay, code stuff.” Many PPC analysts don’t have a coding background but don’t despair. You don’t need to know anything other than the destination URLs you need to test. In the second section of the page there’s a big text-entry block where by default, the main domain URL is usually entered. This is where you type or paste in the URLs you want to test against the file.
At the bottom of the page is a bot-crawl selector. It’s under the heading “Choose User Agents.” There will be a default User Agent (bot) for Google, then there’s a drop-down where you can choose a second bot to test this file against. The purpose of the test is to see what the bot thinks about the robots.txt file, with regard to the URLs you put in – can the bot get there?
One of the selections in the list is Adsbot-Google: the bot responsible for landing page quality score assessment!
Sample Robots.txt test results
The Bot Test Results – Not For Webmasters Only
In the case of the incurable poor quality score, we discovered the latest robots.txt file was designed by the SEO guy to be extremely restrictive because of instances of duplicate content within the site itself – he wanted to control which versions of the content got indexed. The file was designed to disallow anything that was not specifically allowed otherwise, which is the opposite of how the average robots.txt file might be designed. Additionally, the client did not take the steps to make sure that the SEO analyst, PPC analyst and in-house web staff were all working cooperatively. When the web guys added the new landing pages for the PPC account, no one told the SEO guy to make a new allowance in the robots.txt file to allow spidering of the new landing pages. Therefore, Adsbot-Google could not ever get to the new URLs to check their Quality Score. This doesn’t keep you from linking to the pages, it just means the little spider responsible for assigning your QS numbers on the landing page can’t get in there to do its job. Thus, Adwords gets no new data with which to update your landing page’s Quality Scores.
Once the robots.txt file was updated, it became obvious that the new landing pages were being indexed within a week by Adsbot-Google, both through analytics reporting and in Adwords, as QS began to improve.
This is almost an extreme example of a case where no matter what the average pay per click analyst did, nothing would have improved the quality score on the landing pages without some extra background in a totally different discipline. Even the PPC agency’s Google team didn’t know enough about how this process works to suggest examining the robots.txt file for a blockade. Sometimes the breadth of knowledge required to effectively assess and manage a pay per click account stretches the limits a little bit.
Don’t let the “webmaster” label make you think that you can’t use these tools for PPC. Becoming familiar with all the tools that Google provides, regardless of their intent, can help you develop a well-rounded skill set that looks like “voodoo” to other people.
*We get something along the lines of “Well I didn’t have any quality score issues before you touched my account, to which we reply, “How do you know? You did’t have that column turned on.”
Sorry, ouija has no special extra insight.
If you enjoyed this post, please consider leaving a comment or subscribe to the feed and get future articles delivered to your feed reader.
I have never thought of this, thanks.
I’m not so sure this is accurate any longer. Adsbot can and does crawl my landing page, even though it is disallowed in my robots.txt.
David Kadavy´s last blog ..WordPress Optimization: How I Reduced Page Load Time by 75%