The AdSense Pub(lisher) Crawl

If so, you might be inadvertently blocking the AdSense crawler from accessing parts of your site. If you aren't sure what a robots.txt file is, it's a text file that you include on your domain that allows you to block crawlers from accessing your site. You can find out if you're using a robots.txt file by going to example.com/robots.txt (replace 'example.com' with your own domain name) or by using Site Diagnostics. If you do use a robots.txt file to block certain crawlers from accessing your site, it's a good idea to add an explicit invitation to the AdSense crawler so it knows it's welcome to visit any page with AdSense code. Please keep in mind that the AdSense crawler is separate from the Google bot for our search index.

To give the AdSense crawler access, add these two lines to your robots.txt file:

User-agent: Mediapartners-Google*
Disallow:

You can use the Site Diagnostics link in your AdSense Reports tab to see whether we're having trouble crawling any pages on your sites. If you're concerned about the privacy of some pages on your site, keep in mind that we don't publish any of the information retrieved by the Mediapartners-Google crawler, also known as the AdSense crawler, in any index, and it will only crawl pages of your site which contain the AdSense code.

Our crawler will also get tripped up by any page that's only accessible to a logged-in user. If certain pages of your site are only available to users that have logged in, and you place ads on these pages, it's important to give the Mediapartners-Google crawler explicit access to view them too. In this case, the answer is site authentication, which you can find under your AdSense Setup tab. (Please note that you'll need to be migrated to Google Accounts to use this feature.) You can give our crawler access while continuing to prevent other users or bots from accessing the content on your site.

3:24 PM - 12/11/2008 - post comment