(Entry was posted by Diego Santos Sicardo on this post in the Dumb SEO Questions community on Facebook, 11/17/2016).

Google can be wasting our crawl budget

Hi guys, how can I know for sure if Google is crawling a specific page (apart from doing a log file analysis)?

We have thousands of internal pages that require authentification. I know Google cannot read the content and they aren`t indexed. However, I suspect Google can be wasting our crawl budget on them since we do not actively block them for being crawled.

Thanks for your help.

This question begins at 00:14:31 into the clip. Did this video clip play correctly? Watch this question on YouTube commencing at 00:14:31

Share on Twitter

Share on Facebook

YOUR ANSWERS

Selected answers from the Dumb SEO Questions Facebook & G+ community.

Diego Santos Sicardo: Hi guys, how can I know for sure if Google is crawling a specific page (apart from doing a log file analysis)?

We have thousands of internal pages that require authentification. I know Google cannot read the content and they aren't indexed. However, I suspect Google can be wasting our crawl budget on them since we do not actively block them for being crawled.

Thanks for your help.
Federico Sasso: As a rule of thumb assume that if the page is linked (with non "nofollow" links), and not robots.txt blocked, Google will attempt to crawl it.
You already mentioned log file analysis, it's the definitive way to establish if it has already crawled it.
Imran Ahmed: Bro! Try this ! type "site:your url " in the address bar of your browser.You will find the list of pages google has indexed.
Diego Santos Sicardo: +Imran Ahmed
;Thanks but my question was regarding crawling, not indexing.