(Entry was posted by Neil Cheesma on this post in the Dumb SEO Questions community on Facebook, 03/12/2020).

How to prevent Google from crawling my RSS?

I see in GSC the Crawled Not Indexed includes /feed/ at the end of several hundred urls - this doesn`t seem like good practice - how can the /feed/ not be crawled? (Wordpress)

This question begins at 00:23:52 into the clip. Did this video clip play correctly? Watch this question on YouTube commencing at 00:23:52

Share on Twitter

Share on Facebook

YOUR ANSWERS

Selected answers from the Dumb SEO Questions Facebook & G+ community.

Alex Harford: If you don`t want feeds at all, there`s a plugin that can disable them, or you could Google for code to add to functions.php to do the same thing.
Michael Martinez: You don`t want those feeds to show up in search results. That would be a terrible user experience. It`s good to have them crawled for discovery.
Neil Cheesman: Why would you want them `crawled for discover`? What am I missing...
Michael Martinez: Neil Cheesman Because search engines drop URLs from their indexes all the time. Bringing crawl back to a site as much as possible is a good thing.

Also, if there is user-generated content on those pages then you want it to be crawled and indexed, too.
Neil Cheesman: the urls get crawled and indexed - I just don`t see why the url/feed would benefit by being crawled?
Michael Martinez: Neil Cheesman If the search engine wants to crawl XML and RSS files, why object? It`s not like you`re going to run out of crawl budget. They determine how much of a site to crawl and how often to crawl it.

If they are crawling those feeds that means they are looking for something else to find on your site. Let them crawl.

As long as you show them what is important to you, anything else they collect is just icing on the cake.
Neil Cheesman: Michael Martinez Sounds reasonable/ :)