(Entry was posted by Valentina Huff on this post in the Dumb SEO Questions community on Facebook, 01/08/2016).

Sitemap and robots.txt file

If we blocked a page from the Robots.txt file of our website (for example, a thank you page that we do not want people to find unless they will out a form), will we need to remove that page from our Sitemap or will the Sitemap be automatically updated to leave the page out?

This question begins at 00:29:21 into the clip. Did this video clip play correctly? Watch this question on YouTube commencing at 00:29:21

Share on Twitter

Share on Facebook

YOUR ANSWERS

Selected answers from the Dumb SEO Questions Facebook & G+ community.

Valentina Huff: If we blocked a page from the Robots.txt file of our website (for example, a thank you page that we do not want people to find unless they will out a form), will we need to remove that page from our Sitemap or will the Sitemap be automatically updated to leave the page out? ;
Ryan Cramer: You will want to remove any page blocked by robots.txt from your sitemap.xml file, otherwise you will get an error in your sitemap section and throw off indexing of your other pages
Mark Steenbakkers: Hi +Valentina Huff you would need to remove the page from your Sitemap.

A webpage still can be indexed even if you blocked the page in the robots.txt. You only state not to crawl the webpage. If a for example a robot will find a link to the thank you page, it will follow the link and the webpage can still be indexed.

I would suggest that you also add a meta tag in the header of the page.

<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">

This meta tag tells the robot not to index the webpage, but it will follow the links on the page.
Edwin Jonk: From the expert panel in this week's SEO Questions hangout on air on 00:29:21 into the YouTube video: https://dumbseoquestions.com/q/sitemap_and_robotstxt_file +Valentina Huff

If our assistance with this issue was useful to you, please consider sharing your success story so that others might benefit.
; ;