(Entry was posted by Roy ODK on this post in the Dumb SEO Questions community on Facebook, 01/23/2015).

User-Agent in robots.txt

Well here`s a dumb SEO question... Which one of the two is the better User-Agent: Disallow: option for several crawlers >
1.
User-agent: nameofbot1
User-agent: nameofbot2
User-agent: nameofbot3
Disallow: /
2.
User-agent: nameofbot1
Disallow: /
User-agent: nameofbot2
Disallow: /
User-agent: nameofbot3
Disallow: /

This question begins at 01:35:27 into the clip. Did this video clip play correctly? Watch this question on YouTube commencing at 01:35:27

Share on Twitter

Share on Facebook

YOUR ANSWERS

Selected answers from the Dumb SEO Questions Facebook & G+ community.

Roy ODK: Well here's a dumb SEO question... ;Which one of the two is the better User-Agent: ;Disallow: option for several crawlers >
1.
User-agent: nameofbot1
User-agent: nameofbot2
User-agent: nameofbot3
Disallow: /
2.
User-agent: nameofbot1
Disallow: / ;
User-agent: nameofbot2
Disallow: / ;
User-agent: nameofbot3
Disallow: / ;
Federico Sasso: I'd surely go for option #2:
While #1 is a valid syntax according to the de-facto standard, I wouldn't expect all bots to support it.
I speak by direct experience and partly against my own interests: as an author of a web crawler I have to say mine doesn't support syntax #1 yet (will do in the near future, though)