Dumb SEO Questions

(Entry was posted by Neil Cheesma on this post in the Dumb SEO Questions community on Facebook, 05/21/2020).

BingBot and Google Bot are making too many requests to my website

A few months ago I was getting some 500 errors that it turned out to be the hosting company`s own seo caching plugin - resolved with an update.
A few days ago the 500 errors have started recurring intermittently. I reported this today to the hosting tech support... and the reply I have is this...

"The issue is caused by BingBot and Google Bot, they are making too many requests to your website which causes load and crashes the server.

Here are some of the BingBot requests which are around 12 000 for today"
The question I ask - is it a reasonable assumption that this is the cause of the problem?
This question begins at 00:21:53 into the clip. Did this video clip play correctly? Watch this question on YouTube commencing at 00:21:53
Video would not load
I see YouTube error message
I see static
Video clip did not start at this question

YOUR ANSWERS

Selected answers from the Dumb SEO Questions Facebook & G+ community.

  • Roger Montti: If it`s a shared server then that could be enough to take down your site.
  • Neil Cheesman: It isn`t `shared` - 3 sites of my own.
  • Roger Montti: If it`s a dedicated server then no. That`s not right.

    My dedicated server gets hammered by legit bots and non-legit bots and it keeps on serving.
  • Brenda Michelin: Neil Cheesman What type of hosting---typically a VPS would not be affected. Who is your host?
  • Neil Cheesman: Brenda Michelin Siteground
  • Brenda Michelin: Neil Cheesman OHHHHH! Siteground is NOTORIOUS for blaming the search bots for this. Part of their upsell. Siteground is highly recommended mainly because of their extremely generous affiliate pay.
  • George G.: check the logs. also check if the sg plugin is not on again.

    then register in bing webmaster tools and you can limit the bing crawl from there https://www.bing.com/webmaster/help/crawl-control-55a30302
  • Neil Cheesman: The Google Bot is currently at 5 seconds between requests...
  • Andrew Halliday: Neil Cheesman they ignore that instruction by the way and let their boys decide
  • Andrew Halliday: As someone who specialising in server log auditing.

    Bing bot is usually the most active bot. You can login to bing webmaster tools and instruct to crawl less often but it’s just a request.

    How big is the site. Also they follow every link on the site.

    If googlebot causes 5xx errors your in trouble. If they regularly hit 5xx errors then they will crawl your site less often as they don’t want to be the reason for brining down your site

View original question in the Dumb SEO Questions community on Facebook, 05/21/2020).