Moz crawl crashing my site?

Posted by dave follmer on April 3, 2018

Hello,

Has anyone ever tried to stop rogerbot from crawling their site? My tech team thinks that Rogerbot is crashing our site and causing a 504 error. Does anyone have experience with this or trying to stop the crawl?

Please help.

I did not find the right solution from the internet.

References:-
https://moz.com/community/q/moz-crawl-crashing-my-site
[url=https://blog.advids.co/20-advertising-and-marketing-examples-from-smart-contract-technology/]Smart contract technology marketing[/url]

Thanks

   0 people like this question
15 30 50   per page
   
  
 
 
   
Submit
Hi Dave,

Usually this means there is an issue within your application and the bot crawl is exposing this. If you are unable to resolve the core issue in the application, you can attempt to block the user agent that is crawling the site (at least as an interim solution). You can specify in the robots.txt that you want to block a specify crawler (http://www.robotstxt.org/robotstxt.html) or you can add application code to block the user agent. 



Posted By: James Vidler

Previous Next

LOGIN

Log in with your Agility username and password or
register for the site




Log In

REGISTER












Register

FORGOT PASSWORD