On some of my WordPress sites for security reason I use iThemes Security plugin. I wrote about this plugin previously and I can say over this time I’m very happy using it and work on my sites with no worry at all.
Anyway I was saying that I’m very satisfied with options which this plugin offering, and how good it works on my sites.
Some powerful options are:
Write to file – Whether or not iThemes Security should be allowed to write to wp-config.php and .htaccess automatically. If disabled you will need to manually place configuration options in those files.
Lockout White List – Use the guidelines below to enter hosts that will not be locked out from your site. This will keep you from locking yourself out of any features if you should trigger a lockout. Please note this does not override away mode and will only prevent a temporary ban. Should a permanent ban be triggered you will still be added to the “Ban Users” list unless the IP address is also white listed in that section.
Override Proxy Detection – If you’re not using a proxy service such as Varnish, Cloudflare or others turning this on may result in more accurate IP detection.
Brute Force Deection – If one had unlimited time and wanted to try an unlimited number of password combinations to get into your site they eventually would, right? This method of attack, known as a brute force attack, is something that WordPress is acutely susceptible by default as the system doesn’t care how many attempts a user makes to login. It will always let you try again. Enabling login limits will ban the host user from attempting to login again after the specified bad login threshold has been reached.
and many more
However there is and option where you can setup blocking suspicious IPs and there you should take a look for blocked IPs and see if any of them comes from search engines bots.
In my case I noticed that Google Webmaster Tools gives me an error messages for crawling my site. This is an original message from Google
We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.
Redirect error: Redirect error
HTTP Error: 302
After searching this error I found out that some iThemes Security plugin users wrote about this already and that problem is related to this plugin. Most likely plugin don’t recognize all bot IPs and it keeps blocking some of them.
To resolve this problem and to give your sites full SEO potential you need to edit plugin settings and to White list these IPs.
For example, if I go to my iThemes Security – Logs there I can see that Microsoft Bingbot IP address is blocked.
Why it is blocked I don’t know, but it needs to be unblocked. To do that I will write this IP to White list in my iThemes security dashboard.
In short, to let bots crawl your sites, see if they are blocked and unblock them by writing their IPs to White list table.