So I am feeling better about this whole server thing now. I think I still have some room for improvement and will still test out my non-wordpress ideas. At the same time though I have some basic wp sites that load in 1.5 seconds and that is acceptable to me.
I did 2 things to get the load on the server back to normal. The first thing I did was identify the problem areas using my log files. I watched for the accounts getting hammered with cpu usage and then checked the raw logs on those accounts. Over and over again I found that it was bad bots. So I came up with a good .htaccess to block the bots. I don’t remember exactly where I found this but it was not mine to take credit for…
bot code to put into .htaccess
Then I saw both good and bad bots just killing my search script. That script was dynamic and always will return a page no matter what you search for. Google tells you not to include pages like this in their index. I hated to tell them not to crawl those pages but I did with these lines in my robots.txt
User-agent: twiceler Disallow: / User-agent: * Disallow: /search
The first one is to block cuil/twiceler… it was something I talked about many months ago. I could probably move that over to the htaccess but cuil listens and doesn’t hit my pages. The other lines tells all search engines (*) to not crawl the pages in the search directory.
To my surprise implementing these changes across all my sites has made huge improvements. I noticed it took a little while for some of the bots to look at the robots.txt file again. Looks like they don’t look for it every visit to your site. The stuff in the htaccess file though will block the bots immediately.
Now I want to shift my focus a little from the technical stuff like this server and get back into what it is going to take to get my earnings back to an acceptable level. Even though these server problems sucked I am actually kind of glad it happened. I like being on my own server and in control of my own destiny.
Subscribe in a Reader