Those of you that have been following me know that I have had my share of hosting issues along the way. I thought it was my cron jobs or scripts causing the issues. HostGator told me it was my scripts. Well they are correct to an extent.
I was having issues again about a week ago. Usually this site and my others start to run slow and sluggish or don’t respond at all. When that happens I can expect a call from HostGator saying they disabled something to preserve the health of the server and other users on it. I do not have a dedicated server. I use HostGators reseller package that gives me WHM access. Sure enough I got the call. I emailed them and this time I got a little more information. They said my search script was being abused by Cuil.
Turns out that this is not uncommon. After a little searching I found that they randomly throw stuff into the search script to try and find new pages to index. Well the way my script works, it will always give them a results page no matter what they search for. Sometimes it will say no results found but it is still a page. Cuil is known for this behavior and has brought many servers down by doing this.
I could care less about Cuil. When was the last time you saw a referral to your site from Cuil? I don’t think I ever did. So HostGator said they would block Cuil using Robots.txt for me. They offered to put it in all my sites… 100+! Sure enough they did. This was above and beyond in my book. It saved me a ton of time.
This is what needs to be placed in your robots.txt file to block Cuil:
User-agent: twiceler
Disallow: /
Turns out that the Cuil robot can bring your site to its knees… but for the most part they honor the robots.txt… go figure…
Also you can visit this site to request to be excluded from indexing:
http://www.cuil.com/info/webmaster_info/
I wanted to share this because my sites have been running much faster. It makes it easier for me to setup new sites and manage my existing ones. Every site I put out now has this code in the robots.txt file. I suggest you do the same because once Cuil finds you… look out…
Subscribe in a Reader
Thank you so much for heads up.
I was always wondering why one of my sites runs much slower than other sites tough they are on the same host and use similar plugins.
Now I got the answer. Check my StatPress panel and Cuil stats is right there! It almosts visit my site every other minute!
I will put the robot.txt as my default setting.
Another thing which is unrelated to the topic. Do you find com or net ranks better than info domains? I know you have dropped lots of info domains. Are they not performing well?
Glad this helped you out. It is a shame that Cuil does this. I don’t know how they think this behavior is ok.
In my experience info domains get dropped out of Googles index very quickly if they are thin sites. Almost all my info sites got dropped within 6 months and I didn’t bother renewing them.
Don’t fall into the temptation of using info domains because they are cheap. Google knows they are cheap and watches them closer for spammy sites.
Many people here told me to move away from info domains and it is one of the best things I have done to increase my earnings. I have not have any of my net or com sites dropped yet that are running wordpress. At least that I know about…. knocking on wood!