το σχόλιο κάτω από το άρθρο είναι όλα τα λεφτά...

The method to their madness  Submitted by webman1000 on Wed, 09/24/2008 - 7:04pm
Reading between the lines its quite obvious twicelers screwy design wasnt by accident. You have an Irish Techie who convinced VC's to dump 33 million in small change into his concept, which was based mainly on their massive index size. But guess what, when the VC auditors came to see the index size for real Mr. Tom had to fill it with something. So he came up with a spur of the moment idea to run the system dictionary agaisnt every sites web directory, and presto, Cuil now has generated millions if not billions of indexes to unique ip addresses, ofcourse they are all 40(1,2,3) errors, but who cares, since he was selling index size to his VC's not content. He passes the auditors test, they open up the bank account, and its muffins and chocolates forever (or at least until the money runs out and the VC's try to sell a goldmine full of fools gold.

the trick with robots.txt file did not work as described here http://jazzychad.com/stuff/?p=27 .
So I used the method with rewrite engine.

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} Twiceler
RewriteRule ^.* - [F,L]


You have to put it in every virtual host configuration, and enable apache rewrite engine.
to enable apache rewrite engine in a debian install give:

sudo a2enmod rewrite