This makes the Web.Crawler actually send a user-agent when requesting robots.txt, and makes it the same user-agent as it uses everywhere else.

Also, I emailed jhs, marcus and stensson at the addresses listed on http://pike.ida.liu.se/development/ about getting CVS access, but never got a reply from anyone. Can someone who can give me access send me an email?

Thanks Adam