12 May
2006
12 May
'06
3:02 p.m.
This makes the Web.Crawler actually send a user-agent when requesting robots.txt, and makes it the same user-agent as it uses everywhere else.
Also, I emailed jhs, marcus and stensson at the addresses listed on http://pike.ida.liu.se/development/ about getting CVS access, but never got a reply from anyone. Can someone who can give me access send me an email?
Thanks Adam