Web.Crawler breaks on robots.txt exclusions?