12 May
2006
12 May
'06
4:36 p.m.
Web.Crawler.Crawler calls queue->set_stage(real_uri, 6); if its denied by a robots.txt exclusion. Does anyone know what 6 means? It looks like its just ignored later on in MemoryQueue()->get(). The result is that any time a Crawler hits a uri denied by robots.txt, it loops forever checking that uri, calling the error_callback, and then leaving it the queue to check again.
Adam