I started writing a crawler, but never finished. However, the engine was used in the Cacheability Engine -- http://www.mnot.net/cacheability/ It might be useful to rip the core out as a start for one if you don't have anything else. Cheers,