Open source web crawler with mysql integration

dhenews dheianevans at
Fri Apr 10 01:26:45 CEST 2009

I'm looking for a crawler that can spider my site and toss the results
into mysql so, in turn, that database can be indexed by Sphinx Search.

Since I don't want to reinvent the wheel, is anyone aware of any open
source projects or code snippets that can already handle this?

Thanks for any advice.

More information about the Python-list mailing list