Open source web crawler with mysql integration

dhenews dheianevans at gmail.com
Thu Apr 9 19:26:45 EDT 2009


I'm looking for a crawler that can spider my site and toss the results
into mysql so, in turn, that database can be indexed by Sphinx Search.

Since I don't want to reinvent the wheel, is anyone aware of any open
source projects or code snippets that can already handle this?

Thanks for any advice.



More information about the Python-list mailing list