Open source web crawler with mysql integration
philip at semanchuk.com
Fri Apr 10 04:45:34 CEST 2009
On Apr 9, 2009, at 7:37 PM, Daniel Fetchinson wrote:
>> I'm looking for a crawler that can spider my site and toss the
>> into mysql so, in turn, that database can be indexed by Sphinx
>> Since I don't want to reinvent the wheel, is anyone aware of any open
>> source projects or code snippets that can already handle this?
> Have a look at http://nikitathespider.com/python/
As the author of Nikita, I can say that (a) she used Postgres and (b)
the code wasn't open sourced except for a couple of small parts. The
service is now defunct. It wasn't making money. Ideally I'd like to
open source the code one day, but it would take a lot of documentation
work to make it installable by others, and I won't have the time to do
that for the foreseeable future.
At the URL provided there's a nice module for parsing robots.txt files
(better than the one in the standard library IMHO) but that's about it.
FYI, I wrote my spider in Python because I couldn't find a decent one
written in Python. There's Nutch, but that's not Python (Java I think).
More information about the Python-list