Something faster then sgmllib for sucking out URLs

Järvinen Petri jarvin24 at tutor.cc.tut.fi
Thu Jun 13 06:16:37 EDT 2002


In article <mailman.1023913936.26808.python-list at python.org>, Alex Polite wrote:
> I'm working on a webspider to fit my sick needs. The profiler
> tells me that about 95% of the time is spent in sgmllib. I use sgmllib
> solely for extracting URLs. I'm looking for a faster way of doing
> this. Regular expressions, string searches? What's the way to go? I'm
> not a python purist. Calling some fast C program with the html as
> argument and getting back a list of URLs would be fine by me.
> 

*****

Thanks guys from your help! My list is now somewhat sorted :)

-Petri



More information about the Python-list mailing list