[Baypiggies] Scraping with authentication: Scrapy vs BeautifulSoup?

Aaron Peterson aaron at midnightresearch.com
Sun Jun 26 04:14:19 CEST 2011


Hello:

Mechanize is another good module for automating this kind of thing.

HTH,

Aaron
On Jun 25, 2011 1:43 PM, "Stephen McInerney" <spmcinerney at hotmail.com>
wrote:
>
>
> What do people use for scraping on a website requiring (login form-based)
authentication?
> BeautifulSoup: does not handle authentication or cookiesScrapy: does but
more heavyweight paradigm to learn, incl. XPath
> Some discussion:
http://stackoverflow.com/questions/4328271/best-way-for-a-beginner-to-learn-screen-scraping-with-python
>
> Thanks,
> Stephen
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/baypiggies/attachments/20110625/0244ee0a/attachment-0001.html>


More information about the Baypiggies mailing list