[Baypiggies] Scraping with authentication: Scrapy vs BeautifulSoup?

Stephen McInerney spmcinerney at hotmail.com
Sat Jun 25 22:42:16 CEST 2011



What do people use for scraping on a website requiring (login form-based) authentication?
BeautifulSoup: does not handle authentication or cookiesScrapy: does but more heavyweight paradigm to learn, incl. XPath
Some discussion: http://stackoverflow.com/questions/4328271/best-way-for-a-beginner-to-learn-screen-scraping-with-python

Thanks,
Stephen

 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/baypiggies/attachments/20110625/17db6291/attachment.html>


More information about the Baypiggies mailing list