<html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12pt"><div>For scraping with authentication I find the twill module is very good.</div><div><br><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px;"><div style="font-size: 12pt; font-family: 'times new roman', 'new york', times, serif; "><div style="font-size: 12pt; font-family: 'times new roman', 'new york', times, serif; "><font size="2" face="Arial"><hr size="1"><b><span style="font-weight:bold;">From:</span></b> Glen Jarvis <glen@glenjarvis.com><br><b><span style="font-weight: bold;">To:</span></b> Stephen McInerney <spmcinerney@hotmail.com><br><b><span style="font-weight: bold;">Cc:</span></b> "<baypiggies@python.org>" <baypiggies@python.org><br><b><span style="font-weight: bold;">Sent:</span></b> Saturday, June 25, 2011 6:48 PM<br><b><span style="font-weight:
bold;">Subject:</span></b> Re: [Baypiggies] Scraping with authentication: Scrapy vs BeautifulSoup?<br></font><br>
<div id="yiv1698901936"><div>Stephen,</div><div> Beautiful soup really just parses the HTML. It doesn't (have to) retrieve the page for you.</div><div><br></div><div> You can use the built-in httplib2, urllib libraries to retrieve the page (also with authentication) and then use BeautifulSoup to parse the page.</div><div><br>Cheers,</div><div><br></div><div><br></div><div>Glen</div><div><br>On Jun 25, 2011, at 1:42 PM, Stephen McInerney <<a rel="nofollow" ymailto="mailto:spmcinerney@hotmail.com" target="_blank" href="mailto:spmcinerney@hotmail.com">spmcinerney@hotmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">
<br>What do people use for scraping on a website requiring (login form-based) authentication?<br><ul><li>BeautifulSoup: does not handle authentication or cookies</li><li>Scrapy: does but more heavyweight paradigm to learn, incl. XPath</li></ul><br>Some discussion: <a rel="nofollow" target="_blank" href="http://stackoverflow.com/questions/4328271/best-way-for-a-beginner-to-learn-screen-scraping-with-python">http://stackoverflow.com/questions/4328271/best-way-for-a-beginner-to-learn-screen-scraping-with-python</a><br><br>Thanks,<br>Stephen<br><br> </div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>Baypiggies mailing list</span><br><span><a rel="nofollow" ymailto="mailto:Baypiggies@python.org" target="_blank" href="mailto:Baypiggies@python.org">Baypiggies@python.org</a></span><br><span>To change your subscription options or unsubscribe:</span><br><span>http://mail.python.org/mailman/listinfo/baypiggies</span></div></blockquote></div><br>_______________________________________________<br>Baypiggies mailing list<br><a ymailto="mailto:Baypiggies@python.org" href="mailto:Baypiggies@python.org">Baypiggies@python.org</a><br>To change your subscription options or unsubscribe:<br><a href="http://mail.python.org/mailman/listinfo/baypiggies" target="_blank">http://mail.python.org/mailman/listinfo/baypiggies</a><br><br></div></div></blockquote></div></div></body></html>