Urllib vs. FireFox

Rex rex.eastbourne at gmail.com
Fri Oct 24 21:53:05 CEST 2008


Right. If you want to get the same results with your Python script
that you did with Firefox, you can modify the browser headers in your
code.

Here's an example with urllib2:
http://vsbabu.org/mt/archives/2003/05/27/urllib2_setting_http_headers.html

By the way, if you're doing non-trivial web scraping, the mechanize
module might make your work much easier. You can install it with
easy_install.
http://wwwsearch.sourceforge.net/mechanize/




More information about the Python-list mailing list