How do I correctly download Wikipedia pages?
Steven D'Aprano
steven at REMOVE.THIS.cybersource.com.au
Wed Nov 25 22:45:19 EST 2009
I'm trying to scrape a Wikipedia page from Python. Following instructions
here:
http://en.wikipedia.org/wiki/Wikipedia:Database_download
http://en.wikipedia.org/wiki/Special:Export
I use the URL "http://en.wikipedia.org/wiki/Special:Export/Train" instead
of just "http://en.wikipedia.org/wiki/Train". But instead of getting the
page I expect, and can see in my browser, I get an error page:
>>> import urllib
>>> url = "http://en.wikipedia.org/wiki/Special:Export/Train"
>>> print urllib.urlopen(url).read()
...
Our servers are currently experiencing a technical problem. This is
probably temporary and should be fixed soon
...
(Output is obviously truncated for your sanity and mine.)
Is there a trick to downloading from Wikipedia with urllib?
--
Steven
More information about the Python-list
mailing list