Error while downloading webpages

TimB timbovaird at gmail.com
Wed Jul 6 03:39:57 EDT 2011


Hi everyone, new to python. I'm attempting to download a large amount
of webpages (about 600) to disk and for some reason a few of them
fail.

I'm using this in a loop where pagename and urlStr change each time:
import urllib
    try:
        urllib.urlretrieve(urlStr, 'webpages/'+pagename+'.htm')
    except IOError:
        print 'Cannot open URL %s for reading' % urlStr
        str1 = 'error!'

Out of all the webpages, it does not work for these three:
http://exoplanet.eu/planet.php?p1=WASP-11/HAT-P-10&p2=b
http://exoplanet.eu/planet.php?p1=HAT-P-27/WASP-40&p2=b
http://exoplanet.eu/planet.php?p1=HAT-P-30/WASP-51&p2=b
giving "Cannot open URL http://exoplanet.eu/planet.php?p1=WASP-11/HAT-P-10&p2=b
for reading" etc.

however copying and pasting the URL from the error message
successfully opens in firefox

it successfully downloads the 500 or so other pages such as:
http://exoplanet.eu/planet.php?p1=HD+88133&p2=b

I guess it has something to do with the forward slash in the names
(e.g. HAT-P-30/WASP-51 compared to HD+88133 in the examples above)

Is there a way I can fix this? Thanks.



More information about the Python-list mailing list