urlretrieve hangs on large files (sometimes)
joshl at commenspace.org
Mon Sep 20 22:11:58 CEST 2004
Hi. I am writing a script that downloads lots of zips from a usgs site.
All is going well except that occasionally, in the midst of
downloading a file, the script just hangs and i must either ctrl-break
it (running from command line) or, if I wait about 20 minutes after it
hangs, even that won't allow me to close it. Either way, I must then
restart the program and have it pick up where it left off.
I don't know much about python or http, but I believe that the problem
is website sometimes does not respond to the 'get' (or whatever
urlretrieve ) request is in a timely manner: certainly the site
sometimes takes a bit of a break even when downloading with IE, for
example. However IE or other download managers continue -- my python
script just hangs.
Here is the relevant code, I think:
out = file(outFile, "wb")
(filename, headers,) = urllib.urlretrieve(dlinks[count], outFile,
I am making a haphazard guess that somewhere in the bowels of
urlretrieve there is a timeout setting for the http get command, and if
the site does not respond in time it gets all upset and confused.
Whether I am wrong or right -- any suggestions?
Thanks a lot.
More information about the Python-list