urlretrieve hangs on large files (sometimes)

Josh joshl at commenspace.org
Mon Sep 20 16:11:58 EDT 2004


Hi.  I am writing a script that downloads lots of zips from a usgs site. 
  All is going well except that occasionally, in the midst of 
downloading a file, the script just hangs and i must either ctrl-break 
it (running from command line) or, if I wait about 20 minutes after it 
hangs, even that won't allow me to close it.  Either way, I must then 
restart the program and have it pick up where it left off.

I don't know much about python or http, but I believe that the problem 
is website sometimes does not respond to the 'get' (or whatever 
urlretrieve ) request is in a timely manner:  certainly the site 
sometimes takes a bit of a break even when downloading with IE, for 
example.  However IE or other download managers continue -- my python 
script just hangs.

Here is the relevant code, I think:

	out = file(outFile, "wb")
	(filename, headers,) = urllib.urlretrieve(dlinks[count], outFile, 
reporthook=report)
	out.close()

I am making a haphazard guess that somewhere in the bowels of 
urlretrieve there is a timeout setting for the http get command, and if 
the site does not respond in time it gets all upset and confused. 
Whether I am wrong or right -- any suggestions?

Thanks a lot.

   -Josh



More information about the Python-list mailing list