Downloading huge files via urllib

[Andrew Markebo]

| response = urllib.urlretrieve(serverURL, 'run.log')
| 
| However, I then get a MemoryError.  Tracking down the source of the
| error, I see the offending function in httplib:
| 
|     def makefile(self, mode, bufsize=None):
|         """Return a readable file-like object with data from socket.
[...]
|         The returned object contains *all* of the file data
|         """
| 
| I think the problem is that bufsize argument that is ignored.  Does

Read the docs, and you will see "the bufsize argument is ignored" and
"The returned object contains *all* of the file data", so it tries to
squeeze in 20 megs there.. 

| anyone know if this is correct, and what I can do about it?  I would
| like to automate the process of downloading this file, but is it
| possible?

Maybe read directly with recv() from the [Fake]Socket instead of doing
makefile.. And well this requires you probably to use httplib instead
of urllib, shouldn't be any problems I think.

Or maybe throw an eye at my implementation on the http-standard,
should be able to chew https also, http://flognat.myip.org/hacks/ it
is webification.tgz that might interest you.

        /Andy

-- 
 The eye of the beholder rests on the beauty!