Downloading huge files via urllib

Andrew Markebo flognat at
Tue Sep 24 05:44:11 EDT 2002

| response = urllib.urlretrieve(serverURL, 'run.log')
| However, I then get a MemoryError.  Tracking down the source of the
| error, I see the offending function in httplib:
|     def makefile(self, mode, bufsize=None):
|         """Return a readable file-like object with data from socket.
|         The returned object contains *all* of the file data
|         """
| I think the problem is that bufsize argument that is ignored.  Does

Read the docs, and you will see "the bufsize argument is ignored" and
"The returned object contains *all* of the file data", so it tries to
squeeze in 20 megs there.. 

| anyone know if this is correct, and what I can do about it?  I would
| like to automate the process of downloading this file, but is it
| possible?

Maybe read directly with recv() from the [Fake]Socket instead of doing
makefile.. And well this requires you probably to use httplib instead
of urllib, shouldn't be any problems I think.

Or maybe throw an eye at my implementation on the http-standard,
should be able to chew https also, it
is webification.tgz that might interest you.


 The eye of the beholder rests on the beauty!

More information about the Python-list mailing list