memory error with zipfile module
bruno at modulix
onurb at xiludom.gro
Fri May 19 14:45:25 CEST 2006
Hari Sekhon wrote:
> I do
> import zipfile
> then either of the two:
> A) file('someimage.iso','w').write(zip.read('someimage.iso'))
> B) content=zip.read('someimage.iso')
> but both result in the same error:
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "D:\u\Python24\lib\zipfile.py", line 357, in read
> bytes = dc.decompress(bytes)
<ot>Is that the *full* traceback ?</ot>
> I thought python was supposed to handle memory for you?
Err... This doesn't mean that it bypasses system's memory management.
MemoryError is raised by Python when an underlying (OS-level) allocation
Normally this would mean that you were out of even virtual memory
(swap), but it could also be a symptom of a libc bug, a bad RAM chip, etc.
What do you think will append if you try to allocate a huge block when
you've already eaten all available memory ? Do you really hope that
Python will give you extra ram for free ?-)
Please try this code:
info = zip.getinfo('someimage.iso')
csize = info.compress_size
fsize = info.file_size
print "someimage compressed size is : %s" % csize
print "someimage real file size is : %s" % fsize
So, knowing how zipfile.read() is actually implemented,
total needed ram is : %s
""" % (csize + fsize)
print "Well... Do I have that much memory available ???"
> The python zipfile module is obviously broken...
s/is obviously broken/could be improved to handle huge files/
Making such statements may not be the best way to make friends...
> Any advise?
Yes : Python is free software ('free' as in 'free speach' *and* as in
'free beer'), mostly written by benevolent contributors. So try and
improve the zipfile module by yourself, and submit your enhancements.
Then we all will be very grateful, and your name will be forever in the
Python Hall of Fame.
Or choose to behave as a whiny-whiny clueless luser making dumb
statements, and your name will be buried for a long long time in a lot
It's up to you !-)
NB : If you go the first route, this may help:
with particular attention to the decompressobj.
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'onurb at xiludom.gro'.split('@')])"
More information about the Python-list