[Mailman-Developers] [ mailman-Patches-647083 ] compress dies w/ MemoryError

[noreply at sourceforge.net]

Patches item #647083, was opened at 2002-12-02 10:38
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=300103&aid=647083&group_id=103

Category: command line scripts
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: John-Mark Gurney (jmg)
Assigned to: Nobody/Anonymous (nobody)
Summary: compress dies w/ MemoryError

Initial Comment:
If you have a large mailing list, it is very easy to
quickly exhast memory in many conditions.  nightly_gzip
script tries to load the entire file into memory before
compressing it, not only is this terribly ineffecient,
it also means you are limited to file sizes that can
fit in memory.  Some machines also further restrict how
much memory each process can access at a time.  If that
limit is 512megs, and you have a 400+meg file, compress
will die.

Solution:  read/write the file in blocks.

----------------------------------------------------------------------

>Comment By: John-Mark Gurney (jmg)
Date: 2002-12-02 16:22

Message:
Logged In: YES 
user_id=27410

Well, from the looks of it, it is best to use the gzip
module if we want to be able to use gunzip (cmd line) to
uncompress the file.  zlib only provides for compression
objects, and doesn't say if the output is compatible for
direct input to the gunzip program.

I also went with the easiest modification, least source
modifications.

Now that I'm thinking about it, it might of been implemented
as a call to writelines(xreadlines()) w/o having to use my
own while loop resulting in even less modification of the
code.  The only problem is that this may produce very long
lines for binary code.  Also, this requires more scanning
for the new lines, while the block while does not have to
scan the input.

To bad write doesn't support file objects to copy in a block
manner. so the code could simply be:
outfp.write(infp)

P.S.  I forgot to mention in the body that this work is
supported by Vernier Networks, Inc.

----------------------------------------------------------------------

Comment By: Bryan Fullerton (fehwalker)
Date: 2002-12-02 10:53

Message:
Logged In: YES 
user_id=660772

would it make more sense to do this with zlib routines on a
stream than with gzip on blocks?


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=300103&aid=647083&group_id=103