[Mailman-Developers] [ mailman-Patches-647083 ] compress dies w/ MemoryError

[noreply at sourceforge.net]

Patches item #647083, was opened at 2002-12-02 13:38
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=300103&aid=647083&group_id=103

Category: command line scripts
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: John-Mark Gurney (jmg)
Assigned to: Nobody/Anonymous (nobody)
Summary: compress dies w/ MemoryError

Initial Comment:
If you have a large mailing list, it is very easy to
quickly exhast memory in many conditions.  nightly_gzip
script tries to load the entire file into memory before
compressing it, not only is this terribly ineffecient,
it also means you are limited to file sizes that can
fit in memory.  Some machines also further restrict how
much memory each process can access at a time.  If that
limit is 512megs, and you have a 400+meg file, compress
will die.

Solution:  read/write the file in blocks.

----------------------------------------------------------------------

Comment By: Bryan Fullerton (fehwalker)
Date: 2002-12-02 13:53

Message:
Logged In: YES 
user_id=660772

would it make more sense to do this with zlib routines on a
stream than with gzip on blocks?


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=300103&aid=647083&group_id=103