Patches item #647083, was opened at 2002-12-02 13:38 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=300103&aid=647083&group_id=103
Category: command line scripts Group: None Status: Open Resolution: None Priority: 5 Submitted By: John-Mark Gurney (jmg) Assigned to: Nobody/Anonymous (nobody) Summary: compress dies w/ MemoryError
Initial Comment: If you have a large mailing list, it is very easy to quickly exhast memory in many conditions. nightly_gzip script tries to load the entire file into memory before compressing it, not only is this terribly ineffecient, it also means you are limited to file sizes that can fit in memory. Some machines also further restrict how much memory each process can access at a time. If that limit is 512megs, and you have a 400+meg file, compress will die.
Solution: read/write the file in blocks.
Comment By: Bryan Fullerton (fehwalker) Date: 2002-12-02 13:53
Message: Logged In: YES user_id=660772
would it make more sense to do this with zlib routines on a stream than with gzip on blocks?
You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=300103&aid=647083&group_id=103