Is there no compression support for large sized strings in Python?

[Fredrik Lundh]

Claudio Grondi wrote:

> What started as a simple test if it is better to load uncompressed data
> directly from the harddisk or
> load compressed data and uncompress it (Windows XP SP 2, Pentium4  3.0 GHz
> system with 3 GByte RAM)
> seems to show that none of the in Python available compression libraries
> really works for large sized
> (i.e. 500 MByte) strings.
>
> Test the provided code and see yourself.
>
> At least on my system:
>  zlib fails to decompress raising a memory error
>  pylzma fails to decompress running endlessly consuming 99% of CPU time
>  bz2 fails to compress running endlessly consuming 99% of CPU time
>
> The same works with a 10 MByte string without any problem.
>
> So what? Is there no compression support for large sized strings in Python?

you're probably measuring windows' memory managment rather than the com-
pression libraries themselves (Python delegates all memory allocations >256 bytes
to the system).

I suggest using incremental (streaming) processing instead; from what I can tell,
all three libraries support that.

</F>