marshal.dumps quadratic growth and marshal.dump not allowing file-like objects

Peter Otten __peter__ at
Sun Jun 15 13:41:23 CEST 2008

John Machin wrote:

>> Here's how marshal resizes the string:
>>         newsize = size + size + 1024;
>>         if (newsize > 32*1024*1024) {
>>                 newsize = size + 1024*1024;
>>         }
>> Maybe you can split your large objects and marshal multiple objects to
>> keep the size below the 32MB limit.
> But that change went into the svn trunk on 11-May-2008; perhaps the OP
> is using a production release which would have the previous version,
> which is merely "newsize = size + 1024;".

That is indeed much worse. Depending on what the OP means by "large objects"
the problem may be fixed in subversion then.
> Do people really generate 32MB pyc files, or is stopping doubling at
> 32MB just a safety valve in case someone/something runs amok?

A 32MB pyc would correspond to a module of roughly the same size. So
someone/something runs amok in either case.


More information about the Python-list mailing list