[issue1792] o(n*n) marshal.dumps performance for largish objects with patch

[Aaron Watters]

Aaron Watters added the comment:

Facundo

1) the +1024 was an accelerator to jump up to over 1k at the first resize.
I think it's a good idea or at least doesn't hurt.

2) Here is an example program:

def test():
    from marshal import dumps
    from time import time
    testString = "abc"*100000000
    print "now testing"
    now = time()
    dump = dumps(testString)
    elapsed = time()-now
    print "elapsed", elapsed

if __name__=="__main__":
    test()

Here are two runs: the first with the old marshal and the second with the
patched marshal.  The second is
better than 2* faster than the first.

arw:/home/arw/test> ~/apache2/htdocs/pythonsrc/Python/python_old mtest1.py
now testing
elapsed 4.13367795944
arw:/home/arw/test> ~/apache2/htdocs/pythonsrc/Python/python mtest1.py
now testing
elapsed 1.7495341301
arw:/home/arw/test>

The example that inspired this research was very complicated and involved
millions of calls to dumps
which caused a number of anomalies (system calls went berzerk for some
reason, maybe paging).

   -- Aaron Watters

On Jan 11, 2008 9:25 AM, Facundo Batista <report at bugs.python.org> wrote:

>
> Facundo Batista added the comment:
>
> Why not just double the size? The "doubling + 1024" address some
> specific issue? If so, it should be commented.
>
> Also, do you have an example of a marshal.dumps() that suffers from this
> issue?
>
> Thank you!
>
> ----------
> nosy: +facundobatista
>
> __________________________________
> Tracker <report at bugs.python.org>
> <http://bugs.python.org/issue1792>
> __________________________________
>

Added file: http://bugs.python.org/file9124/unnamed

__________________________________
Tracker <report at bugs.python.org>
<http://bugs.python.org/issue1792>
__________________________________
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: unnamed
Url: http://mail.python.org/pipermail/python-bugs-list/attachments/20080111/6ffa3a93/attachment.txt