[Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)
victor.stinner at gmail.com
Tue Jan 28 11:22:40 CET 2014
2014-01-28 "Martin v. Löwis" <martin at v.loewis.de>:
> Debugging reveals that it is actually the many integer objects which
> trigger the sharing code. So a much simplified example of Victor's
> benchmarking code can use
> data = *10000000
> The difference between version 2 and version 3 here is that v2 marshals
> a lot of "0" integers, whereas version 3 marshals a single one, and then
> a lot of references to this integer.
Since the output size looks to be the same, it may be interesting to
special-case small integers, or even integers and floats in general.
Handling references to these numbers takes probably more CPU, whereas
the gain on the file size is probably minor.
I wrote a short patch:
"dumps v3 is 60% faster, loads v3 is also 14% *faster*."
"dumps v4 is 66% faster, loads v4 is 16% faster."
"file size (on version 3 and 4) is unchanged with my patch."
"So with the patch, the Python 3.4 default version (4) is *faster*
(dump 20% faster, load 16% faster) and produces *smaller files* (10%
It looks like a win-win patch :-)
The drawback is that files storing many duplicated huge numbers will
not be smaller with marshal version >= 3.
More information about the Python-Dev