[Python-ideas] Fast pickle

Antoine Pitrou solipsis at pitrou.net
Sat Nov 17 22:36:13 CET 2012


On Sat, 17 Nov 2012 23:18:29 +0200
Serhiy Storchaka <storchaka at gmail.com>
> I have an idea which can increase pickle/unpickle performance. This requires a change of format, so we need a new version of the protocol. May be it will be a 4 (PEP 3154) or 5.
> All items should be aligned to 4 bytes. It allow fast reading/writing of small integers, memcpy and UTF-8 codec should be faster on aligned data.

I can see several drawbacks here:
- you will still have to support unaligned data (because of memoryview
- it may not be significantly faster, because actual creation of
  objects also contributes to unpickling performance
- there will be less code sharing between different protocol versions,
  making pickle harder to maintain

If you think this is worthwhile, I think you should first draft a
proof of concept to evaluate the performance gain.



More information about the Python-ideas mailing list