I have an idea which can increase pickle/unpickle performance. This requires a change of format, so we need a new version of the protocol. May be it will be a 4 (PEP 3154) or 5.
All items should be aligned to 4 bytes. It allow fast reading/writing of small integers, memcpy and UTF-8 codec should be faster on aligned data.
In order not to waste space, a byte code should be combined with the data or the size of data.
For integers:
<code> <24-bit integer> <code> <24-bit size> <size-byte integer> <padding> <code> <56-bit size> <size-byte integer> <padding>
For strings:
<code> <8-bit size> <size-byte string> <padding> <code> <24-bit size> <size-byte string> <padding> <code> <56-bit size> <size-byte string> <padding>
For collections:
<code> <24-bit size> <item1> <item2> ... <item #size> <code> <56-bit size> <item1> <item2> ... <item #size>
For references:
<code> <24-bit index> <code> <56-bit index>
For 1- and 2-byte integers, this can be expensive. We can add a special code for grouping. It will be even shorter than in the old protocols.
<group code> <item code> <16-bit count> <integer1> <integer2> ... <integer #count> <padding>
What do you think about this?
Hello,
On Sat, 17 Nov 2012 23:18:29 +0200 Serhiy Storchaka storchaka@gmail.com wrote:
I have an idea which can increase pickle/unpickle performance. This requires a change of format, so we need a new version of the protocol. May be it will be a 4 (PEP 3154) or 5.
All items should be aligned to 4 bytes. It allow fast reading/writing of small integers, memcpy and UTF-8 codec should be faster on aligned data.
I can see several drawbacks here: - you will still have to support unaligned data (because of memoryview slices) - it may not be significantly faster, because actual creation of objects also contributes to unpickling performance - there will be less code sharing between different protocol versions, making pickle harder to maintain
If you think this is worthwhile, I think you should first draft a proof of concept to evaluate the performance gain.
Regards
Antoine.