[Python-Dev] Increasing the C-optimized pickle extensibility

[Pierre Glaser]

Hi All, 

We (Antoine Pitrou, Olivier Grisel and myself) spent some efforts recently on 
enabling pickle extensions to extend the C-optimized Pickler instead of the 
pure Python one. 

Pickle extensions have a crucial role in many distributed computing libraries: 
cloudpickle (https://github.com/cloudpipe/cloudpickle) for example is vendored 
in dask, pyspark, ray, and joblib. 
Early benchmarks show that relying on the C-optimized pickle yields 
significant serialization speed improvements (up to 30x faster). 
(draft PR of the CPickler-backed version of cloudpickle: https://github.com/cloudpipe/cloudpickle/pull/253) 

To make extending the C Pickler possible, we are currently moving forward with 
a few enhancements to the public pickle API. 

* First, we are enabling Pickler subclasses to implement a reducer_override 
method, that will be have priority over the registered reducers in the 
dispatch_table and over the default handling of classes and functions. 
(PR link: https://github.com/python/cpython/pull/12499) 

* Then, we are adding a new keyword argument to save_reduce called state_setter. 
(consequently we allow a reducer's return value to have a new, 6th item). 
This state setter callable is useful to override programmatically the state updating 
behavior of an object, that would otherwise be restricted to its static 
``__setstate__`` method. 
(PR link: https://github.com/python/cpython/pull/12588) 

The PR review process of these changes is in progress, and anyone is welcomed 
to chime in and share some thoughts. 

The first addition is very non-invasive. We estimated that the second point did 
not require introducing a new opcode, as this change could be implemented as 
simple sequence of standard pickle instructions. We therefore think that it is 
not necessary to make this change dependent on the new protocol 5 proposed in 
PEP 574. 

The key advantage in not creating a new opcode that this makes our change 
backward-compatible, meaning that 3.8-written pickles will not break because of 
our change if read using earlier Python versions. 

OTOH, one might argue that a new OPCODE might 
* make the code a little bit cleaner 
* make it easier to interpret disassembled pickle strings. 

If you are interested, here is an example of a disassembled pickle string 
using our currently proposed solution: 
https://github.com/pierreglaser/cpython/pull/2#issuecomment-486243350 

Does anyone have an opinion on this? 

Thanks, 

Pierre 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190426/6578a59e/attachment-0001.html>