<div dir="ltr">I think it's better not to introduce a new opcode, for the reason you stated -- you don't want your pickles to be unreadable by older Python versions, if you can help it.<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 26, 2019 at 5:59 AM Pierre Glaser <<a href="mailto:pierre.glaser@inria.fr">pierre.glaser@inria.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div>Hi All,<br><br>We (Antoine Pitrou, Olivier Grisel and myself) spent some efforts recently on<br>enabling pickle extensions to extend the C-optimized Pickler instead of the<br>pure Python one.<br><br>Pickle extensions have a crucial role in many distributed computing libraries:<br>cloudpickle (<a href="https://github.com/cloudpipe/cloudpickle" target="_blank">https://github.com/cloudpipe/cloudpickle</a>) for example is vendored<br>in dask, pyspark, ray, and joblib.<br>Early benchmarks show that relying on the C-optimized pickle yields<br>significant serialization speed improvements (up to 30x faster).<br>(draft PR of the CPickler-backed version of cloudpickle: <a href="https://github.com/cloudpipe/cloudpickle/pull/253" target="_blank">https://github.com/cloudpipe/cloudpickle/pull/253</a>)<br><br>To make extending the C Pickler possible, we are currently moving forward with<br>a few enhancements to the public pickle API.<br><br>* First, we are enabling Pickler subclasses to implement a reducer_override<br>  method, that will be have priority over the registered reducers in the<br>  dispatch_table and over the default handling of classes and functions.<br>  (PR link: <a href="https://github.com/python/cpython/pull/12499" target="_blank">https://github.com/python/cpython/pull/12499</a>)<br><br>* Then, we are adding a new keyword argument to save_reduce called state_setter.<br>  (consequently we allow a reducer's return value to have a new, 6th item).<br>  This state setter callable is useful to override programmatically the state updating<br>  behavior of an object, that would otherwise be restricted to its static<br>  ``__setstate__`` method.<br>  (PR link: <a href="https://github.com/python/cpython/pull/12588" target="_blank">https://github.com/python/cpython/pull/12588</a>)<br><br>The PR review process of these changes is in progress, and anyone is welcomed<br>to chime in and share some thoughts.<br><br>The first addition is very non-invasive. We estimated that the second point did<br>not require introducing a new opcode, as this change could be implemented as<br>simple sequence of standard pickle instructions. We therefore think that it is<br>not necessary to make this change dependent on the new protocol 5 proposed in<br>PEP 574.<br><br>The key advantage in not creating a new opcode that this makes our change<br>backward-compatible, meaning that 3.8-written pickles will not break because of<br>our change if read using earlier Python versions.<br><br>OTOH, one might argue that a new OPCODE might<br>* make the code a little bit cleaner<br>* make it easier to interpret disassembled pickle strings.<br><br>If you are interested, here is an example of a disassembled pickle string<br>using our currently proposed solution:<br><a href="https://github.com/pierreglaser/cpython/pull/2#issuecomment-486243350" target="_blank">https://github.com/pierreglaser/cpython/pull/2#issuecomment-486243350</a><br><br>Does anyone have an opinion on this?<br><br></div><div>Thanks, <br></div><div><br></div><div>Pierre<br></div></div></div>_______________________________________________<br>
Python-Dev mailing list<br>
<a href="mailto:Python-Dev@python.org" target="_blank">Python-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-dev" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="https://mail.python.org/mailman/options/python-dev/guido%40python.org" rel="noreferrer" target="_blank">https://mail.python.org/mailman/options/python-dev/guido%40python.org</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>--Guido van Rossum (<a href="http://python.org/~guido" target="_blank">python.org/~guido</a>)</div><div><i style="font-family:Arial,Helvetica,sans-serif;font-size:small;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);color:rgb(136,136,136)"><span>Pronouns</span>: he/him/his </i><a href="http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/" style="color:rgb(17,85,204);font-family:Arial,Helvetica,sans-serif;font-size:small;font-style:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" target="_blank"><i>(why is my <span>pronoun</span> here?)</i></a></div></div></div>