Make __reduce__ to correspond to __getnewargs_ex__
Currently __reduce__<http://docs.python.org/3.4/library/pickle.html#object.__reduce__>returns up to five things: (1) self.__new__ (or a substitute) (2) the result of __getnewargs__<http://docs.python.org/3.4/library/pickle.html#object.__getnewargs__>, which returns a tuple of positional arguments for __new__<http://docs.python.org/3.4/reference/datamodel.html#object.__new__> , (3) the result of __getstate__<http://docs.python.org/3.4/library/pickle.html#object.__getstate__>, which returns an object to be passed to __setstate__<http://docs.python.org/3.4/library/pickle.html#object.__setstate__> (4) an iterator of values for appending to a sequence (5) an iterator of key-value pairs for setting on a string. Python 3.4 added the very useful (for me) __getnewargs_ex__<http://docs.python.org/3.4/library/pickle.html#object.__getnewargs_ex__>, which returns a pair: (1) a tuple of positional arguments for __new__<http://docs.python.org/3.4/reference/datamodel.html#object.__new__> (2) a dict of keyword arguments for __new__<http://docs.python.org/3.4/reference/datamodel.html#object.__new__> Therefore, I am proposing that __reduce__ return somehow these keyword arguments for __new__. Best, Neil
Any comments on this? I ended up making reduce work using a metaclass: class KwargsNewMetaclass(type): """ This metaclass reimplements __reduce__ so that it tries to call __getnewargs_ex__. If that doesn't work, it falls back to __getnewargs__. In the first case, it will pass the keyword arguments to object.__new__. It also exposes a kwargs_new static method that can be overridden for use by __reduce__. """ @staticmethod def kwargs_new(cls, new_kwargs, *new_args): retval = cls.__new__(cls, *new_args, **new_kwargs) retval.__init__(*new_args, **new_kwargs) return retval def __new__(cls, name, bases, classdict): result = super().__new__(cls, name, bases, classdict) def __reduce__(self): try: getnewargs_ex = self.__getnewargs_ex__ except AttributeError: new_args, new_kwargs = (self.__getnewargs__(), {}) else: new_args, new_kwargs = getnewargs_ex() return (self.kwargs_new(cls), (type(self), new_kwargs,) + tuple(new_args), self.__getstate__()) result.__reduce__ = __reduce__ return result On Sunday, March 23, 2014 6:20:41 PM UTC-4, Neil Girdhar wrote:
Currently __reduce__ <http://docs.python.org/3.4/library/pickle.html#object.__reduce__> returns up to five things:
(1) self.__new__ (or a substitute) (2) the result of __getnewargs__ <http://docs.python.org/3.4/library/pickle.html#object.__getnewargs__>, which returns a tuple of positional arguments for __new__ <http://docs.python.org/3.4/reference/datamodel.html#object.__new__>, (3) the result of __getstate__ <http://docs.python.org/3.4/library/pickle.html#object.__getstate__>, which returns an object to be passed to __setstate__ <http://docs.python.org/3.4/library/pickle.html#object.__setstate__> (4) an iterator of values for appending to a sequence (5) an iterator of key-value pairs for setting on a string.
Python 3.4 added the very useful (for me) __getnewargs_ex__ <http://docs.python.org/3.4/library/pickle.html#object.__getnewargs_ex__>, which returns a pair: (1) a tuple of positional arguments for __new__ <http://docs.python.org/3.4/reference/datamodel.html#object.__new__> (2) a dict of keyword arguments for __new__ <http://docs.python.org/3.4/reference/datamodel.html#object.__new__>
Therefore, I am proposing that __reduce__ return somehow these keyword arguments for __new__.
Best, Neil
On 06/07/2014 12:39 PM, Neil Girdhar wrote:
Any comments on this?
__reduce__ is already a well-defined part of pickle. We also have __reduce_ex__, __getnewargs__, and now __getnewargs_ex__ -- why do we need __reduce__ to do the same thing as __getnewargs_ex__? -- ~Ethan~
Was my proposal clear? If reduce isn't updated to return the keyword arguments then it is not compatible with classes that require keyword arguments without some serious metaclass fiddling as far as I can tell. On Sat, Jun 7, 2014 at 11:00 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
On 06/07/2014 12:39 PM, Neil Girdhar wrote:
Any comments on this?
__reduce__ is already a well-defined part of pickle.
We also have __reduce_ex__, __getnewargs__, and now __getnewargs_ex__ -- why do we need __reduce__ to do the same thing as __getnewargs_ex__?
-- ~Ethan~ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
--
--- You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group. To unsubscribe from this topic, visit https://groups.google.com/d/ topic/python-ideas/zohH2BCtYzY/unsubscribe. To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
On 8 Jun 2014 16:53, "Neil Girdhar" <mistersheik@gmail.com> wrote:
Was my proposal clear? If reduce isn't updated to return the keyword
arguments then it is not compatible with classes that require keyword arguments without some serious metaclass fiddling as far as I can tell. Those classes shouldn't use reduce for their pickling support. Cheers, Nick.
Of course they should? What should they use? On Sun, Jun 8, 2014 at 6:03 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 8 Jun 2014 16:53, "Neil Girdhar" <mistersheik@gmail.com> wrote:
Was my proposal clear? If reduce isn't updated to return the keyword
arguments then it is not compatible with classes that require keyword arguments without some serious metaclass fiddling as far as I can tell.
Those classes shouldn't use reduce for their pickling support.
Cheers, Nick.
On 06/08/2014 10:59 AM, Neil Girdhar wrote:
On Sun, Jun 8, 2014 at 6:03 AM, Nick Coghlan wrote:
Those classes shouldn't use reduce for their pickling support.
Of course they should? What should they use?
They should use the methods that make sense. Pickling is a protocol. It will use (in, I believe, this order): __getnewargs_ex__ __getnewargs__ __reduce_ex__ __reduce__ I'm not sure where __getstate__ and __setstate__ fit in, and happily I don't need to unless I'm subclassing something that makes use of them. Anyway, back to the story. When you call pickle.dump, that code will look for the most advanced method available on the object you are trying to pickle, and use it. (Well, the most advanced method for the protocol version you have selected, that's available on the object.) So, if you selected protocol 2, then the pickle code will look for __getnewargs__, but not __getnewargs_ex__, as __getnewargs_ex__ isn't available until protocol 4. This is similar to iterating: first choice for iterating is to call an object's __iter__ method, but if there isn't one Python will fall back to using __getitem__ using integers from 0 until IndexError is raised. -- ~Ethan~
On Sun, Jun 8, 2014 at 3:13 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
On 06/08/2014 10:59 AM, Neil Girdhar wrote:
On Sun, Jun 8, 2014 at 6:03 AM, Nick Coghlan wrote:
Those classes shouldn't use reduce for their pickling support.
Of course they should? What should they use?
They should use the methods that make sense. Pickling is a protocol. It will use (in, I believe, this order):
__getnewargs_ex__
__getnewargs__
__reduce_ex__
__reduce__
In fact, the reduce functions come first. The order is: reduce_ex reduce getnewargs_ex and getstate (3.4+) getnewargs_ex (3.4+) getnewargs and getstate getnewargs getstate default code that copies/pickles the dict Up until 3.3, a "default" reduce function might be: def __reduce__(self): return type(self), self.__getnewargs__(), self.__getstate__() It' s no longer possible to write a default reduce function in 3.4 given the addtion __getnewargs_ex__. What I am suggesting is to add a new protocol so that reduce returns the keyword arguments. Best, Neil
I'm not sure where __getstate__ and __setstate__ fit in, and happily I don't need to unless I'm subclassing something that makes use of them.
You really should find out where they fit in before you reply to a message about them :)
Anyway, back to the story.
When you call pickle.dump, that code will look for the most advanced method available on the object you are trying to pickle, and use it. (Well, the most advanced method for the protocol version you have selected, that's available on the object.) So, if you selected protocol 2, then the pickle code will look for __getnewargs__, but not __getnewargs_ex__, as __getnewargs_ex__ isn't available until protocol 4.
This is similar to iterating: first choice for iterating is to call an object's __iter__ method, but if there isn't one Python will fall back to using __getitem__ using integers from 0 until IndexError is raised.
-- ~Ethan~ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
--
--- You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group. To unsubscribe from this topic, visit https://groups.google.com/d/ topic/python-ideas/zohH2BCtYzY/unsubscribe. To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
participants (3)
-
Ethan Furman -
Neil Girdhar -
Nick Coghlan