Mailman 3 Make __reduce__ to correspond to __getnewargs_ex__ - Python-ideas

Make reduce to correspond to __getnewargs_ex__

older
Disable all peephole optimizations

Neil Girdhar

23 Mar 2014 23 Mar '14

5:20 p.m.

Currently __reduce__http://docs.python.org/3.4/library/pickle.html#object.__reduce__returns up to five things: (1) self.__new__ (or a substitute) (2) the result of __getnewargs__http://docs.python.org/3.4/library/pickle.html#object.__getnewargs__, which returns a tuple of positional arguments for __new__http://docs.python.org/3.4/reference/datamodel.html#object.__new__ , (3) the result of __getstate__http://docs.python.org/3.4/library/pickle.html#object.__getstate__, which returns an object to be passed to __setstate__http://docs.python.org/3.4/library/pickle.html#object.__setstate__ (4) an iterator of values for appending to a sequence (5) an iterator of key-value pairs for setting on a string. Python 3.4 added the very useful (for me) __getnewargs_ex__http://docs.python.org/3.4/library/pickle.html#object.__getnewargs_ex__, which returns a pair: (1) a tuple of positional arguments for __new__http://docs.python.org/3.4/reference/datamodel.html#object.__new__ (2) a dict of keyword arguments for __new__http://docs.python.org/3.4/reference/datamodel.html#object.__new__ Therefore, I am proposing that __reduce__ return somehow these keyword arguments for __new__. Best, Neil

Attachments:

attachment.htm (text/html — 1.5 KB)

Show replies by date

Neil Girdhar

7 Jun 7 Jun

2:39 p.m.

New subject: Make __reduce__ to correspond to __getnewargs_ex__

Any comments on this? I ended up making reduce work using a metaclass: class KwargsNewMetaclass(type): """ This metaclass reimplements __reduce__ so that it tries to call __getnewargs_ex__. If that doesn't work, it falls back to __getnewargs__. In the first case, it will pass the keyword arguments to object.__new__. It also exposes a kwargs_new static method that can be overridden for use by __reduce__. """ @staticmethod def kwargs_new(cls, new_kwargs, *new_args): retval = cls.__new__(cls, *new_args, **new_kwargs) retval.__init__(*new_args, **new_kwargs) return retval def __new__(cls, name, bases, classdict): result = super().__new__(cls, name, bases, classdict) def __reduce__(self): try: getnewargs_ex = self.__getnewargs_ex__ except AttributeError: new_args, new_kwargs = (self.__getnewargs__(), {}) else: new_args, new_kwargs = getnewargs_ex() return (self.kwargs_new(cls), (type(self), new_kwargs,) + tuple(new_args), self.__getstate__()) result.__reduce__ = __reduce__ return result On Sunday, March 23, 2014 6:20:41 PM UTC-4, Neil Girdhar wrote:

...

Currently __reduce__ http://docs.python.org/3.4/library/pickle.html#object.__reduce__ returns up to five things:

(1) self.__new__ (or a substitute) (2) the result of __getnewargs__ http://docs.python.org/3.4/library/pickle.html#object.__getnewargs__, which returns a tuple of positional arguments for __new__ http://docs.python.org/3.4/reference/datamodel.html#object.__new__, (3) the result of __getstate__ http://docs.python.org/3.4/library/pickle.html#object.__getstate__, which returns an object to be passed to __setstate__ http://docs.python.org/3.4/library/pickle.html#object.__setstate__ (4) an iterator of values for appending to a sequence (5) an iterator of key-value pairs for setting on a string.

Python 3.4 added the very useful (for me) __getnewargs_ex__ http://docs.python.org/3.4/library/pickle.html#object.__getnewargs_ex__, which returns a pair: (1) a tuple of positional arguments for __new__ http://docs.python.org/3.4/reference/datamodel.html#object.__new__ (2) a dict of keyword arguments for __new__ http://docs.python.org/3.4/reference/datamodel.html#object.__new__

Therefore, I am proposing that __reduce__ return somehow these keyword arguments for __new__.

Best, Neil

Ethan Furman

10 p.m.

New subject: Make __reduce__ to correspond to __getnewargs_ex__

On 06/07/2014 12:39 PM, Neil Girdhar wrote:

...

Any comments on this?

__reduce__ is already a well-defined part of pickle. We also have __reduce_ex__, __getnewargs__, and now __getnewargs_ex__ -- why do we need __reduce__ to do the same thing as __getnewargs_ex__? -- ~Ethan~

Neil Girdhar

8 Jun 8 Jun

1:52 a.m.

New subject: Make __reduce__ to correspond to __getnewargs_ex__

Was my proposal clear? If reduce isn't updated to return the keyword arguments then it is not compatible with classes that require keyword arguments without some serious metaclass fiddling as far as I can tell. On Sat, Jun 7, 2014 at 11:00 PM, Ethan Furman wrote:

...

On 06/07/2014 12:39 PM, Neil Girdhar wrote:

...
Any comments on this?

__reduce__ is already a well-defined part of pickle.

We also have __reduce_ex__, __getnewargs__, and now __getnewargs_ex__ -- why do we need __reduce__ to do the same thing as __getnewargs_ex__?

-- ~Ethan~ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

--

--- You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group. To unsubscribe from this topic, visit https://groups.google.com/d/ topic/python-ideas/zohH2BCtYzY/unsubscribe. To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/d/optout.

Nick Coghlan

5:03 a.m.

New subject: Make __reduce__ to correspond to __getnewargs_ex__

On 8 Jun 2014 16:53, "Neil Girdhar" wrote:

...

Was my proposal clear? If reduce isn't updated to return the keyword

arguments then it is not compatible with classes that require keyword arguments without some serious metaclass fiddling as far as I can tell. Those classes shouldn't use reduce for their pickling support. Cheers, Nick.

Neil Girdhar

12:59 p.m.

New subject: Make __reduce__ to correspond to __getnewargs_ex__

Of course they should? What should they use? On Sun, Jun 8, 2014 at 6:03 AM, Nick Coghlan wrote:

...

On 8 Jun 2014 16:53, "Neil Girdhar" wrote:

...
Was my proposal clear? If reduce isn't updated to return the keyword

arguments then it is not compatible with classes that require keyword arguments without some serious metaclass fiddling as far as I can tell.

Those classes shouldn't use reduce for their pickling support.

Cheers, Nick.

Ethan Furman

2:13 p.m.

New subject: Make __reduce__ to correspond to __getnewargs_ex__

On 06/08/2014 10:59 AM, Neil Girdhar wrote:

...

On Sun, Jun 8, 2014 at 6:03 AM, Nick Coghlan wrote:

...
Those classes shouldn't use reduce for their pickling support.

Of course they should? What should they use?

They should use the methods that make sense. Pickling is a protocol. It will use (in, I believe, this order): __getnewargs_ex__ __getnewargs__ __reduce_ex__ __reduce__ I'm not sure where __getstate__ and __setstate__ fit in, and happily I don't need to unless I'm subclassing something that makes use of them. Anyway, back to the story. When you call pickle.dump, that code will look for the most advanced method available on the object you are trying to pickle, and use it. (Well, the most advanced method for the protocol version you have selected, that's available on the object.) So, if you selected protocol 2, then the pickle code will look for __getnewargs__, but not __getnewargs_ex__, as __getnewargs_ex__ isn't available until protocol 4. This is similar to iterating: first choice for iterating is to call an object's __iter__ method, but if there isn't one Python will fall back to using __getitem__ using integers from 0 until IndexError is raised. -- ~Ethan~

Neil Girdhar

5:18 p.m.

New subject: Make __reduce__ to correspond to __getnewargs_ex__

On Sun, Jun 8, 2014 at 3:13 PM, Ethan Furman wrote:

...

On 06/08/2014 10:59 AM, Neil Girdhar wrote:

...
On Sun, Jun 8, 2014 at 6:03 AM, Nick Coghlan wrote:

...
Those classes shouldn't use reduce for their pickling support.

Of course they should? What should they use?

They should use the methods that make sense. Pickling is a protocol. It will use (in, I believe, this order):

__getnewargs_ex__

__getnewargs__

__reduce_ex__

__reduce__

In fact, the reduce functions come first. The order is: reduce_ex reduce getnewargs_ex and getstate (3.4+) getnewargs_ex (3.4+) getnewargs and getstate getnewargs getstate default code that copies/pickles the dict Up until 3.3, a "default" reduce function might be: def __reduce__(self): return type(self), self.__getnewargs__(), self.__getstate__() It' s no longer possible to write a default reduce function in 3.4 given the addtion __getnewargs_ex__. What I am suggesting is to add a new protocol so that reduce returns the keyword arguments. Best, Neil

...

I'm not sure where __getstate__ and __setstate__ fit in, and happily I don't need to unless I'm subclassing something that makes use of them.

You really should find out where they fit in before you reply to a message about them :)

...

Anyway, back to the story.

When you call pickle.dump, that code will look for the most advanced method available on the object you are trying to pickle, and use it. (Well, the most advanced method for the protocol version you have selected, that's available on the object.) So, if you selected protocol 2, then the pickle code will look for __getnewargs__, but not __getnewargs_ex__, as __getnewargs_ex__ isn't available until protocol 4.

This is similar to iterating: first choice for iterating is to call an object's __iter__ method, but if there isn't one Python will fall back to using __getitem__ using integers from 0 until IndexError is raised.

-- ~Ethan~ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

--

--- You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group. To unsubscribe from this topic, visit https://groups.google.com/d/ topic/python-ideas/zohH2BCtYzY/unsubscribe. To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/d/optout.

3609

Age (days ago)

3686

Last active (days ago)

List overview

Download

7 comments

3 participants

participants (3)

Ethan Furman
Neil Girdhar
Nick Coghlan

Make __reduce__ to correspond to __getnewargs_ex__

tags

participants (3)

Make reduce to correspond to __getnewargs_ex__