Re: [Python-ideas] Put default setstate and getstate on object for use in coöperative inheritance.
On 7 June 2014 16:05, Neil Girdhar <mistersheik@gmail.com> wrote:
I use cooperative multiple inheritance throughout my (large-ish) project, and I find it very comfortable and powerful. I am currently using the class below to serve as an anchor point. The thing is that this behavior is already implemented somewhere in Python (where?) since it is the default behaviour if getstate or setstate don't exist. Why not explicitly make it available to call super?
There is fallback behaviour in the pickle and copy modules that doesn't rely on the getstate/setstate APIs. Those fallbacks are defined by the protocols, not by the object model. https://docs.python.org/3/library/pickle.html#pickle-inst covers the available protocols for instance pickling. https://docs.python.org/3/library/copy.html covers (towards the end) some of the options for making class instances copyable https://docs.python.org/3/library/copyreg.html is an additional registry that allows third parties to make instances of classes defined elsewhere support pickling and copying without relying on monkeypatching.
I think I saw or got an email from Guido that I can't seem to find that rightly points out that object doesn't have __dict__ so this can't be done. I'm curious why object doesn't have __dict__? Where does the __dict__ comes into existence? I assume that objects of type object and instantiated objects of other types have the same metaclass; does the metaclass treat them differently?
Types defined in C extensions and those defined dynamically on the heap share a metaclass at runtime, but their initialisation code is different. You can also define Python level types without a __dict__ by declaring a __slots__ attribute with no __dict__ entry (for example, collections.namedtuple uses that to ensure namedtuple instances are exactly the same size as ordinary tuples - the mapping from field names to tuple indices is maintained on the class). Cheers, Nick. P.S. Posting through Google Groups doesn't work properly - it messes up the reply headers completely. gmane does a better job of interoperating with the mailing list software (as far as I am aware, Google just don't care whether or not interaction with non-Google lists actually works) -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Jun 7, 2014 at 2:18 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 7 June 2014 16:05, Neil Girdhar <mistersheik@gmail.com> wrote:
I use cooperative multiple inheritance throughout my (large-ish) project, and I find it very comfortable and powerful. I am currently using the class below to serve as an anchor point. The thing is that this behavior is already implemented somewhere in Python (where?) since it is the default behaviour if getstate or setstate don't exist. Why not explicitly make it available to call super?
There is fallback behaviour in the pickle and copy modules that doesn't rely on the getstate/setstate APIs. Those fallbacks are defined by the protocols, not by the object model.
Those fallbacks are essentially default implementations of setstate and getstate. It seems to me like it would make sense to implement those fallbacks once rather than twice in the various places that you mention.
https://docs.python.org/3/library/pickle.html#pickle-inst covers the available protocols for instance pickling. https://docs.python.org/3/library/copy.html covers (towards the end) some of the options for making class instances copyable
Yes, personally, I prefer writing setstate and getstate and getting copy for free rather than writing a separate __copy__ method.
https://docs.python.org/3/library/copyreg.html is an additional registry that allows third parties to make instances of classes defined elsewhere support pickling and copying without relying on monkeypatching.
copyreg is unfortunately no use for cooperative inheritance as far as I can see. The whole point is for each class to pickle what it needs to and delegate the rest of the pickling to super.
I think I saw or got an email from Guido that I can't seem to find that rightly points out that object doesn't have __dict__ so this can't be done. I'm curious why object doesn't have __dict__? Where does the __dict__ comes into existence? I assume that objects of type object and instantiated objects of other types have the same metaclass; does the metaclass treat them differently?
Types defined in C extensions and those defined dynamically on the heap share a metaclass at runtime, but their initialisation code is different. You can also define Python level types without a __dict__ by declaring a __slots__ attribute with no __dict__ entry (for example, collections.namedtuple uses that to ensure namedtuple instances are exactly the same size as ordinary tuples - the mapping from field names to tuple indices is maintained on the class).
Very interesting, thanks for explaining what is happening. I don't see why __dict__ isn't just in object though. Is it just for the (minor) efficiency of saving an empty dict reference?
Cheers, Nick.
P.S. Posting through Google Groups doesn't work properly - it messes up the reply headers completely. gmane does a better job of interoperating with the mailing list software (as far as I am aware, Google just don't care whether or not interaction with non-Google lists actually works)
Sorry, I'm just answering via email. I don't know anything about gmane.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Jun 7, 2014 at 2:18 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 7 June 2014 16:05, Neil Girdhar <mistersheik@gmail.com> wrote:
I use cooperative multiple inheritance throughout my (large-ish)
On 7 Jun 2014 16:37, "Neil Girdhar" <mistersheik@gmail.com> wrote: project,
and I find it very comfortable and powerful. I am currently using the class below to serve as an anchor point. The thing is that this behavior is already implemented somewhere in Python (where?) since it is the default behaviour if getstate or setstate don't exist. Why not explicitly make it available to call super?
There is fallback behaviour in the pickle and copy modules that doesn't rely on the getstate/setstate APIs. Those fallbacks are defined by the protocols, not by the object model.
Those fallbacks are essentially default implementations of setstate and getstate. It seems to me like it would make sense to implement those fallbacks once rather than twice in the various places that you mention.
I think I saw or got an email from Guido that I can't seem to find that rightly points out that object doesn't have __dict__ so this can't be done. I'm curious why object doesn't have __dict__? Where does the __dict__ comes into existence? I assume that objects of type object and instantiated objects of other types have the same metaclass; does the metaclass
As far as I am aware, it's not implemented in two places - I believe copy falls back pickling & unpickling if there's no other copy operation defined. We don't try to jam everything into the base object, as library protocols are easier to evolve without breaking backwards compatibility. (For CPython, there's also the practical consideration that "object" methods have to be implemented in C, so having protocol fallbacks in the standard library sometimes makes them easier to work on). treat
them differently?
Types defined in C extensions and those defined dynamically on the heap share a metaclass at runtime, but their initialisation code is different. You can also define Python level types without a __dict__ by declaring a __slots__ attribute with no __dict__ entry (for example, collections.namedtuple uses that to ensure namedtuple instances are exactly the same size as ordinary tuples - the mapping from field names to tuple indices is maintained on the class).
Very interesting, thanks for explaining what is happening. I don't see why __dict__ isn't just in object though. Is it just for the (minor) efficiency of saving an empty dict reference?
A reference is a 64-bit pointer. That would be additional overhead on *every single object*. All ints, all strings, all tuples, all dicts(!), etc. Saving 8 bytes per object adds up fast, which is why a lot of the core types (including object itself) don't have a per-instance __dict__ attribute. Keeping objects as small as possible also impacts how many will fit in the CPU cache, so this approach can end up providing a speed increase as well. Cheers, Nick.
Cheers, Nick.
P.S. Posting through Google Groups doesn't work properly - it messes up the reply headers completely. gmane does a better job of interoperating with the mailing list software (as far as I am aware, Google just don't care whether or not interaction with non-Google lists actually works)
Sorry, I'm just answering via email. I don't know anything about gmane.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Jun 7, 2014 at 4:41 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Sat, Jun 7, 2014 at 2:18 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 7 June 2014 16:05, Neil Girdhar <mistersheik@gmail.com> wrote:
I use cooperative multiple inheritance throughout my (large-ish)
and I find it very comfortable and powerful. I am currently using
On 7 Jun 2014 16:37, "Neil Girdhar" <mistersheik@gmail.com> wrote: project, the class
below to serve as an anchor point. The thing is that this behavior is already implemented somewhere in Python (where?) since it is the default behaviour if getstate or setstate don't exist. Why not explicitly make it available to call super?
There is fallback behaviour in the pickle and copy modules that doesn't rely on the getstate/setstate APIs. Those fallbacks are defined by the protocols, not by the object model.
Those fallbacks are essentially default implementations of setstate and getstate. It seems to me like it would make sense to implement those fallbacks once rather than twice in the various places that you mention.
As far as I am aware, it's not implemented in two places - I believe copy falls back pickling & unpickling if there's no other copy operation defined.
We don't try to jam everything into the base object, as library protocols are easier to evolve without breaking backwards compatibility. (For CPython, there's also the practical consideration that "object" methods have to be implemented in C, so having protocol fallbacks in the standard library sometimes makes them easier to work on).
I see your point.
I think I saw or got an email from Guido that I can't seem to find that rightly points out that object doesn't have __dict__ so this can't be done. I'm curious why object doesn't have __dict__? Where does the __dict__ comes into existence? I assume that objects of type object and instantiated objects of other types have the same metaclass; does the metaclass treat them differently?
Types defined in C extensions and those defined dynamically on the heap share a metaclass at runtime, but their initialisation code is different. You can also define Python level types without a __dict__ by declaring a __slots__ attribute with no __dict__ entry (for example, collections.namedtuple uses that to ensure namedtuple instances are exactly the same size as ordinary tuples - the mapping from field names to tuple indices is maintained on the class).
Very interesting, thanks for explaining what is happening. I don't see why __dict__ isn't just in object though. Is it just for the (minor) efficiency of saving an empty dict reference?
A reference is a 64-bit pointer. That would be additional overhead on *every single object*. All ints, all strings, all tuples, all dicts(!), etc. Saving 8 bytes per object adds up fast, which is why a lot of the core types (including object itself) don't have a per-instance __dict__ attribute.
Keeping objects as small as possible also impacts how many will fit in the CPU cache, so this approach can end up providing a speed increase as well.
Right, that makes sense. I think the flyweight pattern would eliminate this: use a special representation for the common case and then switch to a real representation as soon as things become weird. (I can see how that would be extra development time unless it could be done automatically by a clever JIT.) Best, Neil
Cheers, Nick.
Cheers, Nick.
P.S. Posting through Google Groups doesn't work properly - it messes up the reply headers completely. gmane does a better job of interoperating with the mailing list software (as far as I am aware, Google just don't care whether or not interaction with non-Google lists actually works)
Sorry, I'm just answering via email. I don't know anything about gmane.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 7 June 2014 18:46, Neil Girdhar <mistersheik@gmail.com> wrote:
Right, that makes sense. I think the flyweight pattern would eliminate this: use a special representation for the common case and then switch to a real representation as soon as things become weird. (I can see how that would be extra development time unless it could be done automatically by a clever JIT.)
The flyweight pattern imposes its own costs in terms of additional levels of indirection and even more pointers to carry around. The approach we take is that object instances get a __dict__ attribute by default, unless the creator of the class decides "there are going to be enough of these for it to be worth skipping the space not only for the attribute dicts themselves, but also for the attribute dict reference on each instance". We do the same with weakref support. The other thing to keep in mind is that many of CPython's "internal" representations aren't actually internal: many of them are exposed in various ways through the CPython C API. As other implementations have discovered, preserving full compatibility with that API places some pretty significant constraints on the implementation techniques you use (or else means putting a lot of work into a compatibility shim layer like IronClad, JyNI or cpyext). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I understand your concern for cpython, but I don't think it will be the future of Python. I think every object should have a dict and then the JIT should just make it fast. I think that's possible. Anyway, this is a separate discussion. My new proposal is for setstate and getstate to have default implementations that first check for the __dict__ attribute and do the normal thing (getstate returns {}, setstate does nothing) if it doesn't exist. Best, Neil On Sat, Jun 7, 2014 at 5:34 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 7 June 2014 18:46, Neil Girdhar <mistersheik@gmail.com> wrote:
Right, that makes sense. I think the flyweight pattern would eliminate this: use a special representation for the common case and then switch
to a
real representation as soon as things become weird. (I can see how that would be extra development time unless it could be done automatically by a clever JIT.)
The flyweight pattern imposes its own costs in terms of additional levels of indirection and even more pointers to carry around. The approach we take is that object instances get a __dict__ attribute by default, unless the creator of the class decides "there are going to be enough of these for it to be worth skipping the space not only for the attribute dicts themselves, but also for the attribute dict reference on each instance". We do the same with weakref support.
The other thing to keep in mind is that many of CPython's "internal" representations aren't actually internal: many of them are exposed in various ways through the CPython C API. As other implementations have discovered, preserving full compatibility with that API places some pretty significant constraints on the implementation techniques you use (or else means putting a lot of work into a compatibility shim layer like IronClad, JyNI or cpyext).
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
You haven't explained why you need this. You just stated a proposal. On Jun 7, 2014 12:06 PM, "Neil Girdhar" <mistersheik@gmail.com> wrote:
I understand your concern for cpython, but I don't think it will be the future of Python. I think every object should have a dict and then the JIT should just make it fast. I think that's possible.
Anyway, this is a separate discussion. My new proposal is for setstate and getstate to have default implementations that first check for the __dict__ attribute and do the normal thing (getstate returns {}, setstate does nothing) if it doesn't exist.
Best, Neil
On Sat, Jun 7, 2014 at 5:34 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 7 June 2014 18:46, Neil Girdhar <mistersheik@gmail.com> wrote:
Right, that makes sense. I think the flyweight pattern would eliminate this: use a special representation for the common case and then switch
to a
real representation as soon as things become weird. (I can see how that would be extra development time unless it could be done automatically by a clever JIT.)
The flyweight pattern imposes its own costs in terms of additional levels of indirection and even more pointers to carry around. The approach we take is that object instances get a __dict__ attribute by default, unless the creator of the class decides "there are going to be enough of these for it to be worth skipping the space not only for the attribute dicts themselves, but also for the attribute dict reference on each instance". We do the same with weakref support.
The other thing to keep in mind is that many of CPython's "internal" representations aren't actually internal: many of them are exposed in various ways through the CPython C API. As other implementations have discovered, preserving full compatibility with that API places some pretty significant constraints on the implementation techniques you use (or else means putting a lot of work into a compatibility shim layer like IronClad, JyNI or cpyext).
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Hi, Okay. In my project I have many classes multiply inheriting from each other. Most of these classes derive from "NetworkElement" and objects of this type are stored in a tree. I would now like to serialize the tree of objects so that I can save the state of the network. I would also like to instantiate copies of the tree so that I can rewind the state of the network to given checkpoints and play back the simulation. The easiest way to implement both serialize and copy in such a way that they are consistent (serialization and deserialization is equivalent to copy) is to implement setstate and getstate. In cooperative inheritance, the general pattern is to call super and do whatever is particular to your class around that. I needed to inherit from the mixin I displayed at the top of this message in order to provide a default setstate and getstate as these are not present in object. Intuitively, I think that it would be better for these to exist on object. I don't think I should have to provide these methods using a mixin. It's not a big deal, but I think it's a small wrinkle in Python not to have default implementations of these methods given that that default behavior is being done anyway. Are there any drawbacks to providing these default methods? Best, Neil On Sat, Jun 7, 2014 at 3:12 PM, Guido van Rossum <guido@python.org> wrote:
You haven't explained why you need this. You just stated a proposal. On Jun 7, 2014 12:06 PM, "Neil Girdhar" <mistersheik@gmail.com> wrote:
I understand your concern for cpython, but I don't think it will be the future of Python. I think every object should have a dict and then the JIT should just make it fast. I think that's possible.
Anyway, this is a separate discussion. My new proposal is for setstate and getstate to have default implementations that first check for the __dict__ attribute and do the normal thing (getstate returns {}, setstate does nothing) if it doesn't exist.
Best, Neil
On Sat, Jun 7, 2014 at 5:34 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 7 June 2014 18:46, Neil Girdhar <mistersheik@gmail.com> wrote:
Right, that makes sense. I think the flyweight pattern would eliminate this: use a special representation for the common case and then switch
real representation as soon as things become weird. (I can see how
to a that
would be extra development time unless it could be done automatically by a clever JIT.)
The flyweight pattern imposes its own costs in terms of additional levels of indirection and even more pointers to carry around. The approach we take is that object instances get a __dict__ attribute by default, unless the creator of the class decides "there are going to be enough of these for it to be worth skipping the space not only for the attribute dicts themselves, but also for the attribute dict reference on each instance". We do the same with weakref support.
The other thing to keep in mind is that many of CPython's "internal" representations aren't actually internal: many of them are exposed in various ways through the CPython C API. As other implementations have discovered, preserving full compatibility with that API places some pretty significant constraints on the implementation techniques you use (or else means putting a lot of work into a compatibility shim layer like IronClad, JyNI or cpyext).
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On 8 Jun 2014 05:26, "Neil Girdhar" <mistersheik@gmail.com> wrote:
In cooperative inheritance, the general pattern is to call super and do
whatever is particular to your class around that. I needed to inherit from the mixin I displayed at the top of this message in order to provide a default setstate and getstate as these are not present in object. Intuitively, I think that it would be better for these to exist on object. I don't think I should have to provide these methods using a mixin. You haven't explained why you're trying to do cooperative multiple inheritance without a common base class to define the rules for your type system. Leaving that element out of a cooperative multiple inheritance design is generally a really bad idea.
It's not a big deal, but I think it's a small wrinkle in Python not to have default implementations of these methods given that that default behavior is being done anyway. Are there any drawbacks to providing these default methods?
Yes - increased complexity in the language core. Currently, pickling is completely independent of the language core, so implementations can reuse the same pickling library (although they may want to write an accelerated version eventually). Cheers, Nick.
Yes - increased complexity in the language core. Currently, pickling is completely independent of the language core, so implementations can reuse
Nick Coghlan <ncoghlan@...> writes: the same pickling library (although they may want to write an accelerated version eventually). That's not really true considering the amount of pickle goop in typeobject.c and the fact that many builtin types implement their own pickling.
Neil Girdhar writes:
Sorry, I'm just answering via email. I don't know anything about gmane.
Then please change the To: from @googlegroups to @python.org by hand. If that's too annoying to do every time, learn about GMane once. :-)
participants (5)
-
Benjamin Peterson -
Guido van Rossum -
Neil Girdhar -
Nick Coghlan -
Stephen J. Turnbull