I am generating extension types derived from a type which is derived from int 'int' by calling the metaclass; in order to prevent instances of the most-derived type from getting an instance __dict__ I am putting an empty tuple in the class __dict__ as '__slots__'. The problem with this hack is that it disables pickling of these babies: "a class that defines __slots__ without defining __getstate__ cannot be pickled" Yes, I can define __getstate__, __setstate__, and __getinitargs__ (the only one that can actually do any work, since ints are immutable), but I was wondering if there's a more straightforward way to suppress the instance __dict__ in the derived classes. TIA, -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams <dave@boost-consulting.com> writes:
Yes, I can define __getstate__, __setstate__, and __getinitargs__ (the only one that can actually do any work, since ints are immutable), but I was wondering if there's a more straightforward way to suppress the instance __dict__ in the derived classes.
Setting tp_dictoffset to 0 might help. However, I'm unsure what consequences this has; read the source. Regards, Martin
martin@v.loewis.de (Martin v. Löwis) writes:
David Abrahams <dave@boost-consulting.com> writes:
Yes, I can define __getstate__, __setstate__, and __getinitargs__ (the only one that can actually do any work, since ints are immutable), but I was wondering if there's a more straightforward way to suppress the instance __dict__ in the derived classes.
Setting tp_dictoffset to 0 might help.
AFAICT I don't get to do that, since as I wrote: I am generating extension types derived from a type which is derived from int 'int' by calling the metaclass ^^^^^^^^^^^^^^^^^^^^^^^^
However, I'm unsure what consequences this has; read the source.
Unfortunately, this is one of the twistiest areas of the Python source, so while I could struggle through it I'm hoping there's someone around here who knows the answer off the top of his benevolent Dutch head <wink> -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams <dave@boost-consulting.com> writes:
Unfortunately, this is one of the twistiest areas of the Python source, so while I could struggle through it I'm hoping there's someone around here who knows the answer off the top of his benevolent Dutch head <wink>
Well, I'm familiar enough with that bit of the source (search for "add_dict" in typeobject.c) to answer your question: no, there's no more straightforward way to suppress the instance __dict__ in the derived classes. Cheers, M. -- The rapid establishment of social ties, even of a fleeting nature, advance not only that goal but its standing in the uberconscious mesh of communal psychic, subjective, and algorithmic interbeing. But I fear I'm restating the obvious. -- Will Ware, comp.lang.python
I am generating extension types derived from a type which is derived from int 'int' by calling the metaclass; in order to prevent instances of the most-derived type from getting an instance __dict__ I am putting an empty tuple in the class __dict__ as '__slots__'. The problem with this hack is that it disables pickling of these babies:
"a class that defines __slots__ without defining __getstate__ cannot be pickled"
Yes, I can define __getstate__, __setstate__, and __getinitargs__ (the only one that can actually do any work, since ints are immutable), but I was wondering if there's a more straightforward way to suppress the instance __dict__ in the derived classes.
Actually, even __getinitargs__ won't work, because __init__ is called after the object is created. In Python 2.3, you'd use __getnewargs__, but I expect you're still bound to supporting Python 2.2 (Python 2.3 also doesn't have the error message above when pickling). I think you could subclass the metaclass, override __new__, and delete the bogus __getstate__ from the type's __dict__. Then you'll get the default pickling behavior which ignores slots; that should work just fine in your case. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum <guido@python.org> writes:
I am generating extension types derived from a type which is derived from int 'int' by calling the metaclass; in order to prevent instances of the most-derived type from getting an instance __dict__ I am putting an empty tuple in the class __dict__ as '__slots__'. The problem with this hack is that it disables pickling of these babies:
"a class that defines __slots__ without defining __getstate__ cannot be pickled"
Yes, I can define __getstate__, __setstate__, and __getinitargs__ (the only one that can actually do any work, since ints are immutable), but I was wondering if there's a more straightforward way to suppress the instance __dict__ in the derived classes.
Actually, even __getinitargs__ won't work, because __init__ is called after the object is created.
...and ints are immutable. Right.
In Python 2.3, you'd use __getnewargs__,
Cute. It's almost too bad that the distinction between __new__ and __init__ is there -- as we find we need to legitimize the use of __new__ with things like __getnewargs__ it be comes a little less clear which one should be used, and when. TIMTOWDI and all that. In the absence of clear guidelines I'm tempted to suggest that C++ got this part right. Occasionally we get people who think they want to call overridden virtual functions from constructors (I presume the analogous thing could be done safely from __init__ but not from __new__) but that's pretty rare. I'm interested in gaining insight into the Pythonic thinking behind __new__/__init__; I'm sure I don't have the complete picture.
but I expect you're still bound to supporting Python 2.2
Yup, I think it would be bad to force my users to move to an unreleased Python version at this point ;-)
(Python 2.3 also doesn't have the error message above when pickling).
Nice. Too bad about 2.2.
I think you could subclass the metaclass, override __new__, and delete the bogus __getstate__ from the type's __dict__. Then you'll get the default pickling behavior which ignores slots; that should work just fine in your case. :-)
Ooh, that's sneaky! But I can't quite see how it works. The error message I quoted at the top about __getstate__ happens when you try to pickle an instance of the class. If I delete __getstate__ during __new__, it won't be there for pickle to find when I try to do the pickling. What will keep it from inducing the same error? Thanks, -- Dave Abrahams Boost Consulting www.boost-consulting.com
It's almost too bad that the distinction between __new__ and __init__ is there -- as we find we need to legitimize the use of __new__ with things like __getnewargs__ it be comes a little less clear which one should be used, and when. TIMTOWDI and all that.
__new__ creates a new, initialized object. __init__ sets some values in an exsting object. __init__ is a regular method and can be called to reinitialize an existing object (not that I recommend this, but the mechanism doesn't forbid it). It follows that immutable objects must be initialized using __new__, since by the time __init__ is called the object already exists and is immutable.
In the absence of clear guidelines I'm tempted to suggest that C++ got this part right.
Of course you would. I tend to think that Python's analogon to C++ constructors is __new__, and that __init__ is a different mechanism (although it can often be used where you would use a constructor in C++).
Occasionally we get people who think they want to call overridden virtual functions from constructors (I presume the analogous thing could be done safely from __init__ but not from __new__)
Whether or not that can be done safely from __init__ depends on the subclass __init__; it's easy enough to construct examples that don't work. But yes, for __new__ the situation is more analogous to C++, except that AFAIK in C++ when you try that you get the base class virtual function, while in Python you get the overridden method -- which finds an instance that is incompletely initialized.
but that's pretty rare. I'm interested in gaining insight into the Pythonic thinking behind __new__/__init__; I'm sure I don't have the complete picture.
__new__ was introduced to allow initializing immutable objects. It really applies more to types implemented in C than types implemented in Python. But it is needed so that a Python subclass of an immutable C base classs can pass arguments of its choice to the C base class's constructor.
Nice. Too bad about 2.2.
Maybe the new pickling could be backported, but I fear that it depends on some other 2.3 feature that's harder to backport, so I haven't looked into this. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum <guido@python.org> writes:
It's almost too bad that the distinction between __new__ and __init__ is there -- as we find we need to legitimize the use of __new__ with things like __getnewargs__ it be comes a little less clear which one should be used, and when. TIMTOWDI and all that.
__new__ creates a new, initialized object. __init__ sets some values in an exsting object. __init__ is a regular method and can be called to reinitialize an existing object (not that I recommend this, but the mechanism doesn't forbid it). It follows that immutable objects must be initialized using __new__, since by the time __init__ is called the object already exists and is immutable.
Shouldn't most objects be initialized by __new__, really? IME it's dangerous to have uninitialized objects floating about, especially in the presence of exceptions.
In the absence of clear guidelines I'm tempted to suggest that C++ got this part right.
Of course you would.
Oh, c'mon. C++ is ugly, both brittle *and* inflexible, expensive, painful, etc. There must be at least _one_ well-designed thing about it. Maybe this is it!
I tend to think that Python's analogon to C++ constructors is __new__,
Yup.
and that __init__ is a different mechanism (although it can often be used where you would use a constructor in C++).
Occasionally we get people who think they want to call overridden virtual functions from constructors (I presume the analogous thing could be done safely from __init__ but not from __new__)
Whether or not that can be done safely from __init__ depends on the subclass __init__; it's easy enough to construct examples that don't work. But yes, for __new__ the situation is more analogous to C++, except that AFAIK in C++ when you try that you get the base class virtual function, while in Python you get the overridden method -- which finds an instance that is incompletely initialized.
Either one seems equally likely to be what you don't want.
but that's pretty rare. I'm interested in gaining insight into the Pythonic thinking behind __new__/__init__; I'm sure I don't have the complete picture.
__new__ was introduced to allow initializing immutable objects. It really applies more to types implemented in C than types implemented in Python. But it is needed so that a Python subclass of an immutable C base classs can pass arguments of its choice to the C base class's constructor.
Nice. Too bad about 2.2.
Maybe the new pickling could be backported, but I fear that it depends on some other 2.3 feature that's harder to backport, so I haven't looked into this.
Are people who don't want to upgrade really that much more willing if it doesn't involve a minor revision number? I figure that if I supported 2.2 once, I'd have to be very circumspect about doing something which required an upgrade to 2.2.x. -- Dave Abrahams Boost Consulting www.boost-consulting.com
[David]
It's almost too bad that the distinction between __new__ and __init__ is there -- as we find we need to legitimize the use of __new__ with things like __getnewargs__ it be comes a little less clear which one should be used, and when. TIMTOWDI and all that.
[Guido]
__new__ creates a new, initialized object. __init__ sets some values in an exsting object. __init__ is a regular method and can be called to reinitialize an existing object (not that I recommend this, but the mechanism doesn't forbid it). It follows that immutable objects must be initialized using __new__, since by the time __init__ is called the object already exists and is immutable.
[David again]
Shouldn't most objects be initialized by __new__, really? IME it's dangerous to have uninitialized objects floating about, especially in the presence of exceptions.
Normally, there are no external references to an object until after __init__ returns, so you should be safe unless __init__ saves a reference to self somewhere. It does mean that __del__ can be surprised by an uninitialized object, and that's a known pitfall. And an exception in the middle of __new__ has the same problem. So I don't think __new__ is preferred over __init__, unless you need a feature that only __new__ offers (like initializing an immutable base class or returning an existing object or an object of a different class).
In the absence of clear guidelines I'm tempted to suggest that C++ got this part right.
Of course you would.
Oh, c'mon. C++ is ugly, both brittle *and* inflexible, expensive, painful, etc. There must be at least _one_ well-designed thing about it. Maybe this is it!
You said it. :-)
I tend to think that Python's analogon to C++ constructors is __new__,
Yup.
and that __init__ is a different mechanism (although it can often be used where you would use a constructor in C++).
Occasionally we get people who think they want to call overridden virtual functions from constructors (I presume the analogous thing could be done safely from __init__ but not from __new__)
Whether or not that can be done safely from __init__ depends on the subclass __init__; it's easy enough to construct examples that don't work. But yes, for __new__ the situation is more analogous to C++, except that AFAIK in C++ when you try that you get the base class virtual function, while in Python you get the overridden method -- which finds an instance that is incompletely initialized.
Either one seems equally likely to be what you don't want.
Yeah, this is something where you can't seem to win. :-(
but that's pretty rare. I'm interested in gaining insight into the Pythonic thinking behind __new__/__init__; I'm sure I don't have the complete picture.
__new__ was introduced to allow initializing immutable objects. It really applies more to types implemented in C than types implemented in Python. But it is needed so that a Python subclass of an immutable C base classs can pass arguments of its choice to the C base class's constructor.
Nice. Too bad about 2.2.
Maybe the new pickling could be backported, but I fear that it depends on some other 2.3 feature that's harder to backport, so I haven't looked into this.
Are people who don't want to upgrade really that much more willing if it doesn't involve a minor revision number? I figure that if I supported 2.2 once, I'd have to be very circumspect about doing something which required an upgrade to 2.2.x.
The idea is that an upgrade from 2.2.x to 2.2.(x+1) won't break any code, it will only fix bugs. For example, Zope requires Python 2.2.1 because of a particular bug in Python 2.2[.0] that caused Zope core dumps. Of course, the "breaks no code" promise can't be true 100% (because some code could depend on a bug), but we try a lot harder not to break stuff than with a 2.x to 2.(x+1). Even though there we also try not to break stuff, we're less anal about it (otherwise the language would just get uglier and uglier by maintaining strict backwards compatibility with all past mistakes). --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum <guido@python.org> writes:
[Guido]
__new__ creates a new, initialized object. __init__ sets some values in an exsting object. __init__ is a regular method and can be called to reinitialize an existing object (not that I recommend this, but the mechanism doesn't forbid it). It follows that immutable objects must be initialized using __new__, since by the time __init__ is called the object already exists and is immutable.
[David again]
Shouldn't most objects be initialized by __new__, really? IME it's dangerous to have uninitialized objects floating about, especially in the presence of exceptions.
Normally, there are no external references to an object until after __init__ returns
Good point; that's a feature you don't get unless you build two-phase initialization into the core language. Two-phase initialization is more dangerous in C++ because it's not a core language feature.
so you should be safe unless __init__ saves a reference to self somewhere. It does mean that __del__ can be surprised by an uninitialized object, and that's a known pitfall. And an exception in the middle of __new__ has the same problem.
C++ deals with that by only destroying the fully-initialized bases and subobjects when an exception is thrown during construction. That's hard to do in the presence of two-phase initialization, though. It may be less of a problem for Python because __del__ is much less commonly needed than nontrivial destructors are in C++.
So I don't think __new__ is preferred over __init__, unless you need a feature that only __new__ offers (like initializing an immutable base class or returning an existing object or an object of a different class).
In other words, TIMTOWTDI? <0.3 wink> -- Dave Abrahams Boost Consulting www.boost-consulting.com
participants (5)
-
David Abrahams -
Guido van Rossum -
martin＠v.loewis.de -
Michael Hudson -
Oren Tirosh