On 10/27/2018 2:53 PM, Joy Diamond wrote:

Chris,

Yes, the following works: """ (Note that any object `x` is always considered to be an instance of `type(x)`, and this cannot be overridden.) """

Open a doc issue on bugs.python.org with the problem, motivation, and proposed solution and it should get considered.

NOTE:  I fixed your sentence of `x.__class__` to `type(x)` since it is not always true that `x.__class__ == type(x)`.

For example the actual code reference above:

https://github.com/python/cpython/blob/master/Objects/abstract.c#L2397-L2405

Says "if (Py_TYPE(inst) == (PyTypeObject *)cls)" in the actual C Python implementation:

So it using `type(x)` not `x.__class__`

Thanks,

Joy Diamond.

====

ADDENDUM:

Here is an example where `type(x) is not x.__class__`   (there are other not as perverse examples where you want `.__class__` to differ from `type`)

NOTE:  The final assert will fail, showing that `type(x) is not x.__class__`

# #   This really perverse code demonstrates that `t.__class__` is *NOT* #   always the same as type(t). # #  The code shows an metaclass that when you derive objects from its classes, creates a `7` #  instead of a derived class. # def not_really_a_metaclass__make_a_7(name, bases, member):     return 7

@property def not_really_a_class__make_a_7(t):     return not_really_a_metaclass__make_a_7

class Metaclass__Make_A_7(type):     __class__ = not_really_a_class__make_a_7

Make_A_7 = Metaclass__Make_A_7('Make_A_7', ((object,)), {})

# #   Now then: # #       `Make_A_7` is a class that when inherited from creates a '7' instead #       of a class ... :( # #   Works for python 2 & pypy (not yet fixed to work for python 3) # class Seven(Make_A_7):  # Calls `Make_A_7.__class__('Seven, (('Make_A_7,)), {})` which returns `7`     pass

print('Seven is: %s' % Seven)

assert isinstance(Make_A_7, Metaclass__Make_A_7) assert Make_A_7.__class__ == Metaclass__Make_A_7   #  This will *FAIL* due to the perverse definition of `.__class__`

On Sat, Oct 27, 2018 at 2:25 PM Chris Angelico <rosuav@gmail.com <mailto:rosuav@gmail.com>> wrote:

On Sun, Oct 28, 2018 at 5:03 AM Joy Diamond <python.gem@gmail.com <mailto:python.gem@gmail.com>> wrote: > > Greetings, > > This is a request to fix the documentation for __instancecheck__. > > Please add the following (please rewrite better than I can -- I am not good at explaining concepts in short sentences): > > NOTE:  As an optimization, isinstance(object, classinfo) does NOT call classinfo.__instancecheck__(instance) when type(object) == classinfo. > > Consider the following program: > > class M(type): >     def __instancecheck__(m, t): >         print('instancecheck(%s, %s)' % (m, t)) >         return False                                    #   LIE! > > Test = M('Test', ((object,)), {}) > > something = Test() > > print('Does *NOT* call __instancecheck__:') > print('isinstance(something, Test): %s' % isinstance(something, Test))

Here's the passage in question, for reference:

""" The following methods are used to override the default behavior of the isinstance() and issubclass() built-in functions.

In particular, the metaclass abc.ABCMeta implements these methods in order to allow the addition of Abstract Base Classes (ABCs) as “virtual base classes” to any class or type (including built-in types), including other ABCs. """ https://docs.python.org/3/reference/datamodel.html#customizing-instance-and-...

Since it uses the word "override", I agree that it's not entirely correct. The implication of "override" is that you can completely replace the normal behaviour. In this case, you can change the behaviour of subclass testing (for instance, you can "disown" a subclass by denying that instances of it are instances of yourself), and of course, you can claim an object as an instance of a class it didn't directly inherit from (the way ABCs work), but you cannot fib about direct instances. I think the behaviour is close enough to accurate that it doesn't need major rewording; how about adding this parenthesis:

""" (Note that any object `x` is always considered to be an instance of `x.__class__`, and this cannot be overridden.) """

Would that work?

ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org <mailto:Python-ideas@python.org> https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Terry Jan Reedy

On Sun, Oct 28, 2018 at 05:24:43AM +1100, Chris Angelico wrote:

On Sun, Oct 28, 2018 at 5:03 AM Joy Diamond <python.gem@gmail.com> wrote:

...

NOTE: As an optimization, isinstance(object, classinfo) does NOT call classinfo.__instancecheck__(instance) when type(object) == classinfo.

I'd like to discuss this optimization. It seems very strange to me that a method designed specifically to override the isinstance check isn't actually called to allow it to override the isinstance check. [Chris]

Here's the passage in question, for reference:

""" The following methods are used to override the default behavior of the isinstance() and issubclass() built-in functions.

In particular, the metaclass abc.ABCMeta implements these methods in order to allow the addition of Abstract Base Classes (ABCs) as “virtual base classes” to any class or type (including built-in types), including other ABCs. """ https://docs.python.org/3/reference/datamodel.html#customizing-instance-and-...

Since it uses the word "override", I agree that it's not entirely correct.

Is that a polite way of saying "wrong"? The question we should be asking, is the optimization implementing the desired behaviour: * classes can disown instances of subclasses * but they cannot disown their own instances or is the optimization over-zealous and does too much? I don't think it is obvious that the behaviour is correct. Presumably Joy had a use-case for overriding isinstance(), and this optimization prevented it. Joy, can you comment on your use-case, and did you come up with a work-around?

The implication of "override" is that you can completely replace the normal behaviour.

Indeed.

In this case, you can change the behaviour of subclass testing (for instance, you can "disown" a subclass by denying that instances of it are instances of yourself), and of course, you can claim an object as an instance of a class it didn't directly inherit from (the way ABCs work), but you cannot fib about direct instances.

But we can change the class of direct instances, by changing their __class__ attribute, and that is intentional, supported behaviour. So "fibbing" is allowed. Perhaps changing the __class__ is enough to work-around this optimization, and there's nothing to do here except to document it better. But I think that if it is useful to own a non-instance, we shouldn't be so blasé about prohibiting disowning an instance.

I think the behaviour is close enough to accurate that it doesn't need major rewording; how about adding this parenthesis:

""" (Note that any object `x` is always considered to be an instance of `x.__class__`, and this cannot be overridden.) """

I would rather be precise about what is going on, and state that X.__instancecheck__(x) is not called if type(x) is X, rather than merely imply it. It is not just that the method is called and ignored, but that it isn't called at all. I find the process of checking types rather opaque and mysterious. Do we have a documented (other than the source) algorithm for deciding what is an instance of what? - type(x) and x.__class__ don't necessarily agree; under what circumstances are each used? (I've asked this before, and either never got a good answer, or I can't keep it straight in my head.) - what precisely does type(x) do? - when is __instancecheck__ called? A flowchart would be good :-) -- Steve

On Sat, Oct 27, 2018 at 8:02 PM Steven D'Aprano <steve@pearwood.info> wrote:

I don't think it is obvious that the behaviour is correct. Presumably Joy had a use-case for overriding isinstance(), and this optimization prevented it.

Joy, can you comment on your use-case, and did you come up with a work-around?

I'm implementing traits (basically interfaces + members). I am thus replacing the whole object/class/inheritance mechanism [Also replacing __slots__ with actual members] Thus, what I am doing is replacing all of: __cmp__, __eq__, __ge__ ,__gt__, __le__, __lt__, __ne__ (you cannot compare objects, unless they inherit from interface 'Comparable', or 'Hashable'). __delattr__ (thus my classes properly implement 'immutable', to make immutable members). __format__ (you cannot, currently, format objects) __init__ (you cannot write a constructor; instead, it automatically creates a hidden constructor, like C++. You cannot write a constructor, as assignment to members often fails, since they are immutable & __delattr__ & __setattr_ have been fixed to enforce that -- in debug mode) __hash__ (You cannot hash an object, unless it inherits from interface 'Hashable', in which case you can). __new__ (you cannot write a construtor -- see above under __init__) __reduce__, and __reduce__ex__ (you, currently, cannot pickle or unpickle objects -- this has to do with immutable members; and will be fixed if pickling is required). __setattr__ (thus my classes properly implement 'immutable', to make immutable members) __subclasshook__ (classes do not use inheritance, instead they are all trait based) And also, for metaclasses: __call__ (calls an automatically created constructor to construct the object, and work around the fact that some of its members are immutale). __instancecheck__ (classes do not use inheritance, instead they are all trait based) __subclasscheck__ (classes do not use inheritance, instead they are all trait based) __subclasses__ (classes do not use inheritance, instead they are all trait based). My particular use case was: I simply wanted to disable __instancecheck__, which I did, but my unit test code, failed, when the call to __instancecheck__ was bypassed.

- type(x) and x.__class__ don't necessarily agree; under what circumstances are each used?

(I've asked this before, and either never got a good answer, or I can't keep it straight in my head.)

- what precisely does type(x) do?

1. `type(x)` gets the true actual type of `x`. 2. `x.__class__` gets the `.__class__` attribute for `x`, which by default gets the actual true type of `x`, but may be replace by the user to do other stuff. In pythonic terms, `type(x)` does the following: 1. It looks like a constructor to `type` -- Hence it calls type's metaclass `__call__` function. 2. To be precise it calls: `type.__class__.__call__(type, x)` 3. This code can be seen here: https://github.com/python/cpython/blob/master/Objects/typeobject.c#L914-L964 4. Which basically calls `type.__new__` and if `type.__new__` returns a type, calls `type.__init__` (i.e.: the usual way an object is constructed). 5. `type.__new__` is special, it actually returns "the true actual type of `x`". 6. The code for `type.__new__` is here: https://github.com/python/cpython/blob/master/Objects/typeobject.c#L2354-L23... 7. As the code reads: /* Special case: type(x) should return x->ob_type */ /* We only want type itself to accept the one-argument form (#27157) Note: We don't call PyType_CheckExact as that also allows subclasses */ if (metatype == &PyType_Type) { const Py_ssize_t nargs = PyTuple_GET_SIZE(args); const Py_ssize_t nkwds = kwds == NULL ? 0 : PyDict_GET_SIZE(kwds); if (nargs == 1 && nkwds == 0) { PyObject *x = PyTuple_GET_ITEM(args, 0); Py_INCREF(Py_TYPE(x)); return (PyObject *) Py_TYPE(x); } 8. So after going through `type.__call__`, `type.__new__` returns the true actual type of `x` (why this is not optimized in `type.__call__` I've never understood). 9. Thus `type(x)` actually looks like: A CONSTRUCTION of a type object; which is short-circuited to return the previous actual type of `x`. Thus, in many ways, I find `type(x)` to be usage, as I am sure many others do, since it looks like a CONSTRUCtON of a type object, even though it is not. In pythonic terms, `x.__class__` does the following: 1. Call `x.__getattribute__('__class__')`. In general this will find `Object.__dict__["__class__]"` (unless the user is playing games). 2. The default implementation (if not replaced by the user) is here: https://github.com/python/cpython/blob/master/Objects/typeobject.c#L4017-L40... 3. Which calls `object_get_class` as defined here: https://github.com/python/cpython/blob/master/Objects/typeobject.c#L3831-L38... static PyObject * object_get_class(PyObject *self, void *closure) { Py_INCREF(Py_TYPE(self)); return (PyObject *)(Py_TYPE(self)); } 4. Thus the default implementation of `.__class__` is to return the true type of `x`. In terms of actual usage: 1. It is cleaner, to use `.__class__` 2. It is better to use `.__class__`, and the user can replace it, in rare circumstances. 3. *SOME* internal python code attempts to use `.__class__` (and if this fails; then does a fall back to the true type of `x`) 4. *SOME* internal ptyhon code, bypasses `.__class__` and uses the true type of `x` directly [as for example the implementation of `isinstance` does] 5. Use of `type(x)` does not looks as good [since it looks like a construtor]; but is always accurate, and returns the true type of `x`, and cannot be replaced by the user. Hopefully this helps :)

On Sat, Oct 27, 2018 at 10:00 PM Chris Angelico <rosuav@gmail.com> wrote:

On Sun, Oct 28, 2018 at 12:53 PM Joy Diamond <python.gem@gmail.com> wrote:

...

...

- type(x) and x.__class__ don't necessarily agree; under what circumstances are each used?

(I've asked this before, and either never got a good answer, or I can't keep it straight in my head.)

- what precisely does type(x) do?

1. `type(x)` gets the true actual type of `x`. 2. `x.__class__` gets the `.__class__` attribute for `x`, which by default gets the actual true type of `x`, but may be replace by the user to do other stuff.

Not that simple.

...

...

...

class X: ... cls = "X" ... class Y: ... cls = "Y" ... x = X() x.__class__ = Y x.cls 'Y' type(x) <class '__main__.Y'>

I don't know what the "true actual type" is, since I just changed it. In this case, type() and __class__ do exactly the same thing. The only way I know of (in Py3) to have them show different values is to make __class__ a property, and if you do that, I have no idea what uses the property and what uses type().

Maybe this needs to be actually documented somewhere. It keeps on being a point of confusion.

ChrisA

Yes, in your example, you actually changed the true actual type of `x` from an `X` to a `Y`. This is permitted since `X` and `Y` are "compatible". For example, if instead you did [making `X` and `Y` no longer compatible]: class X(object): __slots__ = (('x',)) class Y(object): __slots__ = (('y', 'z')) x = X() x.__class__ = Y You get the following: TypeError: __class__ assignment: 'X' object layout differs from 'Y' Thus assigning to `.__class__` (when it has not been replaced by the user), actually transforms an instance to a new true type. [This is actually really useful in some rare edge cases, which is why Python supports it]. Correct, you can make `__class__` a property to replace it (or play some really difficult games with metaclasses to support a different `__class__`). And, yes it is a point of confusion: 1. As per my earlier email, its best to use `.__class__`, as this is the new way of doing things. 2. `type(x)` was the old [Python 1] way of doing things, looks incorrectly like a constructor. 3. It would take weeks to fully document it; and there are many subtle bugs probably in the python source code as to when it uses `.__class__` and when it uses the true type -- i.e.: bypasses it all by using PY_Type(x). CLARIFICATION: By "True actual type" -- I mean it's actual type as implemented in the C code, and returns by the C [macro] `Py_Type(x)`. That is, as defined at https://github.com/python/cpython/blob/master/Include/object.h#L121 #define Py_TYPE(ob) (((PyObject*)(ob))->ob_type) There are hundreds (704 in Python 2.7.12 for example) references to `Py_TYPE` in the C code; thus it would take weeks to document it all.