Hi,
If you got issues with using the Python C API <Python.h> in C++,
please speak up! I'm looking for feedback :-)
Extending Python by writing C++ code is now easy with the pybind11 project:
https://pybind11.readthedocs.io/
It seems like over the last years, building C++ extensions with the
Python C API started to emit more C++ compiler warnings. One
explanation may be that converting macros to static inline functions
(PEP 670) introduce new warnings, even if the old and the new code …
[View More]is
exactly the same. I just discover this issue recently. C and C++
compilers treat static inline functions differently. Macros are
treated at legacy code which cannot be fixed, like system headers or
old C header files, and so many warnings are made quiet. Static inline
functions (defined in header files) are treated as regular code and
compilers are more eager to emit warnings.
I just modified the Python C API to use C++ static_cast<type>(expr)
and reinterpret_cast<type>(expr) if it's used with C++. In C, the
regular (type)expr cast (called "old-style cast" by C++ compilers ;-))
is still used as before.
I'm also working on adding an unit test to make suite that using the
Python C API works with a C++ compiler and doesn't emit compiler
warnings:
* https://github.com/python/cpython/issues/91321
* https://github.com/python/cpython/pull/32175
In terms of C++ version, it was proposed to target C++11.
In the pythoncapi-compat project, I got warnings when the NULL
constant is used in static inline functions. I modified the
pythoncapi_compat.h header file to use nullptr if used with C++ to fix
these compiler warnings. So far, I'm unable to reproduce the issue
with <Python.h>, and so I didn't try to address this issue in Python.
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
[View Less]
There is a special handling of `__hash__` set to None in the interpreter
core. This is because every class inherited the `__hash__` attribute
from "object", and setting `__hash__ = None` is a simple way to make it
unhashable. It makes hash() raising the correct type of exception
(TypeError), but with unhelpful error message "'NoneType' object is not
callable". The special case was added to make the error message more
relevant: "unhashable type: '{typename}'".
There is similar situation …
[View More]with other special methods defined in
"object" or other common classes. Sometimes we want to cancel the
default inherited behavior.
>>> dir(object)
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__',
'__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__',
'__hash__', '__init__', '__init_subclass__', '__le__', '__lt__',
'__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__',
'__setattr__', '__sizeof__', '__str__', '__subclasshook__']
I propose to support officially the idiom "__dunder__ = None" and add
special cases to raise more specialized exception instead of "TypeError:
'NoneType' object is not callable" for most of special method where
cancelling the default behavior makes sense (for example I do not think
that we need better error message for `__repr__ = None`).
The question is how to interpret value None:
* Always raise TypeError (with changed message)? This is what happen
currently when you set the method to None, this is the most compatible
option.
* Always raise an error, but allow to change it to more appropriate type
(for example AttributeError for __setattr__)?
* Interpret value None the same way as an absent attribute?
For `__hash__` or `__class_getitem__` all three options mean the same.
But absent `__mro_entries__` and `__mro_entries__ = None` currently give
different results. It is even more complicated for pickling: absent
`__reduce_ex__` and `__reduce_ex__ = None` mean the same in the Python
implementation, but give different results in the C implementation.
[View Less]
Sorry, folks, but I've been busy the last few days--the Language Summit
is Wednesday, and I had to pack and get myself to SLC for PyCon, &c.
I'll circle back and read the messages on the existing threads
tomorrow. But for now I wanted to post "the wonderful third option" for
forward class definitions we've been batting around for a couple of days.
The fundamental tension in the proposal: we want to /allocate/ the
object at "forward class" time so that everyone can take a …
[View More]reference to
it, but we don't want to /initialize/ the class (e.g. run the class
body) until "continue class" time. However, the class might have a
metaclass with a custom __new__, which would be responsible for
allocating the object, and that isn't run until after the "class body".
How do we allocate the class object early while still supporting custom
metaclass.__new__ calls?
So here's the wonderful third idea. I'm going to change the syntax and
semantics a little, again because we were batting them around quite a
bit, so I'm going to just show you our current thinking.
The general shape of it is the same. First, we have some sort of
forward declaration of the class. I'm going to spell it like this:
forward class C
just for clarity in the discussion. Note that this spelling is also viable:
class C
That is, a "class" statement without parentheses or a colon. (This is
analogous to how C++ does forward declarations of classes, and it was
survivable for them.) Another viable spelling:
C = ForwardClass()
This spelling is nice because it doesn't add new syntax. But maybe it's
less obvious what is going on from a user's perspective.
Whichever spelling we use here, the key idea is that C is bound to a
"ForwardClass" object. A "ForwardClass" object is /not/ a class, it's a
forward declaration of a class. (I suspect ForwardClass is similar to a
typing.ForwardRef, though I've never worked with those so I couldn't say
for sure.) Anyway, all it really has is a name, and the promise that it
might get turned into a class someday. To be explicit about it,
"isinstance(C, type)" is False.
I'm also going to call instances of ForwardClass "immutable". C won't
be immutable forever, but for now you're not permitted to set or change
attributes of C.
Next we have the "continue" class statement. I'm going to spell it like
this:
continue class C(BaseClass, ..., metaclass=MyMetaclass):
# class body goes here
...
I'll mention other possible spellings later. The first change I'll
point out here: we've moved the base classes and the metaclass from the
"forward" statement to the "continue" statement. Technically we could
put them either place if we really cared to. But moving them here seems
better, for reasons you'll see in a minute.
Other than that, this "continue class" statement is similar to what I
(we) proposed before. For example, here C is an expression, not a name.
Now comes the one thing that we might call a "trick". The trick: when
we allocate the ForwardClass instance C, we make it as big as a class
object can ever get. (Mark Shannon assures me this is simply "heap
type", and he knows far more about CPython internals than I ever will.)
Then, when we get to the "continue class" statement, we convince
metaclass.__new__ call to reuse this memory, and preserve the reference
count, but to change the type of the object to "type" (or
what-have-you). C has now been changed from a "ForwardClass" object
into a real type. (Which almost certainly means C is now mutable.)
These semantics let us preserve the entire existing class creation
mechanism. We can call all the same externally-visible steps in the
same externally-visible order. We don't add any new dunder methods, we
don't remove any dunder methods, we don't expose a new dunder attribute
for users to experiment with.
What mechanism do we use to achieve this? metaclass.__new__ always has
to do one of these two things to create the class object: either it
calls "super().__new__", or what we usually call "three-argument type".
In both cases, it passes through the **kwargs that it received into the
super().__new__ call or the three-argument type call. So the "continue
class C" statement will internally add a new kwarg: "__forward__ = C".
If super().__new__ or three-argument type get this kwarg, they won't
allocate a new object, they'll reuse C. They'll preserve the current
reference count, but otherwise overwrite C with all the juicy vitamins
and healthy minerals packed into a Python class object.
So, technically, this means we could spell the "continue class" step
like so:
class C(BaseClass, ..., metaclass=MyMetaClass, __forward__=C):
...
Which means that, combined with the "C = ForwardClass()" statement
above, we could theoretically implement this idea without changing the
syntax of the language. And since we already don't have to change the
underlying semantics of Python class creation, the technical debt
incurred by adding this to the language becomes much smaller.
What could go wrong? My biggest question so far: is there such a thing
as a metaclass written in C, besides type itself? Are there metaclasses
with a __new__ that /doesn't/ call super().__new__ or three-argument
type? If there are are metaclasses that allocate their own class
objects out of raw bytes, they'd likely sidestep this entire process. I
suspect this is rare, if indeed it has ever been done. Anyway, that'd
break this mechanism, so exotic metaclasses like these wouldn't work
with "forward-declared classes". But at least they needn't fail
silently. We just need to add a guard after the call to
metaclass.__new__: if we passed in "__forward__=C" into
metaclass.__new__, and metaclass.__new__ didn't return C, we raise an
exception.
Cheers,
//arry/
p.s. When I say "we" above, I generally mean Eric V. Smith, Barry
Warsaw, Mark Shannon, and myself. But please assume that any dumb ideas
in the proposal are mine, and I was too wrong-headed to listen to the
sage advice from these three wise men when I wrote this email.
[View Less]