[Python-Dev] PEP 580/590 discussion

[Petr Viktorin]

On 4/25/19 10:42 AM, Jeroen Demeyer wrote:
> On 2019-04-25 00:24, Petr Viktorin wrote:
>> I believe we can achieve
>> that by having PEP 590's (o+offset) point not just to function pointer,
>> but to a {function pointer; flags} struct with flags defined for two
>> optimizations:
> 
> What's the rationale for putting the flags in the instance? Do you 
> expect flags to be different between one instance and another instance 
> of the same class?

I'm not tied to that idea. If there's a more reasonable place to put the 
flags, let's go for it, but it's not a big enough issue so it shouldn't 
complicate the protocol much. Quoting Mark from the other subthread:
> Callables are either large or transient. If large, then the extra few bytes makes little difference. If transient then, it matters even less.


>> Both type flags and
>> nargs bits are very limited resources.
> 
> Type flags are only a limited resource if you think that all flags ever 
> added to a type must be put into tp_flags. There is nothing wrong with 
> adding new fields tp_extraflags or tp_vectorcall_flags to a type.

Indeed. Extra flags are just what I think PEP 590 is missing.

>> What I don't like about it is that it has
>> the extensions built-in; mandatory for all callers/callees.
> 
> I don't agree with the above sentence about PEP 580:
> - callers should use APIs like PyCCall_FastCall() and shouldn't need to 
> worry about the implementation details at all.
> - callees can opt out of all the extensions by not setting any special 
> flags and setting cr_self to a non-NULL value. When using the flags 
> CCALL_FASTCALL | CCALL_KEYWORDS, then implementing the callee is exactly 
> the same as PEP 590.

Imagine an extension author sitting down to read the docs and implement 
a callable:

- PEP 580 introduces 6 CCALL_* combinations: you need to select the best 
one for your use case. Also, add two structs to the instance & link them 
via pointers, make sure you support descriptor behavior and the __name__ 
attribute. (Plus there are features for special purposes: CCALL_DEFARG, 
CCALL_OBJCLASS, self-slicing, but you can skip that initially.)
- My proposal: to the instance, add a function pointer with known 
signature and flags which you set to zero. Add an offset to the type, 
and set a type flag. (There are additional possible optimizations, but 
you can skip them initially.)

PEP 580 makes a lot of sense if you read it all, but I fear there'll be 
very few people who read and understand it.
And is not important just for extension authors (admittedly, 
implementing a callable directly using the C API is often a bad idea). 
The more people understand the mechanism, the more people can help with 
further improvements.


I don't see the benefit of supporting METH_VARARGS, METH_NOARGS, and 
METH_O calling conventions (beyond backwards compatibility and 
comptibility with Python's *args syntax).
For keywords, I see a benefit in supporting *only one* of kwarg dict or 
kwarg tuple: if the caller and callee don't agree on which one to use, 
you need an expensive conversion. If we say tuple is the way, some of 
them will need to adapt, but within the set of those that do it any 
caller/callee combination will be fast. (And if tuple only turns out to 
be wrong choice, adding dict support in the future shouldn't be hard.)

That leaves fastcall (with tuple only) as the focus of this PEP, and the 
other calling conventions essentially as implementation details of 
builtin functions/methods.


>> As in PEP 590, any class that uses this mechanism shall not be usable as
>> a base class.
> 
> Can we please lift this restriction? There is really no reason for it. 
> I'm not aware of any similar restriction anywhere in CPython. Note that 
> allowing subclassing is not the same as inheriting the protocol.

Sure, let's use PEP 580 treatment of inheritance.
Even if we don't, I don't think dropping this restriction would be a 
PEP-level change. It can be dropped as soon as an implementation and 
tests are ready, and inheritance issues ironed out. But it doesn't need 
to be in the initial implementation.

> As a compromise, we could simply never inherit the protocol.

That also sounds reasonable for the initial implementation.