Hi, On 02/04/2019 1:49 pm, Petr Viktorin wrote:
On 3/30/19 11:36 PM, Jeroen Demeyer wrote:
On 2019-03-30 17:30, Mark Shannon wrote:
2. The claim that PEP 580 allows "certain optimizations because other code can make assumptions" is flawed. In general, the caller cannot make assumptions about the callee or vice-versa. Python is a dynamic language.
PEP 580 is meant for extension classes, not Python classes. Extension classes are not dynamic. When you implement tp_call in a given way, the user cannot change it. So if a class implements the C call protocol or the vectorcall protocol, callers can make assumptions about what that means.
PEP 579 is mainly a list of supposed flaws with the 'builtin_function_or_method' class. The general thrust of PEP 579 seems to be that builtin-functions and builtin-methods should be more flexible and extensible than they are. I don't agree. If you want different behaviour, then use a different object. Don't try an cram all this extra behaviour into a pre-existing object.
I think that there is a misunderstanding here. I fully agree with the "use a different object" solution. This isn't a new solution: it's already possible to implement those different objects (Cython does it). It's just that this solution comes at a performance cost and that's what we want to avoid.
It does seem like there is some misunderstanding.
PEP 580 defines a CCall structure, which includes the function pointer, flags, "self" and "parent". Like the current implementation, it has various METH_ flags for various C signatures. When called, the info from CCall is matched up (in relatively complex ways) to what the C function expects.
PEP 590 only adds the "vectorcall". It does away with flags and only has one C signatures, which is designed to fit all the existing ones, and is well optimized. Storing the "self"/"parent", and making sure they're passed to the C function is the responsibility of the callable object. There's an optimization for "self" (offsetting using PY_VECTORCALL_ARGUMENTS_OFFSET), and any supporting info can be provided as part of "self". >
I'll reiterate that PEP 590 is more general than PEP 580 and that once the callable's code has access to the callable object (as both PEPs allow) then anything is possible. You can't can get more extensible than that.
Anything is possible, but if one of the possibilities becomes common and useful, PEP 590 would make it hard to optimize for it. Python has grown many "METH_*" signatures over the years as we found more things that need to be passed to callables. Why would "METH_VECTORCALL" be the last? If it won't (if you think about it as one more way to call functions), then dedicating a tp_* slot to it sounds quite expensive.
I doubt METH_VECTORCALL will be the last. Let me give you an example: It is quite common for a function to take two arguments, so we might want add a METH_OO flag for builtin-functions with 2 parameters. To support this in PEP 590, you would make exactly the same change as you would now; which is to add another case to the switch statement in _PyCFunction_FastCallKeywords. For PEP 580, you would add another case to the switch in PyCCall_FastCall. No difference really. PEP 580 uses a slot as well. It's only 8 bytes per class.
In one of the ways to call C functions in PEP 580, the function gets access to: - the arguments, - "self", the object - the class that the method was found in (which is not necessarily type(self)) I still have to read the details, but when combined with LOAD_METHOD/CALL_METHOD optimization (avoiding creation of a "bound method" object), it seems impossible to do this efficiently with just the callable's code and callable's object.
It is possible, and relatively straightforward. Why do you think it is impossible?
I would argue the opposite: PEP 590 defines a fixed protocol that is not easy to extend. PEP 580 on the other hand uses a new data structure PyCCallDef which could easily be extended in the future (this will intentionally never be part of the stable ABI, so we can do that).
I have also argued before that the generality of PEP 590 is a bad thing rather than a good thing: by defining a more rigid protocol as in PEP 580, more optimizations are possible.
PEP 580 has the same limitation for the same reasons. The limitation is necessary for correctness if an object supports calls via `__call__` and through another calling convention.
I don't think that this limitation is needed in either PEP. As I explained at the top of this email, it can easily be solved by not using the protocol for Python classes. What is wrong with my proposal in PEP 580: https://www.python.org/dev/peps/pep-0580/#inheritance
I'll add Jeroen's notes from the review of the proposed PEP 590 (https://github.com/python/peps/pull/960):
The statement "PEP 580 is specifically targetted at function-like objects, and doesn't support other callables like classes, partial functions, or proxies" is factually false. The motivation for PEP 580 is certainly function/method-like objects but it's a general protocol that every class can implement. For certain classes, it may not be easy or desirable to do that but it's always possible. > Given that `PY_METHOD_DESCRIPTOR` is a flag for tp_flags, shouldn't it be called `Py_TPFLAGS_METHOD_DESCRIPTOR` or something?
Py_TPFLAGS_HAVE_VECTOR_CALL should be Py_TPFLAGS_HAVE_VECTORCALL, to be consistent with tp_vectorcall_offset and other uses of "vectorcall" (not "vector call")
Thanks for the comments, I'll update the PEP when I get the chance.
And mine, so far:
I'm not clear on the constness of the "args" array. If it is mutable (PyObject **), you can't, for example, directly pass a tuple's storage (or any other array that could be used in the call). If it is not (PyObject * const *), you can't insert the "self" argument in. The reference implementations seems to be inconsistent here. What's the intention?
I'll make it clearer in the PEP. My thinking was that if `PY_VECTORCALL_ARGUMENTS_OFFSET` is set then the caller is allowing the callee to mutate element -1. It would make sense to generalise that to any element of the vector (including -1). When passing the contents of a tuple, `PY_VECTORCALL_ARGUMENTS_OFFSET` should not be set, and thus the vector could not be mutated. Cheers, Mark.