PEP 579 and PEP 580: refactoring C functions and methods
Hello, Let me present PEP 579 and PEP 580. PEP 579 is an informational meta-PEP, listing some of the issues with functions/methods implemented in C. The idea is to create several PEPs each fix some part of the issues mentioned in PEP 579. PEP 580 is a standards track PEP to introduce a new "C call" protocol, which is an important part of PEP 579. In the reference implementation (which is work in progress), this protocol will be used by built-in functions and methods. However, it should be used by more classes in the future. You find the texts at https://www.python.org/dev/peps/pep-0579 https://www.python.org/dev/peps/pep-0580 Jeroen.
Hi Jeroen,
On Wed, 20 Jun 2018 10:53:18 +0200
Jeroen Demeyer
PEP 579 is an informational meta-PEP, listing some of the issues with functions/methods implemented in C. The idea is to create several PEPs each fix some part of the issues mentioned in PEP 579.
PEP 580 is a standards track PEP to introduce a new "C call" protocol, which is an important part of PEP 579. In the reference implementation (which is work in progress), this protocol will be used by built-in functions and methods. However, it should be used by more classes in the future.
This is very detailed and analytic. Thanks. I dislike that the protocol is complicated, with many special cases. Ideally there would be two axes for parametrization of C calls: - the signature of the C callee (either fast call or normal call) - whether the callable is called as a function ("foo(...)") or as a method ("some_obj.foo(...)"). But there seems to be some complication on top of that: - PyCCall_FastCall() accepts several types for the keywords, even a dict; does it get forwarded as-is to the `cc_func` or is it first transformed? - there's CCALL_OBJCLASS and CCALL_SLICE_SELF which have, well, non-obvious behaviour (especially the latter), especially as it is conditioned on the value of other fields or flags I wonder if there's a way to push some of the specificities out of the protocol and into the C API that mediates between the protocol and actual callers? Regards Antoine.
https://www.python.org/dev/peps/pep-0580/#the-c-call-protocol
CCALL_VARARGS: cc_func(PyObject *self, PyObject *args)
If we add a new calling convention, I would prefer to reuse the
FASTCALL calling convention to pass arguments: pass a PyObject **args
array with a size (Py_ssize_t nargs) rather than requiring to create a
temporary tuple object to pass positional arguments.
Victor
2018-06-20 10:53 GMT+02:00 Jeroen Demeyer
Hello,
Let me present PEP 579 and PEP 580.
PEP 579 is an informational meta-PEP, listing some of the issues with functions/methods implemented in C. The idea is to create several PEPs each fix some part of the issues mentioned in PEP 579.
PEP 580 is a standards track PEP to introduce a new "C call" protocol, which is an important part of PEP 579. In the reference implementation (which is work in progress), this protocol will be used by built-in functions and methods. However, it should be used by more classes in the future.
You find the texts at https://www.python.org/dev/peps/pep-0579 https://www.python.org/dev/peps/pep-0580
Jeroen. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
How often "custom method type" are used?
I thought Cython use it by default.
But when I read code generated by Cython, I can't find it.
It uses normal PyMethodDef and tp_methods.
I found CyFunction in Cython repository, but I can't find
how to use it. Cython document doesn't explain any information
about it.
When, and how often custom method type is used?
Isn't it very rare? If there are only 0.1% custom method type,
why reducing 30% calling overhead is important for them?
I want more possible target applications to motivate me
for such complex protocols.
--
INADA Naoki
INADA Naoki schrieb am 07.07.2018 um 06:10:
How often "custom method type" are used?
I thought Cython use it by default. But when I read code generated by Cython, I can't find it. It uses normal PyMethodDef and tp_methods.
I found CyFunction in Cython repository, but I can't find how to use it. Cython document doesn't explain any information about it.
Its usage is disabled by default because of some of the problems that Jeroen addresses in his PEP(s). You can enable Cython's own function type by setting the compiler directive "binding=True", e.g. from your setup.py or in a comment at the very top of your source file: # cython: binding=True The directive name "binding" stems from the fact that CyFunctions bind as methods when put into classes, but it's really misleading these days because the main advantage is that it makes Cython compiled functions look and behave much more like Python functions, including introspection etc. Stefan
On Sat, Jul 7, 2018 at 4:35 PM Stefan Behnel
INADA Naoki schrieb am 07.07.2018 um 06:10:
How often "custom method type" are used?
I thought Cython use it by default. But when I read code generated by Cython, I can't find it. It uses normal PyMethodDef and tp_methods.
I found CyFunction in Cython repository, but I can't find how to use it. Cython document doesn't explain any information about it.
Its usage is disabled by default because of some of the problems that Jeroen addresses in his PEP(s).
You can enable Cython's own function type by setting the compiler directive "binding=True", e.g. from your setup.py or in a comment at the very top of your source file:
# cython: binding=True
The directive name "binding" stems from the fact that CyFunctions bind as methods when put into classes, but it's really misleading these days because the main advantage is that it makes Cython compiled functions look and behave much more like Python functions, including introspection etc.
Stefan
Thank you. Do you plan to make it default when PEP 580 is accepted
and implemented?
Personally speaking, I used Cython for quick & easy alternative way to
writing extension types.
I don't need compatibility with pure Python functions. I prefer
minimum and lightweight.
So I will disable it explicitly or stop using Cython.
But if you believe PEP 580 makes many Cython users happy, I believe you.
--
INADA Naoki
INADA Naoki schrieb am 07.07.2018 um 10:08:
Thank you. Do you plan to make it default when PEP 580 is accepted and implemented?
It will become the default at some point, I'm sure. Note that we will still have to support older Python versions, though, currently 2.6+, which would not have the improvements available. Some things might be backportable for us, at least to older Py3.x releases, but we'll see.
Personally speaking, I used Cython for quick & easy alternative way to writing extension types. I don't need compatibility with pure Python functions. I prefer minimum and lightweight. So I will disable it explicitly or stop using Cython.
I'll try to keep the directive available as a compatibility switch for you. ;)
But if you believe PEP 580 makes many Cython users happy, I believe you.
It's more of a transitive thing, for the users of your code. If the functions and methods in the code that I write behave like Python functions, then people who use my code will not run into surprises and quirks when trying to do their stuff with them and things will generally "just work", e.g. inspecting the functions when debugging or using them interactively, looking up their signature, default arguments and annotations, generating documentation from them, pickling, assigning them as methods ... basically anything that people implicitly expect to be able to do with Python functions (or even callables in general) and that doesn't (not well or not at all) work with PyCFunction. Stefan
On 06/20/18 01:53, Jeroen Demeyer wrote:
Hello,
Let me present PEP 579 and PEP 580.
PEP 579 is an informational meta-PEP, listing some of the issues with functions/methods implemented in C. The idea is to create several PEPs each fix some part of the issues mentioned in PEP 579.
PEP 580 is a standards track PEP to introduce a new "C call" protocol, which is an important part of PEP 579. In the reference implementation (which is work in progress), this protocol will be used by built-in functions and methods. However, it should be used by more classes in the future.
You find the texts at https://www.python.org/dev/peps/pep-0579 https://www.python.org/dev/peps/pep-0580
Hi! I finally had time to read the PEPs carefully. Overall, great work! PEP 580 does look complicated, but it's well thought out and addresses real problems. I think the main advantage over the competing PEP 576 is that it's a better foundation for solving Cython (and other C-API users) and my PEP 573 (module state access from methods). With that, I do have some comments. The reference to PEP 573 is premature. If PEP 580 is implemented then PEP 573 will build on top, and I don't plan to update PEP 573 before that. So, I think 580 should be independent. If you agree I can summarize rationale for "parent", as much as it concerns 580. # Using tp_print The tp_print gimmick is my biggest worry. AFAIK there's no guarantee that a function pointer and Py_ssize_t are the same size. That makes the backwards-compatibility typedef in the implementation is quite worrying: typedef Py_ssize_t printfunc I can see the benefit for backporting to earlier Python versions, and maybe that outweighs worries about exotic architectures, but the PEP should at least have more words on why this is not a problem. # The C Call protocol I really like the fact that, in the reference implementation, the flags are arranged in a way that allows a switch statement to select what to call. That should be noted, if only to explain why there's no guarantee of compatibility between Python versions. # Descriptor behavior I'd say "SHOULD" rather than "MUST" here. The section describes how to implement expected/reasonable behavior, but I see no need to limit that. "if func supports the C call protocol, then func.__set__ must not be implemented." -- also, __delete__ should not be implemented, right?. # Generic API functions I'm a bit worried about PyCCall_FASTCALL's "kwds" argument accepting a dict, which is mutable. I wouldn't mind dropping that capability, but if it stays, we need to require that the callable promises to not modify it. PyCCall_FASTCALL is not a macro, shouldn't it be named PyCCall_FastCall? # C API functions The function PyCFunction_GetFlags is, for better or worse, part of the stable ABI. We shouldn't just give up on it. I'm fine with documenting that it shouldn't be used, but for functions defined using PyCFunction_New etc. it should continue behaving as before. One solution could be to preserve the "definition time" METH_* flags in the 0xFFF bits of cc_flags and use the other bits for CCALL_*. # Stable ABI The section should repeat that PyCFunction_ClsNew is added to the stable ABI (but nothing else).
participants (6)
-
Antoine Pitrou -
INADA Naoki -
Jeroen Demeyer -
Petr Viktorin -
Stefan Behnel -
Victor Stinner