PEP proposal: unifying function/method classes
Dear python-ideas, I would like to draft a PEP to change the implementation of functions and methods in CPython. The idea for this PEP come from a discussion on https://bugs.python.org/issue30071 The core idea of the PEP is to reorganize classes for functions and methods with the goal of removing the difference between functions/methods implemented in C and functions/methods implemented in Python. Currently, CPython has two different function classes: the first is Python functions, which is what you get when defining a function with def or lambda. The second is built-in functions such as len, isinstance or numpy.dot. These are implemented in C. These two classes are completely independent with different functionality. For example, built-in functions do not have __get__ and therefore cannot be used as methods. And only Python functions support introspection like inspect.getargspec or inspect.getsourcefile. This is a problem for projects like Cython that want to implement functions in C but still want to offer functionality close to Python functions. In Cython, this was solved by inventing a new function class called cyfunction. Unfortunately, this new function class creates problems, for example the inspect module does not recognize such functions as being functions (this is the bug mentioned above) and the performance is worse (CPython has specific optimizations for calling built-in functions). When you look at methods, the inconsistency increases: On the one hand, the implementation of built-in functions and bound methods is very similar but built-in unbound methods are different. On the other hand, Python functions and unbound methods are identical but bound methods are different. Do you think that it makes sense to propose a PEP for this? Jeroen.
On 23 March 2018 at 06:34, Jeroen Demeyer <J.Demeyer@ugent.be> wrote:
Reading the discussion on the issue, this feels more to me like PEP 357, which added the __index__ protocol such that arbitrary objects could be used for slicing. In this case, rather than the problem being operations that require a "native Python integer", the issue is with operations which currently require a native Python function object, even though most of the operations that folks implement on them can be successfully ducktyped. As Antoine notes, unifying user-defined functions and builtin functions would be fraught with backwards compatibility problems, so you probably don't want to go down that path when your goal is "Let third parties define types that are recognised by the core interpreter as being user-defined functions". If it was just about introspection, then we could define a new protocol method or attribute, update the inspect module to respect it, and call it done. However, it seems to me that the most plausible path towards granting Cython access to the optimised fast paths for native functions would be setting Py_TPFLAGS_BASETYPE on types.FunctionType, and introducing a new Py_TPFLAGS_FUNCTION_SUBCLASS flag. We can't introduce a new protocol into those paths without slowing them down, but we can replace the current PyFunction_Check() with the typical combination of a PyFunction_Check() that accepts subclasses (by looking for the new subclass flag if the exact check fails) and PyFunction_CheckExact() (which would continue to disallow native function subclasses). The PyFunction_Check() calls in the code base would then need to be reviewed to determine which of them should be replaced with PyFunction_CheckExact(). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 23 March 2018 at 06:34, Jeroen Demeyer <J.Demeyer@ugent.be> wrote:
Reading the discussion on the issue, this feels more to me like PEP 357, which added the __index__ protocol such that arbitrary objects could be used for slicing. In this case, rather than the problem being operations that require a "native Python integer", the issue is with operations which currently require a native Python function object, even though most of the operations that folks implement on them can be successfully ducktyped. As Antoine notes, unifying user-defined functions and builtin functions would be fraught with backwards compatibility problems, so you probably don't want to go down that path when your goal is "Let third parties define types that are recognised by the core interpreter as being user-defined functions". If it was just about introspection, then we could define a new protocol method or attribute, update the inspect module to respect it, and call it done. However, it seems to me that the most plausible path towards granting Cython access to the optimised fast paths for native functions would be setting Py_TPFLAGS_BASETYPE on types.FunctionType, and introducing a new Py_TPFLAGS_FUNCTION_SUBCLASS flag. We can't introduce a new protocol into those paths without slowing them down, but we can replace the current PyFunction_Check() with the typical combination of a PyFunction_Check() that accepts subclasses (by looking for the new subclass flag if the exact check fails) and PyFunction_CheckExact() (which would continue to disallow native function subclasses). The PyFunction_Check() calls in the code base would then need to be reviewed to determine which of them should be replaced with PyFunction_CheckExact(). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (3)
-
Antoine Pitrou -
Jeroen Demeyer -
Nick Coghlan