Re: [Python-ideas] PEP proposal: unifying function/method classes
Dear Nick Coghlan, First of all, thanks for your insightful comments! On 2018-03-24 09:09, Nick Coghlan wrote:
As Antoine notes, unifying user-defined functions and builtin functions would be fraught with backwards compatibility problems, so you probably don't want to go down that path when your goal is "Let third parties define types that are recognised by the core interpreter as being user-defined functions".
First of all, my post was mainly meant to get an answer to "is this idea PEP-able?". I have never written a PEP and PEP 1 recommends this step. So it's not clear whether you want to say "that potential PEP is a bad idea" or just "go ahead with your PEP but be aware of backwards compatibility issues". It would be good to know what your backwards compatibility worries are. If I have a clearer idea of what could break, it would be easier to see if it's possible to not break that. Anyway, many (not all though!) backwards compatibility issues could be fixed by keeping the current built-in functions as a distinct type (say, old_style_builtin although I wouldn't use that name). My vague plan in the PEP is to introduce a base class "basefunction" and have old_style_builtin and Python functions be subclasses of that. Cython functions would then be another subclass. That way, we could implement "basefunction" properly and then implement old_style_builtin to support things like __get__ not binding as a method.
If it was just about introspection, then we could define a new protocol method or attribute, update the inspect module to respect it, and call it done.
True. In fact, I got a long way by just defining inspect.isfunction as def isfunction(obj): return hasattr(type(obj), "__code__")
However, it seems to me that the most plausible path towards granting Cython access to the optimised fast paths for native functions would be setting Py_TPFLAGS_BASETYPE on types.FunctionType
Why FunctionType and not BuiltinFunctionType? That's really the problem here: I want a function which is as fast as a built-in function but which behaves from a user's point of view as a Python function.
We can't introduce a new protocol into those paths without slowing them down
I don't really follow you here. Are you saying that it's impossible to allow custom function classes that are as fast as the current builtin functions? Instead of adding a flag for a subclass, we could add a flag for "can be called as if it is a built-in function". Jeroen.
On 25 March 2018 at 07:38, Jeroen Demeyer
Dear Nick Coghlan,
First of all, thanks for your insightful comments!
On 2018-03-24 09:09, Nick Coghlan wrote:
As Antoine notes, unifying user-defined functions and builtin functions would be fraught with backwards compatibility problems, so you probably don't want to go down that path when your goal is "Let third parties define types that are recognised by the core interpreter as being user-defined functions".
First of all, my post was mainly meant to get an answer to "is this idea PEP-able?". I have never written a PEP and PEP 1 recommends this step. So it's not clear whether you want to say "that potential PEP is a bad idea" or just "go ahead with your PEP but be aware of backwards compatibility issues".
It would be good to know what your backwards compatibility worries are. If I have a clearer idea of what could break, it would be easier to see if it's possible to not break that.
The biggest problem would have been the potential change to the descriptor behaviour - making CPython's C functions behave like instance methods instead of static methods would create a significant compatibility risk for no clear benefit. However, from the rest of your email, it sounds like you're not actually proposing to change that and I'd just misunderstood what you were suggesting, so I now think it's a promising idea that should help not only Cython, but also other binding generation projects like cffi, SWIG, Boost, etc :) Cheers, Nick. P.S. You're also right that I'd missed the fact that the fast paths that you want to follow at runtime are the PyCFunction ones, so you really do need to be able to define a hybrid type that executes like a builtin C function, but supports introspection like a native Python function. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Jeroen Demeyer schrieb am 24.03.2018 um 22:38:
On 2018-03-24 09:09, Nick Coghlan wrote:
If it was just about introspection, then we could define a new protocol method or attribute, update the inspect module to respect it, and call it done.
True. In fact, I got a long way by just defining inspect.isfunction as
def isfunction(obj): return hasattr(type(obj), "__code__")
This relies on the fact that Cython attaches an (empty) code object to its C implemented functions, which could be considered a hack but is needed for things like source file reporting, and also line reporting (although we don't currently use that, but an even worse hack). The main problem, though, is that a lot of code out there does not use the "inspect" module but "types" to distinguish its input. That probably got a bit better over time with the ABCs, but for things like functions, generators and coroutines, I doubt that people would easily consider them. "types" just seems too obvious at first thought. Which suggests that it could help to add a visible note to the CPython docs of the "types" module that it's often better to use "inspect" or the ABCs. Seems worth a PR. Stefan
On Sun, Mar 25, 2018 at 6:22 PM, Stefan Behnel
Jeroen Demeyer schrieb am 24.03.2018 um 22:38:
On 2018-03-24 09:09, Nick Coghlan wrote:
If it was just about introspection, then we could define a new protocol method or attribute, update the inspect module to respect it, and call it done.
True. In fact, I got a long way by just defining inspect.isfunction as
def isfunction(obj): return hasattr(type(obj), "__code__")
This relies on the fact that Cython attaches an (empty) code object to its C implemented functions, which could be considered a hack but is needed for things like source file reporting, and also line reporting (although we don't currently use that, but an even worse hack).
The main problem, though, is that a lot of code out there does not use the "inspect" module but "types" to distinguish its input. That probably got a bit better over time with the ABCs, but for things like functions, generators and coroutines, I doubt that people would easily consider them. "types" just seems too obvious at first thought.
Which suggests that it could help to add a visible note to the CPython docs of the "types" module that it's often better to use "inspect" or the ABCs. Seems worth a PR.
If people are checking "if type(x) is types.FunctionType", they're doing it wrong. But if the check is "isinstance(x, types.FunctionType)", maybe the solution is to register builtin_function_or_method as a virtual subclass? ChrisA
On Sun, Mar 25, 2018 at 06:52:09PM +1100, Chris Angelico wrote:
If people are checking "if type(x) is types.FunctionType", they're doing it wrong.
That depends on what "it" is that they're doing. If they want a callable, then they're doing it wrong. If they want a function regardless of whether it's written in Python or C or Cython, then they're doing it wrong. But if they *specifically* want a Python function, perhaps so they can pull it apart in ways you can't do to functions written in other languages, then it is fine. -- Steve
On Sun, Mar 25, 2018 at 9:53 PM, Steven D'Aprano
On Sun, Mar 25, 2018 at 06:52:09PM +1100, Chris Angelico wrote:
If people are checking "if type(x) is types.FunctionType", they're doing it wrong.
That depends on what "it" is that they're doing.
If they want a callable, then they're doing it wrong.
If they want a function regardless of whether it's written in Python or C or Cython, then they're doing it wrong.
But if they *specifically* want a Python function, perhaps so they can pull it apart in ways you can't do to functions written in other languages, then it is fine.
My point was that checking if the type of something *is* some type object, that's the wrong way to do things, and that a proper isinstance check has other ways to solve the underlying problem. ChrisA
participants (5)
-
Chris Angelico -
Jeroen Demeyer -
Nick Coghlan -
Stefan Behnel -
Steven D'Aprano