Allow user extensions to operators [related to: Moving PEP 584 forward (dict + and += operators)]
What if there was a general mechanism to allow operators to be implemented by user code that does not belong to the class? If the name [e.g.] __operatorhook_or__ is defined anywhere in a module, within that module all calls to that operator are replaced with a two-step process: call __operatorhook_or__(dict1, dict2) if it returns a value, use that value if it returns NotImplemented use the ordinary operator lookup process, i.e. dict1.__or__ and dict2.__ror___ [if __operatorhook_or__ is not defined, treat the same as if it had returned NotImplemented] Then a user could simply do: def __operatorhook_or__(obj1, obj2): if isinstance(obj1, dict) and isinstance(obj2, dict): return {**obj1, **obj2} return NotImplemented def __operatorhook_ior__(obj1, obj2): if isinstance(obj1, dict): obj1.update(obj2) return obj1 return NotImplemented
What if there was a general mechanism to allow operators to be implemented by user code that does not belong to the class?
If the name [e.g.] __operatorhook_or__ is defined anywhere in a module, within that module all calls to that operator are replaced with a two-step process:
* within that module * So this would not happen in other modules when that module was imported...? But would this mean that if I wanted the operation be effective for all modules, I would either have to define it in every module I have written, or import the function by name? E.g.: # mod1 def __operatorhook_or__(obj1, obj2): ... # mod2 from mod1 import __operatorhook_or__
On Tue, Dec 3, 2019, at 11:36, Ricky Teachey wrote:
What if there was a general mechanism to allow operators to be implemented by user code that does not belong to the class?
If the name [e.g.] __operatorhook_or__ is defined anywhere in a module, within that module all calls to that operator are replaced with a two-step process:
* within that module *
So this would not happen in other modules when that module was imported...? But would this mean that if I wanted the operation be effective for all modules, I would either have to define it in every module I have written, or import the function by name? E.g.:
I figured this was the lesser evil, vs two different modules that want to define an operator differently interfering with each other, or a badly written one interfering with a stdlib module... it also allows modules that don't use the facility to not have to pay for it, since overriding an operator like __getattr__ or __call__ in this way would have an enormous performance cost. This also allows you to get the original operator if you need it by importing from the 'operator' module.
# mod1 def __operatorhook_or__(obj1, obj2): ...
# mod2 from mod1 import __operatorhook_or__
On Tue, Dec 3, 2019 at 8:29 AM Random832 <random832@fastmail.com> wrote:
What if there was a general mechanism to allow operators to be implemented by user code that does not belong to the class?
If the name [e.g.] __operatorhook_or__ is defined anywhere in a module, within that module all calls to that operator are replaced with a two-step process:
call __operatorhook_or__(dict1, dict2) if it returns a value, use that value if it returns NotImplemented use the ordinary operator lookup process, i.e. dict1.__or__ and dict2.__ror___ [if __operatorhook_or__ is not defined, treat the same as if it had returned NotImplemented]
Then a user could simply do:
def __operatorhook_or__(obj1, obj2): if isinstance(obj1, dict) and isinstance(obj2, dict): return {**obj1, **obj2} return NotImplemented
def __operatorhook_ior__(obj1, obj2): if isinstance(obj1, dict): obj1.update(obj2) return obj1 return NotImplemented
-1 from me. I can see someone not realizing an operator was changed, assuming it's standard semantics, and then having things break subtly. And debugging this wouldn't be fun either. To me this is monkeypatching without an explicit need for it, i.e. if you really want different semantics in your module then define a function and use that instead of influence-at-a-distance overriding of syntax.
On Tue, Dec 3, 2019, at 13:43, Brett Cannon wrote:
-1 from me. I can see someone not realizing an operator was changed, assuming it's standard semantics, and then having things break subtly. And debugging this wouldn't be fun either. To me this is monkeypatching without an explicit need for it, i.e. if you really want different semantics in your module then define a function and use that instead of influence-at-a-distance overriding of syntax.
Does it make a difference that it'd only apply to code that is physically in the same module where the function is defined? I'd originally planned to suggest full lexical scope for the lookup, in fact, so you could in theory do it within a single function.
On Tue, Dec 3, 2019 at 5:58 PM Random832 <random832@fastmail.com> wrote:
On Tue, Dec 3, 2019, at 13:43, Brett Cannon wrote:
-1 from me. I can see someone not realizing an operator was changed, assuming it's standard semantics, and then having things break subtly. And debugging this wouldn't be fun either. To me this is monkeypatching without an explicit need for it, i.e. if you really want different semantics in your module then define a function and use that instead of influence-at-a-distance overriding of syntax.
Does it make a difference that it'd only apply to code that is physically in the same module where the function is defined? I'd originally planned to suggest full lexical scope for the lookup, in fact, so you could in theory do it within a single function.
Not enough to change my opinion. Changing how fundamental operators work just in a module is still influencing too far from the code site. For instance, if I navigate in my editor directly into the middle of a file or open a file and immediately start searching for the function I care about I won't notice that "+" no longer means what I thought it meant for integers because someone thought it would be smart to redefine that. And as Serhiy pointed out, performance is going to get slammed by this, no opcode or not as you just introduced a new lookup on every syntactic operation.
We could treat it as a kind of future statement. If there’s a top level statement that defines the magic identitier we generate the special bytecode. On Wed, Dec 4, 2019 at 12:26 Brett Cannon <brett@python.org> wrote:
On Tue, Dec 3, 2019 at 5:58 PM Random832 <random832@fastmail.com> wrote:
On Tue, Dec 3, 2019, at 13:43, Brett Cannon wrote:
-1 from me. I can see someone not realizing an operator was changed, assuming it's standard semantics, and then having things break subtly. And debugging this wouldn't be fun either. To me this is monkeypatching without an explicit need for it, i.e. if you really want different semantics in your module then define a function and use that instead of influence-at-a-distance overriding of syntax.
Does it make a difference that it'd only apply to code that is physically in the same module where the function is defined? I'd originally planned to suggest full lexical scope for the lookup, in fact, so you could in theory do it within a single function.
Not enough to change my opinion. Changing how fundamental operators work just in a module is still influencing too far from the code site. For instance, if I navigate in my editor directly into the middle of a file or open a file and immediately start searching for the function I care about I won't notice that "+" no longer means what I thought it meant for integers because someone thought it would be smart to redefine that.
And as Serhiy pointed out, performance is going to get slammed by this, no opcode or not as you just introduced a new lookup on every syntactic operation. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DXZO7U... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
On Wed, Dec 4, 2019 at 3:12 PM Guido van Rossum <guido@python.org> wrote:
We could treat it as a kind of future statement. If there’s a top level statement that defines the magic identitier we generate the special bytecode.
True, that would help solve the performance issue. But I'm still -1 on the idea regardless of the performance. :)
On Wed, Dec 4, 2019 at 12:26 Brett Cannon <brett@python.org> wrote:
On Tue, Dec 3, 2019 at 5:58 PM Random832 <random832@fastmail.com> wrote:
On Tue, Dec 3, 2019, at 13:43, Brett Cannon wrote:
-1 from me. I can see someone not realizing an operator was changed, assuming it's standard semantics, and then having things break subtly. And debugging this wouldn't be fun either. To me this is monkeypatching without an explicit need for it, i.e. if you really want different semantics in your module then define a function and use that instead of influence-at-a-distance overriding of syntax.
Does it make a difference that it'd only apply to code that is physically in the same module where the function is defined? I'd originally planned to suggest full lexical scope for the lookup, in fact, so you could in theory do it within a single function.
Not enough to change my opinion. Changing how fundamental operators work just in a module is still influencing too far from the code site. For instance, if I navigate in my editor directly into the middle of a file or open a file and immediately start searching for the function I care about I won't notice that "+" no longer means what I thought it meant for integers because someone thought it would be smart to redefine that.
And as Serhiy pointed out, performance is going to get slammed by this, no opcode or not as you just introduced a new lookup on every syntactic operation. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DXZO7U... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
03.12.19 20:43, Brett Cannon пише:
-1 from me. I can see someone not realizing an operator was changed, assuming it's standard semantics, and then having things break subtly. And debugging this wouldn't be fun either. To me this is monkeypatching without an explicit need for it, i.e. if you really want different semantics in your module then define a function and use that instead of influence-at-a-distance overriding of syntax.
This will also add a significant performance penalty to any code (even if you do not use any hooks). -1 from me too.
On Wed, Dec 4, 2019, at 08:26, Serhiy Storchaka wrote:
03.12.19 20:43, Brett Cannon пише:
-1 from me. I can see someone not realizing an operator was changed, assuming it's standard semantics, and then having things break subtly. And debugging this wouldn't be fun either. To me this is monkeypatching without an explicit need for it, i.e. if you really want different semantics in your module then define a function and use that instead of influence-at-a-distance overriding of syntax.
This will also add a significant performance penalty to any code (even if you do not use any hooks). -1 from me too.
My proposal was that any module that never uses any hooks compiles to the exact same bytecode, and executes exactly the same way, as it does now. Only modules where the name of the hook is defined get a different operation that looks for the hook. I specifically wrote it that way to *avoid* any performance penalty for modules that do not use any hooks, or for operators other than the one a hook is defined for. (There may be some confusion because of the 'if the hook is undefined' clause in the proposal: I intended that to apply if the hook is deleted or conditionally assigned - if it's never assigned at all, the code won't look for it when the operator is used.)
On 12/04/2019 08:39 AM, Random832 wrote:
On Wed, Dec 4, 2019, at 08:26, Serhiy Storchaka wrote:
03.12.19 20:43, Brett Cannon пише:
-1 from me. I can see someone not realizing an operator was changed, assuming it's standard semantics, and then having things break subtly. And debugging this wouldn't be fun either. To me this is monkeypatching without an explicit need for it, i.e. if you really want different semantics in your module then define a function and use that instead of influence-at-a-distance overriding of syntax.
This will also add a significant performance penalty to any code (even if you do not use any hooks). -1 from me too.
My proposal was that any module that never uses any hooks compiles to the exact same bytecode, and executes exactly the same way, as it does now.
Are the low-level details, such as following the mro, in the bytecode? If not, the bytecode would be the same either way. -- ~Ethan~
On Wed, Dec 4, 2019, at 11:49, Ethan Furman wrote:
On 12/04/2019 08:39 AM, Random832 wrote:
On Wed, Dec 4, 2019, at 08:26, Serhiy Storchaka wrote:
03.12.19 20:43, Brett Cannon пише:
-1 from me. I can see someone not realizing an operator was changed, assuming it's standard semantics, and then having things break subtly. And debugging this wouldn't be fun either. To me this is monkeypatching without an explicit need for it, i.e. if you really want different semantics in your module then define a function and use that instead of influence-at-a-distance overriding of syntax.
This will also add a significant performance penalty to any code (even if you do not use any hooks). -1 from me too.
My proposal was that any module that never uses any hooks compiles to the exact same bytecode, and executes exactly the same way, as it does now.
Are the low-level details, such as following the mro, in the bytecode? If not, the bytecode would be the same either way.
No, but my proposal was for a new bytecode operation that is "check for hook and execute operator", with the explicit goal of avoiding both the performance penalty (especially on frequent operations like getattr and call) and the influence-at-a-distance concern.
On Thu, Dec 5, 2019 at 3:41 AM Random832 <random832@fastmail.com> wrote:
On Wed, Dec 4, 2019, at 08:26, Serhiy Storchaka wrote:
03.12.19 20:43, Brett Cannon пише:
-1 from me. I can see someone not realizing an operator was changed, assuming it's standard semantics, and then having things break subtly. And debugging this wouldn't be fun either. To me this is monkeypatching without an explicit need for it, i.e. if you really want different semantics in your module then define a function and use that instead of influence-at-a-distance overriding of syntax.
This will also add a significant performance penalty to any code (even if you do not use any hooks). -1 from me too.
My proposal was that any module that never uses any hooks compiles to the exact same bytecode, and executes exactly the same way, as it does now. Only modules where the name of the hook is defined get a different operation that looks for the hook. I specifically wrote it that way to *avoid* any performance penalty for modules that do not use any hooks, or for operators other than the one a hook is defined for.
(There may be some confusion because of the 'if the hook is undefined' clause in the proposal: I intended that to apply if the hook is deleted or conditionally assigned - if it's never assigned at all, the code won't look for it when the operator is used.)
Which of these are you expecting to be detected, and thus cause the change in bytecode? __operatorhook_or__ = lambda obj1, obj2: ... def init(): global __operatorhook_or__ def __operatorhook_or__(obj1, obj2): ... init() globals()["__operatorhook_or__"] = lambda obj1, obj2: ... exec("def __operatorhook_or__(obj1, obj2): ...") from othermodule import __operatorhook_or__ from othermodule import * ChrisA
On Wed, Dec 4, 2019, at 11:51, Chris Angelico wrote:
Which of these are you expecting to be detected, and thus cause the change in bytecode?
More or less the same sort of operations where the use of the name super is detected within methods and causes the enclosing class to have a __class__ cell.
__operatorhook_or__ = lambda obj1, obj2: ... yes
def init(): global __operatorhook_or__ def __operatorhook_or__(obj1, obj2): ... init() yes
globals()["__operatorhook_or__"] = lambda obj1, obj2: ... no
exec("def __operatorhook_or__(obj1, obj2): ...") no
from othermodule import __operatorhook_or__ yes
from othermodule import * no
04.12.19 18:39, Random832 пише:
On Wed, Dec 4, 2019, at 08:26, Serhiy Storchaka wrote:
03.12.19 20:43, Brett Cannon пише:
-1 from me. I can see someone not realizing an operator was changed, assuming it's standard semantics, and then having things break subtly. And debugging this wouldn't be fun either. To me this is monkeypatching without an explicit need for it, i.e. if you really want different semantics in your module then define a function and use that instead of influence-at-a-distance overriding of syntax.
This will also add a significant performance penalty to any code (even if you do not use any hooks). -1 from me too.
My proposal was that any module that never uses any hooks compiles to the exact same bytecode, and executes exactly the same way, as it does now. Only modules where the name of the hook is defined get a different operation that looks for the hook. I specifically wrote it that way to *avoid* any performance penalty for modules that do not use any hooks, or for operators other than the one a hook is defined for.
It is not possible. First, the code using operators can be above the code for the hook. Second, the code is generated at the compile time, but the hook is defined at the run time (when all module variables are defined). Third, the hook can be defined not only by the "def" statement in the module body. So the hook should be checked dynamically and it is much more expensive than using a field of the C structure.
On Dec 3, 2019, at 08:31, Random832 <random832@fastmail.com> wrote:
What if there was a general mechanism to allow operators to be implemented by user code that does not belong to the class?
This seems like something you could build as an import hook pretty easily. Obviously it won’t be as efficient as something using new bytecodes would be, but I don’t think it would be too slow to play with and show off how it can be useful. And instead of people having to ask you how it deals with every edge case (as they think them up one by one), they could just test it.
participants (8)
-
Andrew Barnert -
Brett Cannon -
Chris Angelico -
Ethan Furman -
Guido van Rossum -
Random832 -
Ricky Teachey -
Serhiy Storchaka