Re: [Python-ideas] Operator as first class citizens -- like in scala -- or yet another new operator?
On Wed, May 22, 2019 at 10:03 PM Yanghao Hua <yanghao.py@gmail.com> wrote:
To be first-class citizens, operators would have to be able to be passed to functions - for instance:
def frob(x, y, oper): return x oper y assert frob(10, 20, +) == 30 assert frob(10, 20, *) == 200
The nearest Python currently has to this is the "operator" module, in which you'll find function versions of the operators:
def frob(x, y, oper): return oper(x, y) assert frob(10, 20, operator.add) == 30 assert frob(10, 20, operator.mul) == 200
Is this what you're talking about?
Yes this is exactly what I am talking about .. and this is exactly what scala does. In scala, you can do "a + b", as well as "a.+(b)", where "+" is just a function of object a.
You know people keep building domain specific languages for specific problems, and scala seems currently the only one that can truely enable people to build DSL elegantly. I actually do not know at all how to do this in CPython as I just started playing with Python internals a few days ago, but having an arrow operator would solve my immediate problem in terms of using Python to build a DSL for hardware design (elegantly!).
Does this make sense?
Yes, it does make sense. Forgive my lack of Scala knowledge, but is it possible for 'b' in your example to be the one that handles the addition? Specific case: class Seven: def __add__(self, other): return other + 7 def __radd__(self, other): return 7 + other seven = Seven() print(seven + 1) print(2 + seven) This works in Python even though 2.+(seven) wouldn't know how to handle this object. So it returns NotImplemented, and Python says, oh okay, maybe the other object knows how to do this. (That's what __radd__ is for.) This is why Python's operators are all defined by the language, with multiple levels of protocol. ChrisA
Yes, it does make sense. Forgive my lack of Scala knowledge, but is it possible for 'b' in your example to be the one that handles the addition? Specific case:
class Seven: def __add__(self, other): return other + 7 def __radd__(self, other): return 7 + other
seven = Seven() print(seven + 1) print(2 + seven)
This works in Python even though 2.+(seven) wouldn't know how to handle this object. So it returns NotImplemented, and Python says, oh okay, maybe the other object knows how to do this. (That's what __radd__ is for.) This is why Python's operators are all defined by the language, with multiple levels of protocol.
Problem is python do not allow you to define new operators in the language itself, except those pre-defined you can modify their behavior. Even in case, if python would have been able to allow me to redefine the behavior of "=", e.g. by checking if the left hand side has a method (e.g. __assign__) to override the default behavior of the equal sign, I would be 100% happy already :) I have looked into all the existing operators that python supports which looks like an "assignment", none has been fallen in love with me. The example in Chisel w/ Scala is: they defined quite a few operators: ":=" for signal assignment, "<>" for bulk signal connections etc. And you can keep inventing new operators with which your design can look really elegant. With Python, if people already had the thought to allow user to define arbitrary operators, I would like to wait and contribute to that effort. Because I think this will make Python the language of choice for almost all future domain specific language designs. If not, with my limited knowledge in Python internals, right now I can only implement this "<-" "->" two arrow operators and I myself are not sure if you guys would think it is worth a PEP and actually make it into the language. And this is something I have in mind for a Python DSL for HDL: def combinational_or_sequential_logic(in: Signal, out: Signal): local_signal = Signal() local_signal <- in << 10 # read as: local_signal <- (in << 10) out <- local_signal + 5 # read as out <- (local_signal + 5) # new_signal <- 3 will raise exception, as <- does not create new object. And the arrow operators could also be used for bulk connections of a chain of hardware modules like this: module_instance1 -> instance2 -> instance3->... to naturally form a hardware pipeline ... this looks very elegant.
On 5/22/19 8:29 AM, Yanghao Hua wrote:
Yes, it does make sense. Forgive my lack of Scala knowledge, but is it possible for 'b' in your example to be the one that handles the addition? Specific case:
class Seven: def __add__(self, other): return other + 7 def __radd__(self, other): return 7 + other
seven = Seven() print(seven + 1) print(2 + seven)
This works in Python even though 2.+(seven) wouldn't know how to handle this object. So it returns NotImplemented, and Python says, oh okay, maybe the other object knows how to do this. (That's what __radd__ is for.) This is why Python's operators are all defined by the language, with multiple levels of protocol. Problem is python do not allow you to define new operators in the language itself, except those pre-defined you can modify their behavior. Even in case, if python would have been able to allow me to redefine the behavior of "=", e.g. by checking if the left hand side has a method (e.g. __assign__) to override the default behavior of the equal sign, I would be 100% happy already :) I have looked into all the existing operators that python supports which looks like an "assignment", none has been fallen in love with me. The example in Chisel w/ Scala is: they defined quite a few operators: ":=" for signal assignment, "<>" for bulk signal connections etc. And you can keep inventing new operators with which your design can look really elegant.
With Python, if people already had the thought to allow user to define arbitrary operators, I would like to wait and contribute to that effort. Because I think this will make Python the language of choice for almost all future domain specific language designs. If not, with my limited knowledge in Python internals, right now I can only implement this "<-" "->" two arrow operators and I myself are not sure if you guys would think it is worth a PEP and actually make it into the language.
And this is something I have in mind for a Python DSL for HDL:
def combinational_or_sequential_logic(in: Signal, out: Signal): local_signal = Signal() local_signal <- in << 10 # read as: local_signal <- (in << 10) out <- local_signal + 5 # read as out <- (local_signal + 5) # new_signal <- 3 will raise exception, as <- does not create new object.
And the arrow operators could also be used for bulk connections of a chain of hardware modules like this: module_instance1 -> instance2 -> instance3->... to naturally form a hardware pipeline ... this looks very elegant.
I think the largest problem with this idea has to do with where the new operator would be defined, and then where would it be used. At first blush, it seems like you'd want to define the operator in one file (let's call it hdl.py), then import that into another file (circuit.py), which could use the new operator. But Python compiles circuit.py without reading hdl.py at all. It merely compiles "import hdl" to some bytecode to import hdl.py. So how would the operator's definition be used during the compilation of circuit.py? The compiler has no idea that new operators have been defined. --Ned.
And this is something I have in mind for a Python DSL for HDL:
def combinational_or_sequential_logic(in: Signal, out: Signal): local_signal = Signal() local_signal <- in << 10 # read as: local_signal <- (in << 10) out <- local_signal + 5 # read as out <- (local_signal + 5) # new_signal <- 3 will raise exception, as <- does not create new object.
And the arrow operators could also be used for bulk connections of a chain of hardware modules like this: module_instance1 -> instance2 -> instance3->... to naturally form a hardware pipeline ... this looks very elegant.
I think the largest problem with this idea has to do with where the new operator would be defined, and then where would it be used. At first blush, it seems like you'd want to define the operator in one file (let's call it hdl.py), then import that into another file (circuit.py), which could use the new operator.
But Python compiles circuit.py without reading hdl.py at all. It merely compiles "import hdl" to some bytecode to import hdl.py. So how would the operator's definition be used during the compilation of circuit.py? The compiler has no idea that new operators have been defined.
The scala implementation purely relies on the object to provide the operator definition, e.g. "a b c" would be interpreted as "a.b(c)". With the current implementation of the arrow operators I have, "<-" and "->" are part of of the basic python operators pretty much like "+" and "-". Where it relies on the object to provide the special method "__arrow__". so, this operation could be defined in a separate file with a base class e.g. Signal or Module, where other python module has to import it to use it or inherit it. In this case, essentially when python compiler reads circuit.py, it reads hdl.py too as it is imported in circuit.py. This is exactly how __matmult__ special method works too in PEP465.
On 5/22/19 8:56 AM, Yanghao Hua wrote:
And this is something I have in mind for a Python DSL for HDL:
def combinational_or_sequential_logic(in: Signal, out: Signal): local_signal = Signal() local_signal <- in << 10 # read as: local_signal <- (in << 10) out <- local_signal + 5 # read as out <- (local_signal + 5) # new_signal <- 3 will raise exception, as <- does not create new object.
And the arrow operators could also be used for bulk connections of a chain of hardware modules like this: module_instance1 -> instance2 -> instance3->... to naturally form a hardware pipeline ... this looks very elegant.
I think the largest problem with this idea has to do with where the new operator would be defined, and then where would it be used. At first blush, it seems like you'd want to define the operator in one file (let's call it hdl.py), then import that into another file (circuit.py), which could use the new operator.
But Python compiles circuit.py without reading hdl.py at all. It merely compiles "import hdl" to some bytecode to import hdl.py. So how would the operator's definition be used during the compilation of circuit.py? The compiler has no idea that new operators have been defined. The scala implementation purely relies on the object to provide the operator definition, e.g. "a b c" would be interpreted as "a.b(c)". With the current implementation of the arrow operators I have, "<-" and "->" are part of of the basic python operators pretty much like "+" and "-". Where it relies on the object to provide the special method "__arrow__". so, this operation could be defined in a separate file with a base class e.g. Signal or Module, where other python module has to import it to use it or inherit it.
In this case, essentially when python compiler reads circuit.py, it reads hdl.py too as it is imported in circuit.py. This is exactly how __matmult__ special method works too in PEP465.
If you are proposing to add "<-" and "->" to the fundamental Python grammar, then it could work as PEP465 does. But if you are proposing that classes can define new lexical operators, it would have to work very very differently. You started with "operators as first class citizens," but I don't know what you mean by that. Usually "first class citizens" means that a thing can be used as a value, for example, passed as an argument to a function. I don't think that is what you mean. Earlier you said:
With Python, if people already had the thought to allow user to define arbitrary operators, I would like to wait and contribute to that effort. Because I think this will make Python the language of choice for almost all future domain specific language designs. So I think what you mean by "first-class citizens" is that ability: to define new operator tokens in Python itself. Unfortunately, this would require untenable changes to how Python operates.
When Python compiles a file, it does not read the imported files at all. That doesn't happen until run time. So how will Python know what "<-" means in circuit.py? It's never seen this token before, so it can't know what special method it should map onto. It can't even know what the token is. What if it encounters "x ~++<-+ y"? What are the tokens in that source? --Ned.
On Wed, May 22, 2019 at 9:57 AM Yanghao Hua <yanghao.py@gmail.com> wrote:
And this is something I have in mind for a Python DSL for HDL:
Perhaps you might be able to do what you want using an import hook. I have done some experiments with introducing some new operators that way: https://github.com/aroberge/experimental/blob/master/experimental/transforme...
def combinational_or_sequential_logic(in: Signal, out: Signal): local_signal = Signal() local_signal <- in << 10 # read as: local_signal <- (in << 10) out <- local_signal + 5 # read as out <- (local_signal + 5) # new_signal <- 3 will raise exception, as <- does not create new
object.
And the arrow operators could also be used for bulk connections of a chain of hardware modules like this: module_instance1 -> instance2 -> instance3->... to naturally form a hardware pipeline ... this looks very elegant.
I think the largest problem with this idea has to do with where the new operator would be defined, and then where would it be used. At first blush, it seems like you'd want to define the operator in one file (let's call it hdl.py), then import that into another file (circuit.py), which could use the new operator.
But Python compiles circuit.py without reading hdl.py at all. It merely compiles "import hdl" to some bytecode to import hdl.py. So how would the operator's definition be used during the compilation of circuit.py? The compiler has no idea that new operators have been defined.
The scala implementation purely relies on the object to provide the operator definition, e.g. "a b c" would be interpreted as "a.b(c)". With the current implementation of the arrow operators I have, "<-" and "->" are part of of the basic python operators pretty much like "+" and "-". Where it relies on the object to provide the special method "__arrow__". so, this operation could be defined in a separate file with a base class e.g. Signal or Module, where other python module has to import it to use it or inherit it.
In this case, essentially when python compiler reads circuit.py, it reads hdl.py too as it is imported in circuit.py. This is exactly how __matmult__ special method works too in PEP465. _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On May 22, 2019, at 10:51 AM, Andre Roberge <andre.roberge@gmail.com> wrote:
On Wed, May 22, 2019 at 9:57 AM Yanghao Hua <yanghao.py@gmail.com> wrote:
And this is something I have in mind for a Python DSL for HDL:
Perhaps you might be able to do what you want using an import hook. I have done some experiments with introducing some new operators that way: https://github.com/aroberge/experimental/blob/master/experimental/transforme...
You might also want to look at macropy https://github.com/lihaoyi/macropy/blob/master/readme.rst , although I don’t know if it supports new operators. Eric
def combinational_or_sequential_logic(in: Signal, out: Signal): local_signal = Signal() local_signal <- in << 10 # read as: local_signal <- (in << 10) out <- local_signal + 5 # read as out <- (local_signal + 5) # new_signal <- 3 will raise exception, as <- does not create new object.
And the arrow operators could also be used for bulk connections of a chain of hardware modules like this: module_instance1 -> instance2 -> instance3->... to naturally form a hardware pipeline ... this looks very elegant.
I think the largest problem with this idea has to do with where the new operator would be defined, and then where would it be used. At first blush, it seems like you'd want to define the operator in one file (let's call it hdl.py), then import that into another file (circuit.py), which could use the new operator.
But Python compiles circuit.py without reading hdl.py at all. It merely compiles "import hdl" to some bytecode to import hdl.py. So how would the operator's definition be used during the compilation of circuit.py? The compiler has no idea that new operators have been defined.
The scala implementation purely relies on the object to provide the operator definition, e.g. "a b c" would be interpreted as "a.b(c)". With the current implementation of the arrow operators I have, "<-" and "->" are part of of the basic python operators pretty much like "+" and "-". Where it relies on the object to provide the special method "__arrow__". so, this operation could be defined in a separate file with a base class e.g. Signal or Module, where other python module has to import it to use it or inherit it.
In this case, essentially when python compiler reads circuit.py, it reads hdl.py too as it is imported in circuit.py. This is exactly how __matmult__ special method works too in PEP465. _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
You might also want to look at macropy https://github.com/lihaoyi/macropy/blob/master/readme.rst , although I don’t know if it supports new operators.
Thanks for the pointer, I am also not sure how to apply it after a quick going through of its documentation. So far the quickest thing to get something working is to modify CPython itself ;-)
On 22/05/2019 13:29, Yanghao Hua wrote:
Problem is python do not allow you to define new operators in the language itself, except those pre-defined you can modify their behavior. Even in case, if python would have been able to allow me to redefine the behavior of "=", e.g. by checking if the left hand side has a method (e.g. __assign__) to override the default behavior of the equal sign, I would be 100% happy already :) I have looked into all the existing operators that python supports which looks like an "assignment", none has been fallen in love with me. The example in Chisel w/ Scala is: they defined quite a few operators: ":=" for signal assignment, "<>" for bulk signal connections etc. And you can keep inventing new operators with which your design can look really elegant.
A note of caution: your design can look really elegant *as long as you know what your new operators mean*. If you don't, it's looks like incomprehensible noise. I have had far too many bad experiences with "clever" C++ with redefined operators to assume that "elegant" and "obvious" go hand in hand. If you really must prat around with redefining assignment, make the relevant attribute a property. -- Rhodri James *-* Kynesim Ltd
participants (6)
-
Andre Roberge -
Chris Angelico -
Eric V. Smith -
Ned Batchelder -
Rhodri James -
Yanghao Hua