[new idea] dynamic return with using function argument
Dear Python community, I am a heavy python user in numerical simulation. In python 3.8, we will have a new syntax `:=` that could assign values to variables as part of larger expression like, ``` if (n:=len(a)) > 10: .... ``` On the other hand, I also think that it would be useful if `:=` could be used in function argument, so that a local variable created inside a function could be returned **optionally**, for example, ``` def func(arg1, arg2, karg1=karg1_default, karg_return:=karg_return): ... karg_return = 3 ... return func_output # normal calling of func() without returning variable karg_return # in this way, karg_return:=karg_return is not used and karg_return is only simply a local variable in func() output = func(2,3,karg1=4) # calling func() with using the karg_return argument # in this way, value of the local variable karg_return in func() is created and "passed" to variable a output = func(2,3,karg1=4,a:=karg_return) print(a) # a = 3 ``` Is it possible to add this new feature? Thanks for your attention to this matter. Huang Y.W.
I think you're talking about call-by-reference (Google it). What would be your use case? Do you know you can return multiple values from a function using a tuple? E.g. def foo(): return 3, 42 x, y = foo() print(x) # 3 print(y) # 42 --Guido On Mon, Aug 26, 2019 at 5:57 AM HUANG YUWEI <lelouchkako@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
This also looks a lot like Out Parameters in C# ( https://www.c-sharpcorner.com/article/out-parameter-in-c-sharp-7/) They have a similar dichotomy of declaration and call time semantics using the `out` keyword rather than an assignment operator. On Mon, Aug 26, 2019 at 10:50 AM Guido van Rossum <guido@python.org> wrote:
-- CALVIN SPEALMAN SENIOR QUALITY ENGINEER cspealma@redhat.com M: +1.336.210.5107 [image: https://red.ht/sig] <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
Dear Mr. Guido and Mr. Spealman, Thanks for your quick reply. Yes, something like "Out Parameters in C#" is exactly what I mentioned. "calling by reference" is very close, except that - with "calling by reference", the object is initialized before the function is called, and then being passed to function and modified. - in my idea, the object is initialized (created) inside a function, and its pointer (or reference) is passed to a variable X as the initialization of variable X. The point is that, the returned value may not have fixed type or array shape due to the possible existence of "if condition" inside the function. On the other hand, "calling by reference" fixes these attributes of returned variable. And also you are right, this could be easily solved by returning a tuple of multiple values, and the users just pick the returned value they need. For low level function, the number of return variables may be 1 or 2; For high level function which comprises of multiple low level functions, the number of return variables could be many. Then it would look like ``` res0, res1, res2, res3, res4, res5, res6, res7, ..., resN = high_level_function() ``` or ``` results = high_level_function() res0 = results[0] res1 = results[1] ... ``` This hurts the readability of code sometimes. My idea is simply that, okay we are now able to have assign operator in large expression, then why not in a function calling statement. (if the implementation is not difficult ...) best regards, Huang Y.W Calvin Spealman <cspealma@redhat.com> 于2019年8月27日周二 上午12:00写道：
Sorry, there is no way to leverage the implementation or syntax of := for this purpose. You must be misunderstanding how they it works. You are right that the initial value of the variable at the call site is irrelevant (but there must be some initial value, e.g. None, to be able to identify the variable). Surely there are other ways to deal with the UX problem you indicate (e.g. namedtuple). --Guido On Mon, Aug 26, 2019 at 8:33 AM HUANG YUWEI <lelouchkako@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Aug 26, 2019, at 08:38, Guido van Rossum <guido@python.org> wrote:
Sorry, there is no way to leverage the implementation or syntax of := for this purpose. You must be misunderstanding how they it works.
You are right that the initial value of the variable at the call site is irrelevant (but there must be some initial value, e.g. None, to be able to identify the variable).
I’m not sure this is quite true. I do think it would be an abuse of := syntax, and any design would be hard to read and teach, but I don’t think it would be impossible to implement. With traditional pass by reference, you need some kind of dereferencing syntax—but once you have that, the “output” variables can be uninitialized. Consider this C-style pattern (which you see all over the place in ObjC): int func(Error* error = NULL) { if (error) { *error = MyError; } } The actual out variable is *error, not error, and it’s optional, and it can be passed uninitialized. But I think the OP is looking for something more magical, where the function definition just sees the Error rather than the Error*, and can just assign to error (which, under the covers, assigns to *error with the null check). That isn’t that hard to add to a C-style language—and it actually fits better into Python, where there is no explicit dereferencing anywhere. And you can still “pass” an uninitialized variable or leave it out, because you’re not really passing anything, you’re just giving a name for the machinery to bind at the end (which is just a normal assignment). To make sure I understand the proposal, let me give a simplified but complete use case so the OP can tell me it I got it wrong: def fun(out := outvar): outvar = 3 This isn’t valid syntax today; a := expression can appear in a default-value expression or a type annotation, but it can’t appear directly as a parameter in a parameter list. So, this would be entirely new syntax that could be made legal. And, once you do that, you can assign entirely new semantics. This doesn’t assign a value to out, or even make it a local variable name; out only exists for the calling machinery to get the name to bind. It does make outvar a local variable name (adds it to the list of local names in the code object), but that still doesn’t have to give it a value. It’s like having “if False: outvar = None” at the top of the body. What about on the call side? Here I’m not quite as sure, but I don’t think you can have a := expression in an argument list without wrapping it in parens? If so, then this is also new syntax: fun(myvar:=out) If I’m wrong, then you’d need some other syntax to say “pass myvar as a keyword argument to the out parameter, but pass by name/by reference/whatever”. (You can’t rely on the fact that out was special in the def, because you can’t see what’s being called; it has to be something in the call syntax.) But I’m sure you could invent something. And whatever the syntax is, the semantics don’t require any existing value in myvar. So, how do you implement this? If you want intermediate assigns to be available even if you exited by exception, you’d probably have to use cells. But I think most languages with explicit outvar syntax don’t actually assign back until the function returns normally. And that’s easy. Within the function, outvar is just a plain old local. The RETURN_VALUE code sees that the code object has N outvars, so it actually returns a tuple of (real_return_value, *outvars). (This is an UnboundLocalError if any of them didn’t get assigned to, even if they don’t get bound to caller names.) Then, on the caller side, the call was just compiled to a CALL_FUNCTION and then a tuple extraction and a bunch of STORE_* ops (ignoring any outvars that weren’t bound). But, as I said at the top, I think this would be hard to teach and learn, and confusing to read even after you’d learned it. And returning a namedtuple (or taking a SimpleNamespace to fill, or just making it a mutating method of an object…) does seem like a much better way to solve the UX problem, as you said.
I'd love to squash this conversation (because I don't think it'll lead anywhere) but you've nerd-sniped me a little bit. In 3.8, foo(a := 42) is in fact valid syntax, meaning the same as foo((a := 42)). If you wanted to do this with explicit syntax, you can just pass any mutable object. We have a convention in Python 2 (which doesn't have nonlocal) for setting a = [None] in an outer scope and then in the inner scope assigning a[0] = <value>. This can be done for function arguments too. In this case you would be right that the initial value of the variable is constrained. You can come up with new syntax in 'def' that make this transparent inside the function (e.g. your 'def fun(out := outvar)') but in the caller it will require special syntax because the Python compiler can't know the signature of the called function (this is not a limitation of the CPython bytecode compiler but a feature of the language). We could probably come up with vaguely acceptable syntax similar to C, making the caller write foo(&a). (We could even make this introduce 'a' as a local variable if it wasn't already known.) This would be a runtime type error if foo() doesn't have an output parameter in that position. But I don't think the use case is as strong as the OP believes it is. --Guido On Mon, Aug 26, 2019 at 10:02 AM Andrew Barnert <abarnert@yahoo.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Aug 26, 2019, at 10:51, Guido van Rossum <guido@python.org> wrote:
I'd love to squash this conversation (because I don't think it'll lead anywhere) but you've nerd-sniped me a little bit.
Well, you started it by saying it was impossible. :)
In 3.8, foo(a := 42) is in fact valid syntax, meaning the same as foo((a := 42)).
OK, I thought that was illegal. Sorry; then we would indeed need new syntax to do it.
You can come up with new syntax in 'def' that make this transparent inside the function (e.g. your 'def fun(out := outvar)') but in the caller it will require special syntax because the Python compiler can't know the signature of the called function (this is not a limitation of the CPython bytecode compiler but a feature of the language).
Right; the caller can’t know any more than that it’s calling some callable expression, so we need explicit syntax on the call side to signal different behavior. And if spam(a:=42) is not currently illegal, we can’t just abuse that. So the question is what’s the closest non-horrible syntax we _could_ use? No, I take that back. I agree with you that Python doesn’t need this feature, so the question of the least horrible way to signal it shouldn’t arise in the first place; apologies for dragging you into that question anyway, and you should stop reading this message here…
We could probably come up with vaguely acceptable syntax similar to C, making the caller write foo(&a). (We could even make this introduce 'a' as a local variable if it wasn't already known.) This would be a runtime type error if foo() doesn't have an output parameter in that position.
This is pretty different from what the OP wanted—it makes you think C-style explicit pointers for pass by reference instead of C#-style out params, and that raises expectations that may not be warranted (like if you assign to the out param before raising, you’d expect the caller to be able to see that assignment). On the other hand, it does allow positional rather than just (pseudo-)keyword arguments to out params, which… I have no idea if that’s a benefit or not, to be honest, but it might be. It also conceivably allows for in-our params: if a had a value, then outvar starts with that value instead of starting unbound. Again, I have no idea if that’s a benefit or just a further source of confusion. Another option would be syntax that just treats out params as a whole different thing from regular params on the call side, say: spam()->(a:=out) Is there any context where a call followed by -> is valid today? (And maybe you could even something similar on the def side, although fitting the “return params” in with the type of the normal return might be tricky.) On the plus side, this is ugly enough that hopefully nobody will ever ask for it. :)
In 3.8, foo(a := 42) is in fact valid syntax, meaning the same as foo((a := 42)).
Yes, I ignored that assign operator is actually a valid syntax in function calling, and what I need is new syntax. Although the "dynamic return" could be implemented, I agree that it is unworthy and namedtuple or a class could solve the problem. Thanks for all your kind explanation. Let us close this conversation. Huang Y.W Greg Ewing <greg.ewing@canterbury.ac.nz> 于2019年8月27日周二 上午8:01写道：
I think you're talking about call-by-reference (Google it). What would be your use case? Do you know you can return multiple values from a function using a tuple? E.g. def foo(): return 3, 42 x, y = foo() print(x) # 3 print(y) # 42 --Guido On Mon, Aug 26, 2019 at 5:57 AM HUANG YUWEI <lelouchkako@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
This also looks a lot like Out Parameters in C# ( https://www.c-sharpcorner.com/article/out-parameter-in-c-sharp-7/) They have a similar dichotomy of declaration and call time semantics using the `out` keyword rather than an assignment operator. On Mon, Aug 26, 2019 at 10:50 AM Guido van Rossum <guido@python.org> wrote:
-- CALVIN SPEALMAN SENIOR QUALITY ENGINEER cspealma@redhat.com M: +1.336.210.5107 [image: https://red.ht/sig] <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
Dear Mr. Guido and Mr. Spealman, Thanks for your quick reply. Yes, something like "Out Parameters in C#" is exactly what I mentioned. "calling by reference" is very close, except that - with "calling by reference", the object is initialized before the function is called, and then being passed to function and modified. - in my idea, the object is initialized (created) inside a function, and its pointer (or reference) is passed to a variable X as the initialization of variable X. The point is that, the returned value may not have fixed type or array shape due to the possible existence of "if condition" inside the function. On the other hand, "calling by reference" fixes these attributes of returned variable. And also you are right, this could be easily solved by returning a tuple of multiple values, and the users just pick the returned value they need. For low level function, the number of return variables may be 1 or 2; For high level function which comprises of multiple low level functions, the number of return variables could be many. Then it would look like ``` res0, res1, res2, res3, res4, res5, res6, res7, ..., resN = high_level_function() ``` or ``` results = high_level_function() res0 = results[0] res1 = results[1] ... ``` This hurts the readability of code sometimes. My idea is simply that, okay we are now able to have assign operator in large expression, then why not in a function calling statement. (if the implementation is not difficult ...) best regards, Huang Y.W Calvin Spealman <cspealma@redhat.com> 于2019年8月27日周二 上午12:00写道：
Sorry, there is no way to leverage the implementation or syntax of := for this purpose. You must be misunderstanding how they it works. You are right that the initial value of the variable at the call site is irrelevant (but there must be some initial value, e.g. None, to be able to identify the variable). Surely there are other ways to deal with the UX problem you indicate (e.g. namedtuple). --Guido On Mon, Aug 26, 2019 at 8:33 AM HUANG YUWEI <lelouchkako@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Aug 26, 2019, at 08:38, Guido van Rossum <guido@python.org> wrote:
Sorry, there is no way to leverage the implementation or syntax of := for this purpose. You must be misunderstanding how they it works.
You are right that the initial value of the variable at the call site is irrelevant (but there must be some initial value, e.g. None, to be able to identify the variable).
I’m not sure this is quite true. I do think it would be an abuse of := syntax, and any design would be hard to read and teach, but I don’t think it would be impossible to implement. With traditional pass by reference, you need some kind of dereferencing syntax—but once you have that, the “output” variables can be uninitialized. Consider this C-style pattern (which you see all over the place in ObjC): int func(Error* error = NULL) { if (error) { *error = MyError; } } The actual out variable is *error, not error, and it’s optional, and it can be passed uninitialized. But I think the OP is looking for something more magical, where the function definition just sees the Error rather than the Error*, and can just assign to error (which, under the covers, assigns to *error with the null check). That isn’t that hard to add to a C-style language—and it actually fits better into Python, where there is no explicit dereferencing anywhere. And you can still “pass” an uninitialized variable or leave it out, because you’re not really passing anything, you’re just giving a name for the machinery to bind at the end (which is just a normal assignment). To make sure I understand the proposal, let me give a simplified but complete use case so the OP can tell me it I got it wrong: def fun(out := outvar): outvar = 3 This isn’t valid syntax today; a := expression can appear in a default-value expression or a type annotation, but it can’t appear directly as a parameter in a parameter list. So, this would be entirely new syntax that could be made legal. And, once you do that, you can assign entirely new semantics. This doesn’t assign a value to out, or even make it a local variable name; out only exists for the calling machinery to get the name to bind. It does make outvar a local variable name (adds it to the list of local names in the code object), but that still doesn’t have to give it a value. It’s like having “if False: outvar = None” at the top of the body. What about on the call side? Here I’m not quite as sure, but I don’t think you can have a := expression in an argument list without wrapping it in parens? If so, then this is also new syntax: fun(myvar:=out) If I’m wrong, then you’d need some other syntax to say “pass myvar as a keyword argument to the out parameter, but pass by name/by reference/whatever”. (You can’t rely on the fact that out was special in the def, because you can’t see what’s being called; it has to be something in the call syntax.) But I’m sure you could invent something. And whatever the syntax is, the semantics don’t require any existing value in myvar. So, how do you implement this? If you want intermediate assigns to be available even if you exited by exception, you’d probably have to use cells. But I think most languages with explicit outvar syntax don’t actually assign back until the function returns normally. And that’s easy. Within the function, outvar is just a plain old local. The RETURN_VALUE code sees that the code object has N outvars, so it actually returns a tuple of (real_return_value, *outvars). (This is an UnboundLocalError if any of them didn’t get assigned to, even if they don’t get bound to caller names.) Then, on the caller side, the call was just compiled to a CALL_FUNCTION and then a tuple extraction and a bunch of STORE_* ops (ignoring any outvars that weren’t bound). But, as I said at the top, I think this would be hard to teach and learn, and confusing to read even after you’d learned it. And returning a namedtuple (or taking a SimpleNamespace to fill, or just making it a mutating method of an object…) does seem like a much better way to solve the UX problem, as you said.
I'd love to squash this conversation (because I don't think it'll lead anywhere) but you've nerd-sniped me a little bit. In 3.8, foo(a := 42) is in fact valid syntax, meaning the same as foo((a := 42)). If you wanted to do this with explicit syntax, you can just pass any mutable object. We have a convention in Python 2 (which doesn't have nonlocal) for setting a = [None] in an outer scope and then in the inner scope assigning a[0] = <value>. This can be done for function arguments too. In this case you would be right that the initial value of the variable is constrained. You can come up with new syntax in 'def' that make this transparent inside the function (e.g. your 'def fun(out := outvar)') but in the caller it will require special syntax because the Python compiler can't know the signature of the called function (this is not a limitation of the CPython bytecode compiler but a feature of the language). We could probably come up with vaguely acceptable syntax similar to C, making the caller write foo(&a). (We could even make this introduce 'a' as a local variable if it wasn't already known.) This would be a runtime type error if foo() doesn't have an output parameter in that position. But I don't think the use case is as strong as the OP believes it is. --Guido On Mon, Aug 26, 2019 at 10:02 AM Andrew Barnert <abarnert@yahoo.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Aug 26, 2019, at 10:51, Guido van Rossum <guido@python.org> wrote:
I'd love to squash this conversation (because I don't think it'll lead anywhere) but you've nerd-sniped me a little bit.
Well, you started it by saying it was impossible. :)
In 3.8, foo(a := 42) is in fact valid syntax, meaning the same as foo((a := 42)).
OK, I thought that was illegal. Sorry; then we would indeed need new syntax to do it.
You can come up with new syntax in 'def' that make this transparent inside the function (e.g. your 'def fun(out := outvar)') but in the caller it will require special syntax because the Python compiler can't know the signature of the called function (this is not a limitation of the CPython bytecode compiler but a feature of the language).
Right; the caller can’t know any more than that it’s calling some callable expression, so we need explicit syntax on the call side to signal different behavior. And if spam(a:=42) is not currently illegal, we can’t just abuse that. So the question is what’s the closest non-horrible syntax we _could_ use? No, I take that back. I agree with you that Python doesn’t need this feature, so the question of the least horrible way to signal it shouldn’t arise in the first place; apologies for dragging you into that question anyway, and you should stop reading this message here…
We could probably come up with vaguely acceptable syntax similar to C, making the caller write foo(&a). (We could even make this introduce 'a' as a local variable if it wasn't already known.) This would be a runtime type error if foo() doesn't have an output parameter in that position.
This is pretty different from what the OP wanted—it makes you think C-style explicit pointers for pass by reference instead of C#-style out params, and that raises expectations that may not be warranted (like if you assign to the out param before raising, you’d expect the caller to be able to see that assignment). On the other hand, it does allow positional rather than just (pseudo-)keyword arguments to out params, which… I have no idea if that’s a benefit or not, to be honest, but it might be. It also conceivably allows for in-our params: if a had a value, then outvar starts with that value instead of starting unbound. Again, I have no idea if that’s a benefit or just a further source of confusion. Another option would be syntax that just treats out params as a whole different thing from regular params on the call side, say: spam()->(a:=out) Is there any context where a call followed by -> is valid today? (And maybe you could even something similar on the def side, although fitting the “return params” in with the type of the normal return might be tricky.) On the plus side, this is ugly enough that hopefully nobody will ever ask for it. :)
In 3.8, foo(a := 42) is in fact valid syntax, meaning the same as foo((a := 42)).
Yes, I ignored that assign operator is actually a valid syntax in function calling, and what I need is new syntax. Although the "dynamic return" could be implemented, I agree that it is unworthy and namedtuple or a class could solve the problem. Thanks for all your kind explanation. Let us close this conversation. Huang Y.W Greg Ewing <greg.ewing@canterbury.ac.nz> 于2019年8月27日周二 上午8:01写道：
participants (5)
-
Andrew Barnert -
Calvin Spealman -
Greg Ewing -
Guido van Rossum -
HUANG YUWEI