'where' statement in Python?
Dear members of the python-ideas mailing list, I'm not quite sure if this is the right place to ask for feedback about the idea, I apologize if this is not the case. I'm considering the following extension to Python's grammar: adding the 'where' keyword, which would work as follows: where_expr : expr 'where' NAME '=' expr The idea is to be able to write something like: a = (z**2+5) where z=2 being equivalent to (current Python syntax): a = (lambda z: z**2+5)(z=2) I thinkg this would be especially powerful in cases where the variable to be substituted ('z' in the example) comes in turn from a complicated expression, which makes it confusing to "embed" it in the main expression (the body of the 'lambda'), or in cases where the substitution must be performed more than once, and it may be more efficient to evaluate 'z' once. A more complicated example: vtype = decl[par_pos+1:FindMatching(par_pos, decl)].strip() where par_pos=decl.find('(') equivalent to (current Python syntax): vtype = (lambda par_pos: decl[par_pos+1:FindMatching(par_pos,decl)].strip())(par_pos=decl.find('(')) Extending this syntax to several assignments after the 'where' keyword could be implemented as: where_expr: expr 'where' NAME '=' expr (',' NAME '=' expr )* or (which I think may be more "pythonic"): where_expr: expr 'where' NAME (',' NAME)* '=' expr (',' expr)* as it mimics the same syntax for unpacking tuples. I would appreciate any feedback on the idea, especially if it has some obvious flaw or if it's redundant (meaning there is a clearer way of doing this 'trick' which I don't know about). best regards, Sergio Davis -- Sergio Davis Irarrázabal Grupo de NanoMateriales, Universidad de Chile http://www.gnm.cl/~sdavis
On Mon, Jul 19, 2010 at 9:45 PM, Sergio Davis <sergdavis@gmail.com> wrote:
Dear members of the python-ideas mailing list,
I'm not quite sure if this is the right place to ask for feedback about the idea, I apologize if this is not the case.
I'm considering the following extension to Python's grammar: adding the 'where' keyword, which would work as follows:
where_expr : expr 'where' NAME '=' expr
The idea is to be able to write something like:
a = (z**2+5) where z=2
being equivalent to (current Python syntax):
a = (lambda z: z**2+5)(z=2)
I thinkg this would be especially powerful in cases where the variable to be substituted ('z' in the example) comes in turn from a complicated expression, which makes it confusing to "embed" it in the main expression (the body of the 'lambda'), or in cases where the substitution must be performed more than once, and it may be more efficient to evaluate 'z' once. A more complicated example:
vtype = decl[par_pos+1:FindMatching(par_pos, decl)].strip() where par_pos=decl.find('(')
equivalent to (current Python syntax):
vtype = (lambda par_pos: decl[par_pos+1:FindMatching(par_pos,decl)].strip())(par_pos=decl.find('('))
Extending this syntax to several assignments after the 'where' keyword could be implemented as:
where_expr: expr 'where' NAME '=' expr (',' NAME '=' expr )*
or (which I think may be more "pythonic"):
where_expr: expr 'where' NAME (',' NAME)* '=' expr (',' expr)*
as it mimics the same syntax for unpacking tuples.
I would appreciate any feedback on the idea, especially if it has some obvious flaw or if it's redundant (meaning there is a clearer way of doing this 'trick' which I don't know about).
I think the "trick" to making it readable is putting the assignment first. par_pos = decl.find('(') vtype = decl[par_pos+1:FindMatching(par_pos, decl)].strip() versus: vtype = decl[par_pos+1:FindMatching(par_pos, decl)].strip() where par_pos=decl.find('(') -Jack
On Tue, Jul 20, 2010 at 11:52 AM, Jack Diederich <jackdied@gmail.com> wrote:
I think the "trick" to making it readable is putting the assignment first.
par_pos = decl.find('(') vtype = decl[par_pos+1:FindMatching(par_pos, decl)].strip()
versus:
vtype = decl[par_pos+1:FindMatching(par_pos, decl)].strip() where par_pos=decl.find('(')
Note that with a "given" clause, I would recommend writing something along these lines: vtype = decl[open_paren_pos+1:close_paren_pos] given: open_paren_pos = decl.find('(') close_paren_pos = FindMatching(open_paren_pos, decl) The positions of the open and closing parentheses are only relevant in the assignment statement and you can understand what the code does based just on the names of the subexpressions without necessarily worrying about how they are determined. The question here is whether this offers *enough* benefit over just writing open_paren_pos = decl.find('(') close_paren_pos = FindMatching(open_paren_pos, decl) vtype = decl[open_paren_pos+1:close_paren_pos] to be worth the significant additional complexity it introduces. Currently I'd say the scales are leaning heavily towards "not worth the hassle", but I'd be interesting to see what people can make of the PEP 359 use cases and judicious use of the locals() function in the context of PEP 3150 (assuming the given clause semantics are exactly as described by the implementation sketch in the PEP) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Sergio Davis writes:
I'm considering the following extension to Python's grammar: adding the 'where' keyword, which would work as follows:
where_expr : expr 'where' NAME '=' expr
We just had a long thread about this. http://mail.python.org/pipermail/python-ideas/2010-June/007476.html The sentiment was about -0 to -0.5 on the idea in general, although a couple of people with experience in implementing Python syntax expressed sympathy for it. There have also been threads on earlier variations of the idea, referenced in the thread above. HTH
On Tue, Jul 20, 2010 at 3:29 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
We just had a long thread about this.
http://mail.python.org/pipermail/python-ideas/2010-June/007476.html
The sentiment was about -0 to -0.5 on the idea in general, although a couple of people with experience in implementing Python syntax expressed sympathy for it.
For the record, I am personally +1 on the idea (otherwise I wouldn't have put so much thought into it over the years). It's just a *lot* harder to define complete and consistent semantics for the concept than people often realise. However, having the question come up twice within the last month finally inspired me to write the current status of the topic down in a deferred PEP: http://www.python.org/dev/peps/pep-3150/ Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hello, PEP-3150 is awesome, just a small addition — why not to allow one-liners `where`s: a = (b, b) where b = 43 And that also make sense for generator/list/set/dict comprehensions: mylist = [y for y in another_list if y < 5 where y = f(x)] On Tue, Jul 20, 2010 at 5:27 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Tue, Jul 20, 2010 at 3:29 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
We just had a long thread about this.
http://mail.python.org/pipermail/python-ideas/2010-June/007476.html
The sentiment was about -0 to -0.5 on the idea in general, although a couple of people with experience in implementing Python syntax expressed sympathy for it.
For the record, I am personally +1 on the idea (otherwise I wouldn't have put so much thought into it over the years). It's just a *lot* harder to define complete and consistent semantics for the concept than people often realise.
However, having the question come up twice within the last month finally inspired me to write the current status of the topic down in a deferred PEP: http://www.python.org/dev/peps/pep-3150/
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Andrey Popp phone: +7 911 740 24 91 e-mail: 8mayday@gmail.com
Also, what about some alternative for workaround the following:
Out of Order Execution: the where clause makes execution jump around a little strangely, as the body of the where clause is executed before the simple statement in the clause header. The closest any other part of Python comes to this before is the out of order evaluation in conditional expressions.
result = let: a = retrieve_a() b = retreive_b() in: a*a + b*b On Tue, Jul 20, 2010 at 6:57 PM, Andrey Popp <8mayday@gmail.com> wrote:
Hello,
PEP-3150 is awesome, just a small addition — why not to allow one-liners `where`s:
a = (b, b) where b = 43
And that also make sense for generator/list/set/dict comprehensions:
mylist = [y for y in another_list if y < 5 where y = f(x)]
On Tue, Jul 20, 2010 at 5:27 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Tue, Jul 20, 2010 at 3:29 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
We just had a long thread about this.
http://mail.python.org/pipermail/python-ideas/2010-June/007476.html
The sentiment was about -0 to -0.5 on the idea in general, although a couple of people with experience in implementing Python syntax expressed sympathy for it.
For the record, I am personally +1 on the idea (otherwise I wouldn't have put so much thought into it over the years). It's just a *lot* harder to define complete and consistent semantics for the concept than people often realise.
However, having the question come up twice within the last month finally inspired me to write the current status of the topic down in a deferred PEP: http://www.python.org/dev/peps/pep-3150/
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Andrey Popp
phone: +7 911 740 24 91 e-mail: 8mayday@gmail.com
-- Andrey Popp phone: +7 911 740 24 91 e-mail: 8mayday@gmail.com
Andrey Popp wrote:
Hello,
PEP-3150 is awesome, just a small addition — why not to allow one-liners `where`s:
a = (b, b) where b = 43
And that also make sense for generator/list/set/dict comprehensions:
mylist = [y for y in another_list if y < 5 where y = f(x)]
Do you mean: mylist = [y for x in another_list if y < 5 where y = f(x)] John =:->
Yes, exactly this, it was a typo. On Tue, Jul 20, 2010 at 7:28 PM, John Arbash Meinel <john.arbash.meinel@gmail.com> wrote:
Andrey Popp wrote:
Hello,
PEP-3150 is awesome, just a small addition — why not to allow one-liners `where`s:
a = (b, b) where b = 43
And that also make sense for generator/list/set/dict comprehensions:
mylist = [y for y in another_list if y < 5 where y = f(x)]
Do you mean:
mylist = [y for x in another_list if y < 5 where y = f(x)]
John =:->
-- Andrey Popp phone: +7 911 740 24 91 e-mail: 8mayday@gmail.com
Den 20.07.2010 16:57, skrev Andrey Popp:
a = (b, b) where b = 43
I am +1 for a where module and -1 for a where keyword, and here is the reason: In MATLAB, we have the "find" function that serves the role of where. In NumPy, we have a function numpy.where and also masked arrays. The above statement with NumPy ndarrays would be: idx, = np.where(b == 47) a = (b[idx], b[idx]) or we could simply do this: a = (b[b==47], b[b==47]) And if we look at this proposed expression, mylist = [y for y in another_list if y < 5 where y == f(x)] Using NumPy, we can proceed like this: idx, = np.where(another_array == f(x)) mylist = [y for y in another_array[idx] if y < 5] The intension is just as clear, and it avoids a new "where" keyword. It is also similar to NumPy and Matlab. Not to mention that the "where keyword" in the above expression could be replaced with an "and", so it serve no real purpose: mylist = [y for y in another_list if (y < 5 and y == f(x))] why have a where keyword here? It is just redundant. So I'd rather speak of something useful instead: NumPy's "Fancy indexing". "Fancy indexing" (NumPy jargon) will in this context mean that we allow indexes to be an iterable, not just integers: mylist[(1,2,3)] == mylist[1,2,3] mylist[iterable] == [a(i) for i in iterable] That is what NumPy and Matlab do, as well as Fortran 90 (and certain C++ libraries such as Blitz++). It has all the power of the "where keyword", while being more flexible to use, and intention is more explicit. It is also well tested syntax. Thus with "fancy indexing": alist[iterable] == [alist[i] for i in iterable] That is what we really need! Note that this is not a language syntax change, it is just a change of how __setitem__ and __getitem__ works for certain container types. NumPy already does this, so the syntax itself is completely valid Python. And as for "where", it is just a function. Andrey's proposed where keyword is a crippled tool in comparison. That is, the real power of a list of indexers is that it can be obtained and manipulated with any conceivable method, e.g. slicing. It also allows numpy to have an "argsort" function, since an index list can be reused on multiple arrays: idx = np.argsort(array_a) sorteda = array_a[idx] sortedb = array_b[idx] is the same as tmp = sorted([a,i for i,a in enumerate(lista)]) sorteda = [a for a,i in tmp] sortedb = [listb[i] for a,i in tmp] Which is the more readable? Implementing a generic "where function" can be achieved with a lambda: idx = where(lambda x: x== 47, alist) or a list comprehension (this would be very similar to NumPy): idx = where([x==47 for x in alist]) But to begin with, I think we should get NumPy style "fancy indexing" to standard container types like list, tuple, string, bytes, bytearray, array and deque. That would just be a handful of subclasses, and I think they should (initially) be put in a standard library module, and possibly replace the current cointainers in Python 4000. But as for a where keyword: My opinion is a big -1, if I have the right to vote. We should rather implement a where function and overload the mentioned container types. The where function should go in the same module. So all in all, I am +1 for a "where module" and -1 for a "where keyword". P.S. I'll admit that dict and set might add to some confusion, since "fancy indexing" would be ambigous for them. Regards, Sturla
On Tue, Jul 20, 2010 at 6:18 PM, Sturla Molden <sturla@molden.no> wrote:
Den 20.07.2010 16:57, skrev Andrey Popp:
a = (b, b) where b = 43
I am +1 for a where module and -1 for a where keyword, and here is the reason:
In MATLAB, we have the "find" function that serves the role of where. In NumPy, we have a function numpy.where and also masked arrays.
The above statement with NumPy ndarrays would be:
idx, = np.where(b == 47) a = (b[idx], b[idx])
or we could simply do this:
a = (b[b==47], b[b==47]) <snip>
It looks like NumPy's "where" is more like SQL's, while Nick's is more like Haskell's. These are totally different: in SQL it's a dynamic query (and its argument is a condition), whereas in Haskell it's purely a syntactic construct for defining some variables to be used as shorthands in an expression. Given the large number of Python users familiar with SQL compared to those familiar with Haskell, I think we'd do wise to pick a different keyword instead of 'where'. I can't think of one right now though. Your proposal is completely orthogonal to Nick's; the best thing to do is probably to start a different thread for yours. Note that Microsoft's LINQ is also similar to your suggestion. -- --Guido van Rossum (python.org/~guido)
On Tue, Jul 20, 2010 at 1:16 PM, Guido van Rossum <guido@python.org> wrote:
Given the large number of Python users familiar with SQL compared to those familiar with Haskell, I think we'd do wise to pick a different keyword instead of 'where'. I can't think of one right now though. <http://mail.python.org/mailman/listinfo/python-ideas>
Taking a cue from mathematics, how about "given"? c = sqrt(a*a + b*b) given: a = retrieve_a() b = retrieve_b() -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
On Tue, Jul 20, 2010 at 8:29 PM, Daniel Stutzbach <daniel@stutzbachenterprises.com> wrote:
On Tue, Jul 20, 2010 at 1:16 PM, Guido van Rossum <guido@python.org> wrote:
Given the large number of Python users familiar with SQL compared to those familiar with Haskell, I think we'd do wise to pick a different keyword instead of 'where'. I can't think of one right now though.
Taking a cue from mathematics, how about "given"?
c = sqrt(a*a + b*b) given: a = retrieve_a() b = retrieve_b()
Or if we'd rather overload an existing keyword than add a new one, "with" reads well too. George
i would suggest overloading the 'with', 'as' combo c = sqrt(a*a + b*b) with: get_a(), get_b() as a, b or c = sqrt(a*a + b*b) with: get_a() as a get_b() as b it reads well and plus it follows since this statement acts similarly to a regular with and as statement with the behavior of the context manager being (in psudocode): set up: store original value of variable if any and set variable to new value. tear down: set value back to original or delete from local namespace if it never had one additionally we do not need to introduce any new keywords any way that this is implemented though it would be quite useful On Tue, Jul 20, 2010 at 2:35 PM, George Sakkis <george.sakkis@gmail.com>wrote:
On Tue, Jul 20, 2010 at 8:29 PM, Daniel Stutzbach <daniel@stutzbachenterprises.com> wrote:
On Tue, Jul 20, 2010 at 1:16 PM, Guido van Rossum <guido@python.org> wrote:
Given the large number of Python users familiar with SQL compared to those familiar with Haskell, I think we'd do wise to pick a different keyword instead of 'where'. I can't think of one right now though.
Taking a cue from mathematics, how about "given"?
c = sqrt(a*a + b*b) given: a = retrieve_a() b = retrieve_b()
Or if we'd rather overload an existing keyword than add a new one, "with" reads well too.
George _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
Alex Light wrote:
i would suggest overloading the 'with', 'as' combo
c = sqrt(a*a + b*b) with: get_a(), get_b() as a, b
or
c = sqrt(a*a + b*b) with: get_a() as a get_b() as b
Why use 'as'? Why not: c = sqrt(a*a + b*b) with: a = get_a() b = get_b() which is like: def _(): a = get_a() b = get_b() return sqrt(a*a + b*b) c = _() del _
it reads well and plus it follows since this statement acts similarly to a regular with and as statement with the behavior of the context manager being (in psudocode): set up: store original value of variable if any and set variable to new value. tear down: set value back to original or delete from local namespace if it never had one
additionally we do not need to introduce any new keywords
any way that this is implemented though it would be quite useful
On Tue, Jul 20, 2010 at 2:35 PM, George Sakkis <george.sakkis@gmail.com <mailto:george.sakkis@gmail.com>> wrote:
On Tue, Jul 20, 2010 at 8:29 PM, Daniel Stutzbach <daniel@stutzbachenterprises.com <mailto:daniel@stutzbachenterprises.com>> wrote:
> On Tue, Jul 20, 2010 at 1:16 PM, Guido van Rossum <guido@python.org <mailto:guido@python.org>> wrote: >> >> Given the large number of Python users familiar with SQL compared to >> those familiar with Haskell, I think we'd do wise to pick a different >> keyword instead of 'where'. I can't think of one right now though. > > Taking a cue from mathematics, how about "given"? > > c = sqrt(a*a + b*b) given: > a = retrieve_a() > b = retrieve_b()
Or if we'd rather overload an existing keyword than add a new one, "with" reads well too.
Den 20.07.2010 20:56, skrev Alex Light:
i would suggest overloading the 'with', 'as' combo
c = sqrt(a*a + b*b) with: get_a(), get_b() as a, b
That will not work, the parser would think like this: c = sqrt(a*a + b*b) with: get_a(), (get_b() as a), b
c = sqrt(a*a + b*b) with: get_a() as a get_b() as b
I think it tastes too much like functional programming. Specifically: It forces you to think backwards: it does not evaluate the lines in the order they read. The block below is evaluated before the expression above. It really means: a = get_a() b = get_b() c = sqrt(a*a + b*b) with: del a,b That I think is very annoying. Why not just use context managers instead? with get_a() as a, get_b() as b: c = sqrt(a*a + b*b) Now it reads the right way: top down, not bottom up. Regards, Sturla
On Tue, Jul 20, 2010 at 8:32 PM, Sturla Molden <sturla@molden.no> wrote:
Den 20.07.2010 20:56, skrev Alex Light:
i would suggest overloading the 'with', 'as' combo
c = sqrt(a*a + b*b) with: get_a(), get_b() as a, b
That will not work, the parser would think like this:
c = sqrt(a*a + b*b) with: get_a(), (get_b() as a), b
Not true, we can define the grouping as we like. However if we do something like this I'd rather use 'var = expr' instead of 'expr as var'.
c = sqrt(a*a + b*b) with: get_a() as a get_b() as b
I think it tastes too much like functional programming. Specifically:
It forces you to think backwards: it does not evaluate the lines in the order they read. The block below is evaluated before the expression above. It really means:
a = get_a() b = get_b() c = sqrt(a*a + b*b) with: del a,b
That I think is very annoying.
Like it or not, except for the keyword to use and the 'as' issue, that's exactly the proposal (please read the PEP: http://www.python.org/dev/peps/pep-3150/ ). I personally like it; plus the "think backwards" idea is already used in other parts of Python' syntax, e.g. list comprehensions. And of course forward references work for method calls too.
Why not just use context managers instead?
with get_a() as a, get_b() as b: c = sqrt(a*a + b*b)
Now it reads the right way: top down, not bottom up.
Because this means something different and requires that get_a() return something that obeys the context manager protocol. -- --Guido van Rossum (python.org/~guido)
Den 20.07.2010 20:16, skrev Guido van Rossum:
It looks like NumPy's "where" is more like SQL's,
Yes, it is roughtly like a WHERE statement in SQL or Fortran 90, or Python's built-in "filter" function (albeit more flexible). I am not sure I miss "fancy indexing" for Python lists because it is useful, or because Fortran have crippled my mind. This is of course easy to achieve with two utility functions, also demonstrating what NumPy's fancy indexing does. This is with a lambda: def where(cond, alist): return [i for i,a in enumerate(alist) if cond(a)] def fancyindex(alist, index): return [alist[i] for i in index]
Microsoft's LINQ is also similar to your suggestion.
LINQ is just to compensate for lack of duck-typing in C# ;-) Also when used to Pythons list comprehensions, LINQ syntax feels like thinking backwards (which can be quite annoying) :-( Sturla
Den 20.07.2010 20:16, skrev Guido van Rossum:
Yes, it is roughtly like a WHERE statement in SQL or Fortran 90, or Python's built-in "filter" function (albeit more flexible).
Fancy indexing is actually more like a join. Since the result from a numpy.where one array can be used to filter another array, it fancy indexing would be like a join between tables in a relational database. def join(blist, index): return [blist[i] for i in index] The big difference between fancy indexing and a join method is of course that indexing can appear on the left side of an expression. Sturla
On Tue, Jul 20, 2010 at 2:20 PM, Sturla Molden <sturla@molden.no> wrote:
Microsoft's LINQ is also similar to your suggestion.
LINQ is just to compensate for lack of duck-typing in C# ;-) Also when used to Pythons list comprehensions, LINQ syntax feels like thinking backwards (which can be quite annoying) :-(
Well, LINQ has a bit more going on from what I can tell -- you can actually get at the expressions and work with them. This is something NumPy and database abstraction layers both need, and they both currently use method override hacks that have certain limitations (e.g., you capture ==, but you can't capture "and"). If you work really hard you can decompile the bytecodes (DejaVu did this for lambdas, but not generator expressions). I don't think I've even seen a language proposal that actually tries to tackle this though. -- Ian Bicking | http://blog.ianbicking.org
List comprehensions are the one place where I might find this useful. I can do this with map though: [y for x in another_list if y < 5 where y = f(x)] [y for x in map(f, another_list) if x < 5] [x for x in another_list if y < 5 where y = f(x)] [x for (x,y) in map(lambda x: (x,f(x)), another_list) if y < 5] I think the variant with 'where' or 'with' would be a bit more readable but is it valuable enough? --- Bruce http://www.vroospeak.com http://google-gruyere.appspot.com On Tue, Jul 20, 2010 at 7:57 AM, Andrey Popp <8mayday@gmail.com> wrote:
mylist = [y for y in another_list if y < 5 where y = f(x)]
[x for x in another_list if y < 5 where y = f(x)] [x for (x,y) in map(lambda x: (x,f(x)), another_list) if y < 5]
Or allow nested list comprehensions? [x for (x,y) in [x,f(x) for x in another_list] if y < 5] this is getting closer to LINQ... We could use line-breaks for readability: [x for (x,y) in [x,f(x) for x in another_list] if y < 5] S.
On Wed, Jul 21, 2010 at 12:57 AM, Andrey Popp <8mayday@gmail.com> wrote:
Hello,
PEP-3150 is awesome, just a small addition — why not to allow one-liners `where`s:
a = (b, b) where b = 43
And that also make sense for generator/list/set/dict comprehensions:
mylist = [y for y in another_list if y < 5 where y = f(x)]
As with any other suite, a one-liner would be allowed on the same line as the colon: a = (b, b) where: b = call_once_only() mylist = [y for y in another_list if y < x] where: x = call_once_only() Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan writes:
On Tue, Jul 20, 2010 at 3:29 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
For the record, I am personally +1 on the idea
Gee, and here I had you down as a +0.95. Can you forgive me? :-)
However, having the question come up twice within the last month finally inspired me to write the current status of the topic down in a deferred PEP: http://www.python.org/dev/peps/pep-3150/
Way cool! Thanks!
n Tue, Jul 20, 2010 at 8:27 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
For the record, I am personally +1 on the idea (otherwise I wouldn't have put so much thought into it over the years). It's just a *lot* harder to define complete and consistent semantics for the concept than people often realise.
However, having the question come up twice within the last month finally inspired me to write the current status of the topic down in a deferred PEP: http://www.python.org/dev/peps/pep-3150/
There was a related discussion on python-ideas in July 2009, spanning two threads, that you may want to additionally reference. A lot of corner cases were also brought up in that thread. Here are the starts of the two threads: http://mail.python.org/pipermail/python-ideas/2009-July/005082.html http://mail.python.org/pipermail/python-ideas/2009-July/005132.html -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
On Tue, 20 Jul 2010 23:27:49 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
For the record, I am personally +1 on the idea (otherwise I wouldn't have put so much thought into it over the years). It's just a *lot* harder to define complete and consistent semantics for the concept than people often realise.
However, having the question come up twice within the last month finally inspired me to write the current status of the topic down in a deferred PEP: http://www.python.org/dev/peps/pep-3150/
I am worried that this complexifies Python syntax without any obvious benefit in terms of expressive power, new abstractions, or concision. There is a benefit (learning curve, readibility of foreign code) to a simple syntax. I am somewhere between -0.5 and -1. Regards Antoine.
On Wed, 21 Jul 2010 00:28:03 +0200 Antoine Pitrou <solipsis@pitrou.net> wrote:
I am worried that this complexifies Python syntax without any obvious benefit in terms of expressive power, new abstractions, or concision. There is a benefit (learning curve, readibility of foreign code) to a simple syntax.
I'll add another issue: - currently, lexical blocks (indentation following a colon) are used for control flow statements; this proposal blurs the line and makes visual inspection less reliable I also disagree with the rationale which states that the motivation is similar to that for decorators or list comprehensions. Decorators and list comprehensions add value by making certain constructs more concise and more readable (by allowing to express the construct at a higher level through the use of detail-hiding syntax); as for decorators, they also eliminate the need for repeating oneself. Both have the double benefit of allowing shorter and higher-level code. The "given" syntax (I don't know how to call it: statement? postfix? appendage?), however, brings none of these benefits: it is almost pure syntactic sugar, and one which doesn't bring any lexical compression since it actually increases code size, rather than decrease it. Regards Antoine.
On Tue, Jul 20, 2010 at 11:48 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 21 Jul 2010 00:28:03 +0200 Antoine Pitrou <solipsis@pitrou.net> wrote:
I am worried that this complexifies Python syntax without any obvious benefit in terms of expressive power, new abstractions, or concision. There is a benefit (learning curve, readibility of foreign code) to a simple syntax.
I'll add another issue:
- currently, lexical blocks (indentation following a colon) are used for control flow statements; this proposal blurs the line and makes visual inspection less reliable
This is indeed a bit of a downside; if you see blah blah blah: x = blah y = blah you will have to look more carefully at the end of the first blah blah blah line to know whether the indented block is executed first or last. For all other intended blocks, the *beginning* of the indented block is your clue (class, def, if, try, etc.).
I also disagree with the rationale which states that the motivation is similar to that for decorators or list comprehensions. Decorators and list comprehensions add value by making certain constructs more concise and more readable (by allowing to express the construct at a higher level through the use of detail-hiding syntax); as for decorators, they also eliminate the need for repeating oneself. Both have the double benefit of allowing shorter and higher-level code.
I see a similar possibility as for decorators, actually. A decorator is very simple syntactic sugar too, but it allows one to emphasize the decoration by putting it up front rather than hiding it after the (possibly very long) function. The 'given' block has a similar effect of changing the order in which the (human) reader encounters things: it lets you see the important part first, e.g. c = sqrt(a*a + b*b) and put the definitions for a and b off till later. This can both help the author "stay in the flow" and emphasize the most important parts for the reader, similar to top-down programming using forward references between functions or methods. I personally use top-down just about evenly with bottom-up (though in different circumstances), and I think it would be useful to have more support for top-down coding at the statement level. That's why ABC had refinements, too. I dropped them from Python because I had to cut down the design as much as possible to make it possible to implement in a reasonable time as a skunkworks project. But I always did like them in ABC. Whether this is enough to compensate for the larger grammar is an open question.
The "given" syntax (I don't know how to call it: statement? postfix? appendage?), however, brings none of these benefits: it is almost pure syntactic sugar, and one which doesn't bring any lexical compression since it actually increases code size, rather than decrease it.
But it decreases exposure by limiting the scope of the variables defined in the 'given' block, just like generator expressions and (in Python 3) list comprehensions do. In larger functions it is easy to accidentally reuse variables and this occasionally introduces bugs (the most common case probably being the loop control variable for a small (often inner) loop overwriting a variable that is set far above the loop in the same scope but is used far below it). -- --Guido van Rossum (python.org/~guido)
Questions: 1.) It looks like a lot of the complexity of PEP 3150 is based on wanting thing like this to work: x[index] = 42 given: index = complicated_formula To make that work, you need to figure out if index is a nonlocal or a global or what in order to emit the right bytecode. What happens if we just give up that use case and say that anything on the assignment side of the initial = gets looked up in the original namespace? In other words, make 3150 more similar to the sugar: def _(): index = complicated_formula x[index] = _() #Probably a NameError Would the complexity of PEP 3150 be significantly lessened by that? Or are there other major sources of complexity in the local/nonlocal/global issue? 2.) What happens in this case: x = y given: return "???" Do we just disallow return inside a given? If so, how would the parser know to allow you to do a def inside a given? -- Carl Johnson
Carl M. johnson wrote:
2.) What happens in this case:
x = y given: return "???"
Do we just disallow return inside a given? If so, how would the parser know to allow you to do a def inside a given? i think so because unless i am misunderstanding something the only allowed expressions in a 'given' block would be of the type:
a_variable = an_expression however one way to think of it is that ans = do_something(a, b, c) given: a = get_a() b = get_b() c = get_c() is the same as saying a = get_a() b = get_b() c = get_c() ans = do_something(a, b, c) del a, b, c which would seem to indicate that a = do_something() given: return "???" goes to: return "???" a = do_something() would be valid (if extremely bad style) in the end i think its use in that way would be like the use of a GOTO statement in many languages, technically there is no reason it should not be allowed but still prohibited for stylistic reasons. anyway why would you ever want to do this? it would be like writing def f(): return return it is non-nonsensical and obviously an error On Tue, Jul 20, 2010 at 8:09 PM, Carl M. Johnson < cmjohnson.mailinglist@gmail.com> wrote:
Questions:
1.) It looks like a lot of the complexity of PEP 3150 is based on wanting thing like this to work:
x[index] = 42 given: index = complicated_formula
To make that work, you need to figure out if index is a nonlocal or a global or what in order to emit the right bytecode. What happens if we just give up that use case and say that anything on the assignment side of the initial = gets looked up in the original namespace? In other words, make 3150 more similar to the sugar:
def _(): index = complicated_formula
x[index] = _() #Probably a NameError
Would the complexity of PEP 3150 be significantly lessened by that? Or are there other major sources of complexity in the local/nonlocal/global issue?
2.) What happens in this case:
x = y given: return "???"
Do we just disallow return inside a given? If so, how would the parser know to allow you to do a def inside a given?
-- Carl Johnson _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On Tue, Jul 20, 2010 at 7:50 PM, Alex Light <scialexlight@gmail.com> wrote:
Carl M. johnson wrote:
2.) What happens in this case:
x = y given: return "???"
Do we just disallow return inside a given? If so, how would the parser know to allow you to do a def inside a given? i think so because unless i am misunderstanding something the only allowed expressions in a 'given' block would be of the type: a_variable = an_expression
Incorrect. Yes, you are misunderstanding: On Tue, Jul 20, 2010 at 3:13 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Jul 21, 2010 at 6:13 AM, Alex Light <scialexlight@gmail.com> wrote:
i would use as because this whole where clause acts very similarly to a context manager in that it sets a variable to a value for a small block
No, the idea is for the indented suite to be a perfectly normal suite of Python code. We want to be able to define functions, classes, etc in there. Inventing a new mini-language specifically for these clauses would be a bad idea (and make them unnecessarily hard to understand) <snip> Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Did you not read Nick's reply yet when you wrote this, or...? Cheers, Chris -- http://blog.rebertia.com
nick coughlan wrote:
No, the idea is for the indented suite to be a perfectly normal suite of Python code. We want to be able to define functions, classes, etc in there. @Chris Robert sorry what i meant in saying that " a_variable = an_expression" is that, it seems to me, at least, the only allowed statements are ones where a variable is set to a value, which includes "class" and "def" (and some control flow, if, else etc.)
also in the first post: Sergio Davis wrote:
I'm considering the following extension to Python's grammar: adding the 'where' keyword, which would work as follows:
where_expr : expr 'where' NAME '=' expr
On Tue, Jul 20, 2010 at 10:56 PM, Chris Rebert <pyideas@rebertia.com> wrote:
On Tue, Jul 20, 2010 at 7:50 PM, Alex Light <scialexlight@gmail.com> wrote:
Carl M. johnson wrote:
2.) What happens in this case:
x = y given: return "???"
Do we just disallow return inside a given? If so, how would the parser know to allow you to do a def inside a given? i think so because unless i am misunderstanding something the only allowed expressions in a 'given' block would be of the type: a_variable = an_expression
Incorrect. Yes, you are misunderstanding:
On Tue, Jul 20, 2010 at 3:13 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Jul 21, 2010 at 6:13 AM, Alex Light <scialexlight@gmail.com> wrote:
i would use as because this whole where clause acts very similarly to a context manager in that it sets a variable to a value for a small block
No, the idea is for the indented suite to be a perfectly normal suite of Python code. We want to be able to define functions, classes, etc in there. Inventing a new mini-language specifically for these clauses would be a bad idea (and make them unnecessarily hard to understand) <snip> Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Did you not read Nick's reply yet when you wrote this, or...?
Cheers, Chris -- http://blog.rebertia.com
On Wed, Jul 21, 2010 at 12:56 PM, Chris Rebert <pyideas@rebertia.com> wrote:
On Tue, Jul 20, 2010 at 3:13 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Jul 21, 2010 at 6:13 AM, Alex Light <scialexlight@gmail.com> wrote:
i would use as because this whole where clause acts very similarly to a context manager in that it sets a variable to a value for a small block
No, the idea is for the indented suite to be a perfectly normal suite of Python code. We want to be able to define functions, classes, etc in there. Inventing a new mini-language specifically for these clauses would be a bad idea (and make them unnecessarily hard to understand) <snip> Did you not read Nick's reply yet when you wrote this, or...?
Alex actually has a reasonable point here: break, continue, yield and return actually don't make sense in the top-level of the given clause (since it is conceptually all one statement). For break and continue, they will naturally give a SyntaxError with the proposed implementation (for "'break' outside loop" or "'continue' not properly in loop", just to be randomly inconsistent). yield and return (at the level of the given clause itself) will need to be disallowed explicitly by the compiler (similar to the "'return' outside function" and "'yield' outside function" errors you get if you attempt to use these keywords in a class or module scope). There are also some subtleties as to whether the given clause is compiled as a closure or not (my current thoughts are that it should be compiled as a closure when defined in a function scope, but like a class scope when defined in a class or module scope). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Jul 21, 2010 at 7:20 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Jul 21, 2010 at 12:56 PM, Chris Rebert <pyideas@rebertia.com> wrote:
On Tue, Jul 20, 2010 at 3:13 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Jul 21, 2010 at 6:13 AM, Alex Light <scialexlight@gmail.com> wrote:
i would use as because this whole where clause acts very similarly to a context manager in that it sets a variable to a value for a small block
No, the idea is for the indented suite to be a perfectly normal suite of Python code. We want to be able to define functions, classes, etc in there. Inventing a new mini-language specifically for these clauses would be a bad idea (and make them unnecessarily hard to understand) <snip> Did you not read Nick's reply yet when you wrote this, or...?
Alex actually has a reasonable point here: break, continue, yield and return actually don't make sense in the top-level of the given clause (since it is conceptually all one statement).
For break and continue, they will naturally give a SyntaxError with the proposed implementation (for "'break' outside loop" or "'continue' not properly in loop", just to be randomly inconsistent).
yield and return (at the level of the given clause itself) will need to be disallowed explicitly by the compiler (similar to the "'return' outside function" and "'yield' outside function" errors you get if you attempt to use these keywords in a class or module scope).
I'm -sys.maxint on the PEP for a many reasons. 1) I don't want to have to explain this to people. "It's just like regular python but you can't read it top-to-bottom and you can't include control flow statements." 2a) No control flow statements in the block means if you need to augment the code to do a return/break/continue/yield you then have to refactor so everything in the "given:" block gets moved to the top and a 1-line change becomes a 10 line diff. 2b) Allowing control flow statements in the block would be even more confusing. 2c) Is this legal? x = b given: b = 0 for item in range(100): b += item if b > 10: break 3) I really don't want to have to explain to people why that is, or isn't valid. 4) decorators and "with" blocks read top-to-bottom even if they change the emphasis. This doesn't. 5) There are no compelling use cases. The two examples in the PEP are toys. -Jack
On Wed, Jul 21, 2010 at 12:02 PM, Jack Diederich <jackdied@gmail.com> wrote:
<snip>
2a) No control flow statements in the block means if you need to
augment the code to do a return/break/continue/yield you then have to refactor so everything in the "given:" block gets moved to the top and a 1-line change becomes a 10 line diff. 2b) Allowing control flow statements in the block would be even more confusing. 2c) Is this legal? x = b given: b = 0 for item in range(100): b += item if b > 10: break
2a) you are missing the point of the given clause. you use it to assign values to variables if, and only if, the only possible results of the computation are 1) an exception is raised or 2) a value is returned which is set to the variable and used in the expression no matter its value. if there is the slightest chance that what you describe might be necessary you would not put it in a "given" but do something like this: (assumes that a given is applied to the previous statement in its current block) if some_bool_func(a): ans = some_func1(a, b, c) else: ans = some_func2(a, b, c) given: # note: given applied to the block starting with the "if" statement a = get_a() b = get_b() c = get_c() 2b) agreed but there is no reason for them to be disallowed except readibility but they should be discouraged 2c) see it might be okay, depending on what people think of your second question. in my opinion it should be illegal stylistically and be required to be changed to def summation(start, end) i = start while i < end: start += i i += 1 return start def a_func(): # do stuff x = b given: b = summation(0, 100)
if some_bool_func(a): ans = some_func1(a, b, c) else: ans = some_func2(a, b, c) given: # note: given applied to the block starting with the "if" statement a = get_a() b = get_b() c = get_c()
It seems to me that the postfix construct would confuse the reader in a scenario such as the above, rather than reducing complexity. (Worse, suppose 'a', 'b', and 'c' had been assigned prior to the 'if' statement!) Moreover, our intuitions seem to be fuzzy when it comes to the desirability/behavior of control flow statements in the 'given' block. So I'm -1 on this: I think there is a marked increase in the opportunity for confusion among authors and readers, without a clear set of common patterns that would be improved (even if readability is slightly better on occasion). I think a better alternative to allow for safe localization of variables to a block would be to adapt the 'with' statement to behave as a 'let' (similar to suggestions earlier in the thread). For instance: with fname = sys.argv[1], open(fname) as f, contents = f.read(): do_stuff1(fname, contents) do_stuff2(contents) do_stuff3(fname) # error: out of scope This makes it clear to the reader that the assignments to 'fname' and 'contents', like 'f', only pertain to the contents of the 'with' block. It allows the reader to focus their eye on the 'important' part—the part inside the block—even though it doesn't come first. It helps avoid bugs that might arise if 'fname' were used later on. And it leaves no question as to where control flow statements are permitted/desirable. I'm +0.5 on this alternative: my hesitation is because we'd need to explain to newcomers why 'f = open(fname)' would be legal but bad, owing to the subtleties of context managers. (I worry Alex's proposal for the interpreter to create context managers on the fly would only add confusion.) Nathan On Wed, Jul 21, 2010 at 1:19 PM, Alex Light <scialexlight@gmail.com> wrote:
On Wed, Jul 21, 2010 at 12:02 PM, Jack Diederich <jackdied@gmail.com> wrote:
<snip>
2a) No control flow statements in the block means if you need to augment the code to do a return/break/continue/yield you then have to refactor so everything in the "given:" block gets moved to the top and a 1-line change becomes a 10 line diff. 2b) Allowing control flow statements in the block would be even more confusing. 2c) Is this legal? x = b given: b = 0 for item in range(100): b += item if b > 10: break
2a) you are missing the point of the given clause. you use it to assign values to variables if, and only if, the only possible results of the computation are 1) an exception is raised or 2) a value is returned which is set to the variable and used in the expression no matter its value. if there is the slightest chance that what you describe might be necessary you would not put it in a "given" but do something like this: (assumes that a given is applied to the previous statement in its current block) if some_bool_func(a): ans = some_func1(a, b, c) else: ans = some_func2(a, b, c) given: # note: given applied to the block starting with the "if" statement a = get_a() b = get_b() c = get_c() 2b) agreed but there is no reason for them to be disallowed except readibility but they should be discouraged 2c) see it might be okay, depending on what people think of your second question. in my opinion it should be illegal stylistically and be required to be changed to def summation(start, end) i = start while i < end: start += i i += 1 return start def a_func(): # do stuff x = b given: b = summation(0, 100)
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On Thu, Jul 22, 2010 at 5:55 AM, Nathan Schneider <nathan@cmu.edu> wrote:
I think a better alternative to allow for safe localization of variables to a block would be to adapt the 'with' statement to behave as a 'let' (similar to suggestions earlier in the thread). For instance:
with fname = sys.argv[1], open(fname) as f, contents = f.read(): do_stuff1(fname, contents) do_stuff2(contents) do_stuff3(fname) # error: out of scope
This makes it clear to the reader that the assignments to 'fname' and 'contents', like 'f', only pertain to the contents of the 'with' block. It allows the reader to focus their eye on the 'important' part—the part inside the block—even though it doesn't come first. It helps avoid bugs that might arise if 'fname' were used later on. And it leaves no question as to where control flow statements are permitted/desirable.
I'm +0.5 on this alternative: my hesitation is because we'd need to explain to newcomers why 'f = open(fname)' would be legal but bad, owing to the subtleties of context managers.
Hmm, an intriguing idea. I agree that the subtleties of "=" vs "as" could lead to problems though. There's also the fact that existing semantics mean that 'f' has to remain bound after the block, so it would be surprising if 'fname' and 'contents' were unavailable. It also suffers the same issue as any of the other in-order proposals: without out-of-order execution, the only gain is a reduction in the chance for namespace collisions, and that's usually only a problem for accidental collisions with loop variable names in long functions or scripts. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 7/21/2010 7:20 AM, Nick Coghlan wrote:
yield and return (at the level of the given clause itself) will need to be disallowed explicitly by the compiler
Why introduce an inconsistency? If a = e1 b = f(a) can be flipped to b = f(a) given: a = e1 I would expect a = e1 return f(a) to be flippable to return f(a) given a = e1 -- Terry Jan Reedy
On Wed, Jul 21, 2010 at 7:12 PM, Terry Reedy <tjreedy@udel.edu> wrote:
On 7/21/2010 7:20 AM, Nick Coghlan wrote:
yield and return (at the level of the given clause itself) will need to be disallowed explicitly by the compiler
Why introduce an inconsistency?
If
a = e1 b = f(a)
can be flipped to
b = f(a) given: a = e1
I would expect
a = e1 return f(a)
to be flippable to
return f(a) given a = e1
I believe Nick meant returns/yields *within* the `given` suite (he just phrased it awkwardly), e.g. a = b given: b = 42 return c # WTF The PEP's Syntax Change section explicitly changes the grammar to allow the sort of `given`s you're talking about. Cheers, Chris
There have been questions about whether there are any cases of the given/where/let/whatever solving problems that would otherwise be cumbersome to solve. I think it could help get around certain for-loop gotchas:
funcs = [] for i in range(5): ... def f(): ... print("#", i) ... funcs.append(f) ... [func() for func in funcs] # 4 # 4 # 4 # 4 # 4 [None, None, None, None, None]
D’oh! (This can be a real world problem if you have a list of methods you want to decorate inside a class.) One current workaround:
funcs = [] for i in range(5): ... def _(): ... n = i ... def f(): ... print("#", n) ... funcs.append(f) ... _() ... [func() for func in funcs] # 0 # 1 # 2 # 3 # 4 [None, None, None, None, None]
Not pretty, but it works. In let format (I’m leaning toward the format “let [VAR = | return | yield] EXPRESSION where: BLOCK”): funcs = [] for i in range(5): let funcs.append(f) where: n = i def f(): print("#", n) [func() for func in funcs] Still a little awkward, but not as bad, IMHO. -- Carl Johnson
On Wed, Jul 21, 2010 at 9:07 PM, Carl M. Johnson <cmjohnson.mailinglist@gmail.com> wrote:
There have been questions about whether there are any cases of the given/where/let/whatever solving problems that would otherwise be cumbersome to solve. I think it could help get around certain for-loop gotchas:
funcs = [] for i in range(5): ... def f(): ... print("#", i) ... funcs.append(f) ... [func() for func in funcs] # 4 # 4 # 4 # 4 # 4 [None, None, None, None, None]
D’oh! (This can be a real world problem if you have a list of methods you want to decorate inside a class.)
One current workaround:
funcs = [] for i in range(5): ... def _(): ... n = i ... def f(): ... print("#", n) ... funcs.append(f) ... _() ... [func() for func in funcs] # 0 # 1 # 2 # 3 # 4 [None, None, None, None, None]
Not pretty, but it works.
In let format (I’m leaning toward the format “let [VAR = | return | yield] EXPRESSION where: BLOCK”):
funcs = [] for i in range(5): let funcs.append(f) where: n = i def f(): print("#", n)
[func() for func in funcs]
Still a little awkward, but not as bad, IMHO.
Neither of those seem (imo) better than the other current workaround: funcs = [] for i in range(5): def f(i=i): print("#", i) funcs.append(f) They're all non-obvious idioms, but at least this one is short and easily recognized. Cheers, Chris
On Wed, Jul 21, 2010 at 6:34 PM, Chris Rebert <pyideas@rebertia.com> wrote:
funcs = [] for i in range(5): def f(i=i): print("#", i) funcs.append(f)
They're all non-obvious idioms, but at least this one is short and easily recognized.
I actually had to read that twice before I recognized what you had done, and I knew to look for something out of the ordinary. That said, it *is* the solution GvR recommended as the best solution the last time this came up. I just never understood why. To me, if you set something as a default value for an argument, it should be because it’s a default value, ie. a value that is usually one thing but can also be set to something else at the caller’s discretion. I’m just not comfortable with using the default value to mean “here’s a value you should never change” or “pretty please, don’t pass in an argument, because that will screw everything up” or even “I guess you could pass in an argument if you wanted to, but that’s not a case I’m really very busy thinking about". :-/ But maybe I’m in the minority on this one. -- CJ
On Thu, Jul 22, 2010 at 5:48 PM, Carl M. Johnson <cmjohnson.mailinglist@gmail.com> wrote:
On Wed, Jul 21, 2010 at 6:34 PM, Chris Rebert <pyideas@rebertia.com> wrote:
funcs = [] for i in range(5): def f(i=i): print("#", i) funcs.append(f)
They're all non-obvious idioms, but at least this one is short and easily recognized.
I actually had to read that twice before I recognized what you had done, and I knew to look for something out of the ordinary. That said, it *is* the solution GvR recommended as the best solution the last time this came up. I just never understood why. To me, if you set something as a default value for an argument, it should be because it’s a default value, ie. a value that is usually one thing but can also be set to something else at the caller’s discretion. I’m just not comfortable with using the default value to mean “here’s a value you should never change” or “pretty please, don’t pass in an argument, because that will screw everything up” or even “I guess you could pass in an argument if you wanted to, but that’s not a case I’m really very busy thinking about". :-/ But maybe I’m in the minority on this one.
There's a reason that particular trick is called the "default argument hack" :) The trick is much easier to follow when you *don't* reuse the variable name in the inner function: funcs = [] for i in range(5): def f(early_bound_i=i): print("#", early_bound_i) funcs.append(f) Python doesn't really have a good way to request early binding semantics at this time - the default argument hack is about it. The given clause (as currently specified in the PEP) forces early binding semantics on any functions it contains, so it allows you to write a function definition loop that "does the right thing" with the following fairly straightforward code: funcs = [] for i in range(5): funcs.append(f) given: def f(): print("#", i) That's: a) kinda cool b) veering dangerously close to DWIM* territory (which is not necessarily a good thing) Still, the early binding semantics angle is one I had thought about before - that *is* a genuinely new feature of this proposal. Perhaps not a hugely compelling one though - I think I've only needed to use the default argument hack once in the whole time I've been programming Python. Cheers, Nick. *Don't Worry It's Magic -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Jul 22, 2010 at 10:49 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Still, the early binding semantics angle is one I had thought about before - that *is* a genuinely new feature of this proposal. Perhaps not a hugely compelling one though - I think I've only needed to use the default argument hack once in the whole time I've been programming Python.
Uhh, "... had*n't* thought about ...". 3 little missing characters that completely reverse the intended meaning of a sentence :P Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Jul 22, 2010 at 8:49 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I think I've only needed to use the default argument hack once in the whole time I've been programming Python.
Here is your second: http://bugs.python.org/issue7989#msg109662 or scroll to the end of http://mail.python.org/pipermail/python-dev/2010-July/101900.html
On Thu, Jul 22, 2010 at 1:30 PM, Chris Rebert <pyideas@rebertia.com> wrote:
I believe Nick meant returns/yields *within* the `given` suite (he just phrased it awkwardly), e.g.
a = b given: b = 42 return c # WTF
Yep, that's what I meant by the top level code in the given clause (i.e. in the same sense as I would refer to the top level code in a function body). It didn't occur to me that phrasing could be legitimately confused with the header line of the clause until I read Terry's message. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 2010-07-21, at 02:09 , Carl M. Johnson wrote:
Questions:
1.) It looks like a lot of the complexity of PEP 3150 is based on wanting thing like this to work:
x[index] = 42 given: index = complicated_formula
To make that work, you need to figure out if index is a nonlocal or a global or what in order to emit the right bytecode. What happens if we just give up that use case and say that anything on the assignment side of the initial = gets looked up in the original namespace? In other words, make 3150 more similar to the sugar:
I quite agree with that, the where/given block/scope should only apply to the expression directly to the left of it. So only the RHS should be concerned, and LHS is out of that scope. And that expression would be written as: operator.setitem(x, index, 42) given: index = complicated_formula I think the first Torture Test block is misguided, and I'd be −0.5 on such a complex feature.
On Wed, Jul 21, 2010 at 11:39 AM, Masklinn <masklinn@masklinn.net> wrote:
On 2010-07-21, at 02:09 , Carl M. Johnson wrote:
Questions:
1.) It looks like a lot of the complexity of PEP 3150 is based on wanting thing like this to work:
x[index] = 42 given: index = complicated_formula
To make that work, you need to figure out if index is a nonlocal or a global or what in order to emit the right bytecode. What happens if we just give up that use case and say that anything on the assignment side of the initial = gets looked up in the original namespace? In other words, make 3150 more similar to the sugar:
Why do you think it's more complicated to do it for the LHS than for the RHS?
I quite agree with that, the where/given block/scope should only apply to the expression directly to the left of it. So only the RHS should be concerned, and LHS is out of that scope.
And that expression would be written as:
operator.setitem(x, index, 42) given: index = complicated_formula
Bah.
I think the first Torture Test block is misguided, and I'd be -0.5 on such a complex feature.
-- --Guido van Rossum (python.org/~guido)
On Wed, Jul 21, 2010 at 8:53 PM, Guido van Rossum <guido@python.org> wrote:
On Wed, Jul 21, 2010 at 11:39 AM, Masklinn <masklinn@masklinn.net> wrote:
On 2010-07-21, at 02:09 , Carl M. Johnson wrote:
Questions:
1.) It looks like a lot of the complexity of PEP 3150 is based on wanting thing like this to work:
x[index] = 42 given: index = complicated_formula
To make that work, you need to figure out if index is a nonlocal or a global or what in order to emit the right bytecode. What happens if we just give up that use case and say that anything on the assignment side of the initial = gets looked up in the original namespace? In other words, make 3150 more similar to the sugar:
Why do you think it's more complicated to do it for the LHS than for the RHS?
It's allowing more than simple names on the LHS that makes a naive return-based approach to name binding not work. However, it's class scopes that really make this complicated, and making it "just work" is probably easier than trying to explain which kinds of assignment will and won't work. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Jul 21, 2010 at 1:07 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
However, it's class scopes that really make this complicated, and making it "just work" is probably easier than trying to explain which kinds of assignment will and won't work.
Is support for class scopes an absolute must-have ? While it might be nice to have for consistency's sake, practically speaking module and function scopes should cover all but the most obscure and perverse use cases. I'm on -0.1 at the moment because of the "Two Ways To Do It" objection but I don't think the complication of making it work for class scopes should be grounds for rejection. George
I'm unconvinced of the value at this point but notwithstanding that let me toss in an alternative syntax: given: suite do: suite This executes the two suites in order with any variable bindings created by the first suite being local to the scope of the two suites. I think this is more readable than the trailing clause and is more flexible (you can put multiple statements in the second suite) and avoids the issue with anyone wanting the where clause added to arbitrary expressions. FWIW, in math it's more common to list givens at the top. --- Bruce (via android)
On Thu, Jul 22, 2010 at 1:51 AM, Bruce Leban <bruce@leapyear.org> wrote:
I'm unconvinced of the value at this point but notwithstanding that let me toss in an alternative syntax:
given: suite do: suite
This executes the two suites in order with any variable bindings created by the first suite being local to the scope of the two suites. I think this is more readable than the trailing clause and is more flexible (you can put multiple statements in the second suite) and avoids the issue with anyone wanting the where clause added to arbitrary expressions.
FWIW, in math it's more common to list givens at the top.
However, writing it that way has even less to offer over ordinary local variables than the postfix given clause. I updated the draft PEP again, pointing out that if a decision had to be made today, the PEP would almost certainly be rejected due to a lack of compelling use cases. The bar for adding a new syntactic construct is pretty high and PEP 3150 currently isn't even close to reaching it (see PEP 343 for the kind of use cases that got the with statement over that bar). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Jul 21, 2010 at 2:58 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Thu, Jul 22, 2010 at 1:51 AM, Bruce Leban <bruce@leapyear.org> wrote:
let me toss in an alternative syntax: given: suite do: suite
However, writing it that way has even less to offer over ordinary local variables than the postfix given clause.
Perhaps. I do think it looks more like Python. The PEP says it's "some form of statement local namespace". If that's the advantage, this alternative offers it. If the advantage is that it introduces a different execution order, then the PEP should make the case for that. --- Bruce http://www.vroospeak.com http://google-gruyere.appspot.com
On Jul 21, 2010, at 2:58 PM, Nick Coghlan wrote:
On Thu, Jul 22, 2010 at 1:51 AM, Bruce Leban <bruce@leapyear.org> wrote:
I'm unconvinced of the value at this point but notwithstanding that let me toss in an alternative syntax:
given: suite do: suite
This executes the two suites in order with any variable bindings created by the first suite being local to the scope of the two suites. I think this is more readable than the trailing clause and is more flexible (you can put multiple statements in the second suite) and avoids the issue with anyone wanting the where clause added to arbitrary expressions.
FWIW, in math it's more common to list givens at the top.
Sounds like a LISP letrec.
However, writing it that way has even less to offer over ordinary local variables than the postfix given clause.
I updated the draft PEP again, pointing out that if a decision had to be made today, the PEP would almost certainly be rejected due to a lack of compelling use cases. The bar for adding a new syntactic construct is pretty high and PEP 3150 currently isn't even close to reaching it (see PEP 343 for the kind of use cases that got the with statement over that bar).
I concur. Raymond
On 7/20/2010 7:08 PM, Guido van Rossum wrote:
I see a similar possibility as for decorators, actually. A decorator is very simple syntactic sugar too, but it allows one to emphasize the decoration by putting it up front rather than hiding it after the (possibly very long) function.
Because you chose to put multiple decorators in 'nested' order, I can consistently read decorated definitions as multi-statement'expressions' like so: @f1(arg) #( @f2 # ( def f(): pass # )) with the function name propagated out. Reading expression inside out is nothing new. (The opposite decorator order would have made this 'trick' impossible and decorators harder for me to read.) I do not particularly see this syntax as 'emphasizing' the decoration any more than f(g(2*x+y**z) 'emphasizes' f or g over the operator expression. [changing the order]
This is indeed a bit of a downside; if you see
blah blah blah: x = blah y = blah
you will have to look more carefully at the end of the first blah blah blah line to know whether the indented block is executed first or last. For all other intended blocks, the *beginning* of the indented block is your clue (class, def, if, try, etc.).
Yes! This is a great feature of Python. For decorators, the *initial* '@' is the clue to shift reading mode. If givens were accepted, I would strongly prefer there to be a similar initial clue. On function definition order: I generally prefer to write and read definitions before they are used. (Mathematicians usually present and prove lemmas before a main theorem.) The major exception is the __init__ method of a class, which really is special as it defines instance attributes. I can imagine a module with one main and several helper functions starting with the main function, but there should first be a doc string listing everything with at least a short phrase of explanation. -- Terry Jan Reedy
Hello, On Wed, Jul 21, 2010 at 2:48 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
I'll add another issue:
- currently, lexical blocks (indentation following a colon) are used for control flow statements; this proposal blurs the line and makes visual inspection less reliable
Do class definitions or with-statements represent control flow structures? I think, no (with-statement maybe).
I also disagree with the rationale which states that the motivation is similar to that for decorators or list comprehensions. Decorators and list comprehensions add value by making certain constructs more concise and more readable (by allowing to express the construct at a higher level through the use of detail-hiding syntax); as for decorators, they also eliminate the need for repeating oneself. Both have the double benefit of allowing shorter and higher-level code.
Consider the following: ... value = a*x*x + b*x + c given: a = compute_a() b = compute_b() c = compute_c() ... which is roughly equivalent to ... a = compute_a() b = compute_b() c = compute_c() value = a*x*x + b*x + c ... with two differences: - It emphasizes that `value` is a target of this computation and `a`, `b` and `c` are just auxiliary. - It states that `a`, `b` and `c` are only used in statement, before the `given` keyword, that would help future refactorings. Due to the second point, it can't be considered as syntactic sugar. Is is more readable? I think yes. -- Andrey Popp phone: +7 911 740 24 91 e-mail: 8mayday@gmail.com
Andrey Popp writes:
Do class definitions or with-statements represent control flow structures? I think, no (with-statement maybe).
In both cases, yes. From the point of view of the programmer writing the controlled suite, they're very simple control structures (serial execution). True, the important thing that a class definition does is set up a special configuration of namespaces in which the suite is executed (eg, instead of def registering a function in the global namespace, it registers it in the class). Nevertheless, it does determine control flow, and the statements in a class "declaration" are executed at runtime, not at compile time. The with statement is a nontrivial control flow structure: it ensures that certain code is executed at certain times, although the code that it provides guarantees for is not in the explicit suite it controls. Antoine's point here is not that "given" isn't a control flow *structure*. It is that it is not a *statement*, but rather a fragment that can be added to a wide variety of statements. That is quite a major departure for Python, though I think it a natural one in this context. Antoine might find execute: value = a*x*x + b*x + c given: a = compute_a() b = compute_b() c = compute_c() less objectionable on those grounds. Ie, it is now a proper control statement. I hasten to add that I think this syntax is quite horrible for *other* reasons, not least needing to find two keywords. Worst, "given" advocates want the computation of a, b, and c subordinated lexically to computation of value, but here they are on the same level.
Consider the following:
... value = a*x*x + b*x + c given: a = compute_a() b = compute_b() c = compute_c() ...
which is roughly equivalent to
... a = compute_a() b = compute_b() c = compute_c() value = a*x*x + b*x + c ...
with two differences:
- It emphasizes that `value` is a target of this computation and `a`, `b` and `c` are just auxiliary. - It states that `a`, `b` and `c` are only used in statement, before the `given` keyword, that would help future refactorings.
Due to the second point, it can't be considered as syntactic sugar.
But the second point doesn't prove you can't get the same semantics, only that a naive implementation fails. If you want to ensure that `a`, `b` and `c` are only used in a limited scope, there's always def compute_quadratic(z): a = compute_a() b = compute_b() c = compute_c() return a*x*x + b*x + c value = compute_quadratic(x) del compute_quadratic Now *that* is quite ugly (and arguably unreadable). But it shows that there are other ways of obtaining the same semantics in Python, and thus "given" is syntactic sugar (at least for this use case).
Nick Coghlan, 20.07.2010 15:27:
having the question come up twice within the last month finally inspired me to write the current status of the topic down in a deferred PEP: http://www.python.org/dev/peps/pep-3150/
Thanks for writing that up. I like the idea in general. As for input from the "major Python implementations", we currently do similar things internally in Cython for optimisation purposes, so a syntactic 'where' clause with expression-local scope would be trivial to implement on our side as most of the infrastructure is there anyway. Stefan
On Tue, Jul 20, 2010 at 2:27 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
For the record, I am personally +1 on the idea (otherwise I wouldn't have put so much thought into it over the years). It's just a *lot* harder to define complete and consistent semantics for the concept than people often realise.
However, having the question come up twice within the last month finally inspired me to write the current status of the topic down in a deferred PEP: http://www.python.org/dev/peps/pep-3150/
Is Python's grammar still LL(1) under this proposal? Mark
On Sat, Jul 24, 2010 at 12:11 AM, Mark Dickinson <dickinsm@gmail.com> wrote:
On Tue, Jul 20, 2010 at 2:27 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
For the record, I am personally +1 on the idea (otherwise I wouldn't have put so much thought into it over the years). It's just a *lot* harder to define complete and consistent semantics for the concept than people often realise.
However, having the question come up twice within the last month finally inspired me to write the current status of the topic down in a deferred PEP: http://www.python.org/dev/peps/pep-3150/
Is Python's grammar still LL(1) under this proposal?
It would need to stay that way, as the LL(1) restriction is a rule Guido has stated he is very reluctant to break. I *think* my second grammar change suggestion conforms with that requirement but I haven't actually tried it out. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 7/20/2010 1:29 AM, Stephen J. Turnbull wrote:
Sergio Davis writes:
I'm considering the following extension to Python's grammar: adding the 'where' keyword, which would work as follows:
where_expr : expr 'where' NAME '=' expr
We just had a long thread about this.
http://mail.python.org/pipermail/python-ideas/2010-June/007476.html
The sentiment was about -0 to -0.5 on the idea in general,
I did not comment then because I thought the idea of cluttering python with augmented local namespace blocks, with no functional gain, was rejected and dead, and hence unnecessary of comment. -10 For me, the idea would come close to destroying (what remains of) the simplicity that makes Python relatively easy to learn. It seems to be associated with the (to me, cracked) idea that names are pollution. I agree with Jack Diederich:
I think the "trick" to making it readable is putting the assignment first.
par_pos = decl.find('(') vtype = decl[par_pos+1:FindMatching(par_pos, decl)].strip()
versus:
vtype = decl[par_pos+1:FindMatching(par_pos, decl)].strip() where par_pos=decl.find('(')
The real horror would come with multiple assignments with multiple and nested where or whatever clauses. -- Terry Jan Reedy
Terry Reedy, 20.07.2010 21:49:
I did not comment then because I thought the idea of cluttering python with augmented local namespace blocks, with no functional gain, was rejected and dead, and hence unnecessary of comment. -10 For me, the idea would come close to destroying (what remains of) the simplicity that makes Python relatively easy to learn. It seems to be associated with the (to me, cracked) idea that names are pollution.
Actually, it's about *giving* names to subexpressions, that's quite the opposite.
I agree with Jack Diederich:
I think the "trick" to making it readable is putting the assignment first.
par_pos = decl.find('(') vtype = decl[par_pos+1:FindMatching(par_pos, decl)].strip()
versus:
vtype = decl[par_pos+1:FindMatching(par_pos, decl)].strip() where par_pos=decl.find('(')
The real horror would come with multiple assignments with multiple and nested where or whatever clauses.
I agree that the placement *behind* the expression itself *can* be suboptimal, but then, we also have conditional expressions, where it's good to know when to use them and when they get too long to be readable. The same applies here. However, I take your point that this is nothing that really makes anything simpler or that potentially opens new use cases (like the 'with' statement did, for example). It's a plain convenience syntax and as such not really worth defending over potential draw-backs. Stefan
On Wed, Jul 21, 2010 at 9:16 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Terry Reedy, 20.07.2010 21:49:
I did not comment then because I thought the idea of cluttering python with augmented local namespace blocks, with no functional gain, was rejected and dead, and hence unnecessary of comment. -10 For me, the idea would come close to destroying (what remains of) the simplicity that makes Python relatively easy to learn. It seems to be associated with the (to me, cracked) idea that names are pollution.
Actually, it's about *giving* names to subexpressions, that's quite the opposite.
I think Terry's point was that you can already give names to subexpressions by assigning them to variables in the current scope, but some people object to that approach due to "namespace pollution". I agree with him that avoiding namespace pollution isn't a particular strong argument though (unless you have really long scripts and functions), which is why I've tried to emphasise the intended readability benefits. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 7/21/2010 7:24 AM, Nick Coghlan wrote:
On Wed, Jul 21, 2010 at 9:16 PM, Stefan Behnel<stefan_ml@behnel.de> wrote:
Terry Reedy, 20.07.2010 21:49:
I did not comment then because I thought the idea of cluttering python with augmented local namespace blocks, with no functional gain, was rejected and dead, and hence unnecessary of comment. -10 For me, the idea would come close to destroying (what remains of) the simplicity that makes Python relatively easy to learn. It seems to be associated with the (to me, cracked) idea that names are pollution.
Actually, it's about *giving* names to subexpressions, that's quite the opposite.
I think Terry's point was that you can already give names to subexpressions by assigning them to variables in the current scope, but some people object to that approach due to "namespace pollution".
Right.
I agree with him that avoiding namespace pollution isn't a particular strong argument though (unless you have really long scripts and
Okay, we can leave that issue aside.
functions), which is why I've tried to emphasize the intended readability benefits.
whereas I am trying to emphasize the reading horror for people whose brains are wired differently from yours. The backwards conditional expressions are nearly impossible for me to read, which is to say, painful. To some, something like e = fe(a,b,c, p1) where: c = fc(a, d, p2 where: d = fd(a, p1) where: a = fa(p1, p2) b = fb(a,p2) where p1,p2 are input parameters; looks about as bad (and it was a real effort to write). I would rather something like that were in a branch dialect, Ypthon with its own extension (.yp). Algorithm book authors usually want their books read by lots of people. When they invent a pseudocode language, they usually invent something lots of people can read. (Knuth's MIX was something of an exception.) It is often so close to (a subset of) Python that it is ridiculous that they do not just use (a subset) Python so it is not 'pseudo'. I cannot remember seeing anything like the above. I believe the reason is because it would be, on average, less readable and harder to understand. -- Terry Jan Reedy
On Wed, Jul 21, 2010 at 9:21 PM, Terry Reedy <tjreedy@udel.edu> wrote:
On 7/21/2010 7:24 AM, Nick Coghlan wrote:
On Wed, Jul 21, 2010 at 9:16 PM, Stefan Behnel<stefan_ml@behnel.de> wrote:
Terry Reedy, 20.07.2010 21:49:
I did not comment then because I thought the idea of cluttering python with augmented local namespace blocks, with no functional gain, was rejected and dead, and hence unnecessary of comment. -10 For me, the idea would come close to destroying (what remains of) the simplicity that makes Python relatively easy to learn. It seems to be associated with the (to me, cracked) idea that names are pollution.
Actually, it's about *giving* names to subexpressions, that's quite the opposite.
I think Terry's point was that you can already give names to subexpressions by assigning them to variables in the current scope, but some people object to that approach due to "namespace pollution".
Right.
I agree with him that avoiding namespace pollution isn't a particular strong argument though (unless you have really long scripts and
Okay, we can leave that issue aside.
functions), which is why I've tried to emphasize the intended readability benefits.
whereas I am trying to emphasize the reading horror for people whose brains are wired differently from yours. The backwards conditional expressions are nearly impossible for me to read, which is to say, painful. To some, something like
Ok, so, your brain is wired differently. That's fine- but mine says that there are cases where the up-front syntax is more readable. I particularly like that it clearly marks what variables will not be used elsewhere without forcing me to jump around in the code to find out how they're being computed.
e = fe(a,b,c, p1) where: c = fc(a, d, p2 where: d = fd(a, p1) where: a = fa(p1, p2) b = fb(a,p2)
where p1,p2 are input parameters;
looks about as bad (and it was a real effort to write). I would rather something like that were in a branch dialect, Ypthon with its own extension (.yp).
Of course it looks bad. You can make anything look bad. 37 levels of nested 'for' statements look bad. That isn't an argument against this, it's an argument against bad code period.
Algorithm book authors usually want their books read by lots of people. When they invent a pseudocode language, they usually invent something lots of people can read. (Knuth's MIX was something of an exception.) It is often so close to (a subset of) Python that it is ridiculous that they do not just use (a subset) Python so it is not 'pseudo'. I cannot remember seeing anything like the above. I believe the reason is because it would be, on average, less readable and harder to understand.
I would use this, and would welcome it in a textbook. Telling people what you're doing before you do it motivates the material and helps people to learn and understand with minimal effort IMO/E. Geremy Condra
Richard Jones (who isn't subscribed to this list, else he'd be sending this message himself) would like it pointed out that the "withhacks" module[1] does some things very similar to this (by using bytecode hackery, IIUC). See especially the xkwargs and xargs functions.
from withhacks import xkwargs print(xkwargs.__doc__) WithHack calling a function with extra keyword arguments.
This WithHack captures any local variables created during execution of the block, then calls the given function using them as extra keyword arguments. >>> def calculate(a,b): ... return a * b ... >>> with xkwargs(calculate,b=2) as result: ... a = 5 ... >>> print result 10 -- Mark
participants (26)
-
Alex Light -
Alexander Belopolsky -
Andrey Popp -
Antoine Pitrou -
Bruce Leban -
Carl M. Johnson -
Chris Rebert -
Daniel Stutzbach -
George Sakkis -
geremy condra -
Guido van Rossum -
Ian Bicking -
Jack Diederich -
John Arbash Meinel -
Mark Dickinson -
Masklinn -
MRAB -
Nathan Schneider -
Nick Coghlan -
Raymond Hettinger -
Sergio Davis -
Stefan Behnel -
Stephen J. Turnbull -
Sturla Molden -
Terry Reedy -
Éric Araujo