explicitation lines in python ?

If we could explicitate a too complex expression in an indented next line, I would use this feature very often : htmltable = ''.join( '<tr>{}</tr>'.format(htmlline) for line in table) : # main line htmlline : ''.join( '<td>{}</td>'.format(cell) for cell in line) # explicitation(s) line(s) (Sorry if this has already been discussed earlier on this list, I have not read all the archives) ******* in details : List comprehension "<expression> for x in mylist" often greatly improve readability of python programs, when <expression> is not too complex When <expression> is too complex (ex: nested lists), this become not readable, so we have to find another solution : a) defining a function expression(x), or an iterator function, wich will only used once in the code b) or droping this beautiful syntax to replace it the very basic list construction : newlist = [] for x in myiterable newlist.append(<expression>) I often choose b), but I dislike both solutions : - in solution a) function def can be far from list comprehension, in fact instructions to build the new list are split in two different places in the code. - solution b) seems a bit better to me, but the fact we build a new list from myiterable is not visible in a glance, unlike list comprehensions. Renouncing to list comprehension occurs rather often when I write python code I think we could greatly improve readability if we could keep list comprehension anywhere in all cases, but when necessary explicit a too complex expression in an indented line : htmltable = ''.join( '<tr>{}</tr>'.format(htmlline) for line in table) : # main line htmlline : ''.join( '<td>{}</td>'.format(cell) for cell in line) # explicitation(s) line(s) In the case the main line is the header of a "classical" indented block (starting with "for", "if", "with"...) , this idented block would simply follow the explicitation(s) line(s). The explicitations lines can be surely identified as the lines than begin with "identifier :" (when we are not in an unclosed dict) with open('data.txt') as f : if line in enumerate(mylist) : # main line mylist : f.read().strip().lower() # explicitation(s) line(s) print line # "classical" indented block Another possible use of "explicitations lines" is a coding style wich would start by "the whole picture" first, and completing with details after, wich is the usual way we mentally solve problems. Let's take an example : we want to write a function wich returns a multiplication table in a simle html document. When I solve this problem, il think a bit like that : - I need to return an html page. For that I need a "header" and a "body". My body will contain an "htmltable", wich be built from a "table" of numbers etc. My code could look like that : def tag(content, *tags): # little convenient function retval = content for t in tags: retval = '<{0}>{1}</{0}>'.format(t, retval) return retval def xhtml_mult_table(a, b): return tag(header + body, 'html') : header : tag('multiplication table', 'title') body : tag(htmltable, 'tbody', 'table', 'body') : htmltable : ''.join(tag(xhtmlline, 'tr') for line in table) : table : headerline + otherlines : headerline : [[''] + range(a)] otherlines : [[y] + [x*y for x in range(a)] for y in range(b)] xhtmlline : ''.join(tag(str(cell), 'td') for cell in line) This example is a "heavy use" of the "explicitation line" feature, to illustrate how it could work. I don't mean this should replace the "classical" syntax everywhere possible, but for me this would be for me a nice way to explicitate complex expressions from time to time, and the ability to use list comprehension everywhere I wan't. Daniel

On Fri, Jun 25, 2010 at 7:25 PM, Daniel DELAY <danieldelay@gmail.com> wrote:
I don't know where you got the word "explicitation" -- I've never heard of it. (Maybe it's French? You sound French. :-) However, this feature existed in ABC under the name "refinement". See http://homepages.cwi.nl/~steven/abc/qr.html#Refinements -- --Guido van Rossum (python.org/~guido)

Le 26/06/2010 04:36, Guido van Rossum a écrit :
You guessed well : I'm french and a bad english speaker :-( , "explicitation" was a mistaken translation. From now I'll use the term "refinement" from ABC as this language inspired Python Thanks for the link. It's a bit difficult for me to figure out precisely how this feature works in ABC, this quick reference lacks of examples to illustrate each concept and it seems difficult to find more documentation. But I'm happy to see this feature has been implemented in few languages. Cheers -- Daniel

Daniel DELAY writes:
(Sorry if this has already been discussed earlier on this list, I have not read all the archives)
I think if you search for "first-class blocks" and "lambdas", or similar, you'll find related discussion (although not exactly the same thing). It also looks very similar to the Haskell "where", maybe searching for "Haskell where" would bring it up.
(Edited for readability; it was munged by your mail client. ;-) I'm not sure I like this better than the alternative of rewriting the outer loops explicitly. But if you're going to add syntax, I think the more verbose htmltable = ''.join('<tr>{}</tr>'.format(htmlline) for line in table) \ with htmlline = ''.join('<td>{}</td>'.format(cell) for cell in line) looks better. Note that the "with" clause becomes an optional part of an assignment statement rather than a suite controlled by the assignment, and the indentation is decorative rather than syntactic. I considered "as" instead of "=" in the with clause, but preferred the "=" because that allows nested "with" in a natural way. (Maybe, I haven't thought carefully about that at all.) Obviously "with" was chosen because it's already a keyword. I suspect this has been shot down before, though.

I suspect this has been shot down before, though.
Not so much shot down, as "never found a syntax and semantics that were sufficiently clear". Looking up 'statement local namespaces' for Python brings up some old discussions of the idea: http://www.mail-archive.com/python-list@python.org/msg07034.html (the 'new' compiler in that message is the AST compiler adopted in Python 2.5. The 'nonlocal' keyword now gives us more options in deciding how to handle assignment statements) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Fri, Jun 25, 2010 at 7:58 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Prior thread: [Python-ideas] Where-statement (Proposal for function expressions) http://mail.python.org/pipermail/python-ideas/2009-July/005114.html There certainly was criticism: http://mail.python.org/pipermail/python-ideas/2009-July/005213.html However, the BDFL seemed receptive: http://mail.python.org/pipermail/python-ideas/2009-July/005299.html Cheers, Chris -- http://blog.rebertia.com

I really dislike the idea that when I read an expression I'd have to scan to the end of the statement to figure out if it's a forward or backward reference. Is this def foo(a, b): return x * y: x : a + b y : a - b really significantly better than: def foo(a, b): x = lambda: a + b y = lambda: a - b return x() * y() Note that when I see the () there's an explicit marker that x and y are not simple variables so personally I wouldn't want to "save" those few characters. So really what you're doing is allowing me to put these in a different order and saving 7 characters. But I can reorder them easily enough if I want to: def foo(a, b): result = lambda: x() * y() x = lambda: a + b y = lambda: a - b return result() --- Bruce http://www.vroospeak.com http://jarlsberg.appspot.com On Fri, Jun 25, 2010 at 11:48 PM, geremy condra <debatem1@gmail.com> wrote:

Le 26/06/2010 04:58, Stephen J. Turnbull a écrit :
This syntax on one line is interesting if we see "refinement" as a way to make more readable a too long line. But I'm not sure wether this syntax is compatible with nesting different levels of refinement in a recursive way, as I did in an example. Using "with" as an optional part of assignment seems to me rather restrictive, as too complex expressions may appear anywhere involving an expression, not only where expressions are assigned to a variable with "=". Cheers, Daniel

Daniel DELAY writes:
I'm not sure of all the corner cases myself, but it seems to me that the above example could be extended to htmltable = ''.join(tr.format(htmlline) for line in table) \ with tr = '<tr>{}</tr>', \ htmlline = ''.join(td.format(cell) for cell in line) \ with td = '<td>{}</td>' although it's not as prettily formatted as your examples.
That was deliberate. If it's not an assignment, it's easy enough (and preserves locality) to insert an assignment to a new variable on the preceding line.

On Fri, Jun 25, 2010 at 12:08 PM, Daniel DELAY <danieldelay@gmail.com> wrote:
If we could explicitate
That's not a word. \> a too complex expression in an indented next line,
Again, not a word, and not a great name for this either, IMO.
What do you mean, you can put them right next to each other, and even better, give the expression a meaningful name: def line2html(line): return ''.join( '<td>{}</td>'.format(cell) for cell in line) htmltable = ''.join( '<tr>{}</tr>'.format(line2html(line)) for line in table)
- solution b) seems a bit better to me,
I'm gonna disagree with you there, but it is a somewhat subjective stylistic issue.
In other words, "where" clauses, à la Haskell (see http://www.haskell.org/tutorial/patterns.html section 4.5); just tweak the syntax from expr_involving_bar : bar : expr to expr_involving_bar where: bar = expr which avoids overloading colons further (i.e. constipation ;-P) and the equals sign makes more sense anyway. Having used Haskell a little bit, I can say "where" clauses can indeed make some code easier to read. However, adding them to an imperative language like Python is more problematic, since order of evaluation matters and it complicates the flow of control by causing it to go backward in "where" clauses. Basically, I don't see the problem with solution "(a)"; your general idea isn't without merit though. Cheers, Chris -- http://blog.rebertia.com

Le 26/06/2010 05:26, Chris Rebert a écrit :
If we could explicitate
That's not a word.
Sorry that was a wrong translation of french word "expliciter" (http://www.wordreference.com/fren/expliciter) wich means something like to express or to describe with more details.
explicitation(s) line(s)
Again, not a word, and not a great name for this either, IMO.
Yes perhaps "refinement line" would be better as GVR noticed this feature is named "refinement" in ABC.
Yes that's a solution if this piece of code is run only once. But if this piece of code is in a function (or a loop), your function line2html will be redefined for each function call (or iteration), wich is something I usually try to avoid.
Oh yes that seems to be the same feature.
and the equals sign makes more sense anyway.
Equals sign make sense if you see that feature as a variable assigment, but this would make more difficult to distinguish refinement lines from classical indented lines. I used ":" because I see that more like something equivalent to a {key: value} substitution wich would be done before execution. In fact there are several options about how refinement option could work a) refinement as a code substitution : In this option, refinement is just a syntactic sugar. The refinement name ("htmlline" in my example) is replaced with the refinement expression it means before execution of python code. As "htmlline" dsappear from code at the time of execution, it will not populate locals(). b) refinement as variable assignment. In this option, the values of htmlline are really stored in a local variable "htmlline", wich will remain in locals() after execution of this line. As "htmlline" is not intented to be used elsewhere in the code, I would probably prefer option a), but the pure substitution option has a disadvantage : when the refinement name "htmlline" appears twice or more in the main line, the same expression is evaluated twice or more wich is probably not what we want. That's why I would in fact prefer a third option c) a refinement expression is only evaluated once even if the refinement name (ex: "htmlline") appears twice or more, but that name is not published in locals(). Cheers -- Daniel

On Sun, Jun 27, 2010 at 12:00 AM, Daniel DELAY <danieldelay@gmail.com> wrote:
Yes, but Premature Optimization is The Root of All Evil, and you can always define line2html() at the module level; this trades distance for speed, but if you chose a good descriptive name, it should still be plenty clear. Cheers, Chris -- Again, "where" clauses are certainly an intriguing idea. http://blog.rebertia.com

On Sun, Jun 27, 2010 at 5:00 PM, Daniel DELAY <danieldelay@gmail.com> wrote:
That's definitely the same goal as the old "statement local namespace" idea. Trawling through a couple of old threads, the various keywords suggested were: - with (pro: already a keyword, con: now has completely different meaning from normal with statement) - where (pro: same name as used in Haskell, con: new keyword, also completely different meaning from the SQL meaning many programmers will find more familiar) - using (pro: completely made up name at the time, con: new keyword, now conflicts with the C# equivalents to Python's with and import statements) These days, I'm personally inclined to favour Haskell's "where" terminology, but that preference isn't particularly strong. The availability of "nonlocal" binding semantics also makes the semantics much easier to define than they were in those previous discussions (the lack of clear semantics for name binding statements with an attached local namespace was the major factor blocking creation of a reference implementation for this proposal back then). For example: c = sqrt(a*a + b*b) where: a = retrieve_a() b = retrieve_b() could translate to something like: def _anon(): # *(see below) nonlocal c a = retrieve_a() b = retrieve_b() c = sqrt(a*a + b*b) _anon() *(unlike Python code, the compiler can make truly anonymous functions by storing them solely on the VM stack. It already does this when executing class definitions): The major question mark over the idea is whether or not it would actually help or hinder readability in practice. This is a question that could be addressed by someone trawling the standard library or other large public Python code bases (e.g. things in SciPy, the assorted web frameworks, bzr, hg) for existing code that could be made clearer if less important details could easily be moved out of the way into an indented suite. It's also something that has no chance of being accepted until after the language moratorium ends. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Jun 27, 2010 at 8:25 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I like this idea, but I would tweak it slightly. Maybe we should say EXPRESSION where: BLOCK is equivalent to def _(): BLOCK return EXPRESSION _() That way, c = a where: a = 7 would be equivalent to def _(): a = 7 return a c = _() One advantage of this equivalence is it would make it easier to work around a longstanding scoping gotcha. A naïve coder might expect this code to print out numbers 0 to 4: >>> fs = [] >>> for n in range(5): ... def f(): ... print(item) ... fs.append(f) ... >>> [f() for f in fs] 4 4 4 4 4 [None, None, None, None, None] I think we all have enough experience to know this isn’t a totally unrealistic scenario. I personally stumbled into when I was trying to create a class by looping through a set of method names. To get around it, one could use a where clause like so: fs = [] for n in range(5): fs.append(f) where: shadow = n def f(): print(shadow) This would print out 0 to 4 as expected and be equivalent to >>> fs = [] >>> for n in range(5): ... def _(): ... shadow = n ... def f(): ... print(shadow) ... fs.append(f) ... _() ... >>> [f() for f in fs] 0 1 2 3 4 [None, None, None, None, None] I think a where-clause with def-like namespace semantics would be a positive addition to Python, once the moratorium is up. -- Carl Johnson

On Mon, Jul 12, 2010 at 1:39 PM, Carl M. Johnson <cmjohnson.mailinglist@gmail.com> wrote:
Implement it that way (or find someone who can), then get back to me* :) That said, my suggested semantics still have the desired effect in your use case, since your expression does not contain a name binding operation, so it makes no difference whether name binding would have been handled via a return value (your suggestion, which I tried and failed to implement last time) or via nonlocal name bindings (my suggestion this time around). Cheers, Nick. *P.S. There's a reason I stopped pushing this idea back then: the absolute nightmare that was trying to implement it without ready access to nonlocal variable definitions (trying to figure out what the return value should be and how it should be unpacked in the surrounding scope was seriously ugly). Using nonlocal semantics instead should make it relatively straightforward (fairly similar to a class definition in fact, although the compilation options for the nested code object will be different and there'll be a bit of additional dancing during the symbol pass to figure out any implicit nonlocal declarations for the inner scope). -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Bleh, I just remembered why nonlocal semantics won't work for this use case: nonlocal only looks at function namespaces, so class and module namespaces don't count. That behaviour would be unacceptable for a where clause implementation. So this suggestion going anywhere post-moratorium is firstly dependent on someone figuring out how to properly split name binding operations across the two namespaces (such that the values are generated in the inner scope, but assigned in the outer scope). As an example of the kind of thing that actually makes this a nightmare: x = b[index] = value where: index = calc_target_index() value = calc_value() It turns out that name binding is only part of the problem though. Variable *lookup* actually shares one of the problems of nonlocal name binding: it skips over class scopes, so the inner scope can't see class level names. Generator expressions and most comprehensions (all bar 2.x list comprehensions) already have this problem - at class scope, only the outermost iterator can see names defined in the class body, since everything else is in a nested scope where name lookup skips over the class due to the name lookup semantics that were originally designed for method implementations (i.e. before we had things like generator expressions that implicitly created new scopes). It took a while for all these evil variable referencing semantic problems to come back to me, but they're the kind of thing that needs to be addressed before a where clause proposal can be taken seriously. As I noted in my last message, I *did* try to implement this years ago and I now remember that the only way I can see it working is to define a completely new means of compiling a code object such that variable lookup and nonlocal namebinding can "see" an immediately surrounding class scope (as well as outer function scopes) and still fall back to global semantics if the name is not found explicitly in the symbol table. I believe such an addition would actually be beneficial in other ways, as I personally consider the current name lookup quirks in generator expressions to be something of a wart and these new semantics for implicit scopes could potentially be used to fix that (although perhaps not until Py4k). However, adding such lookup semantics is a seriously non-trivial task (I've been working with the current compiler since I helped get it ready for inclusion in 2.5 back when it was still on the ast-compiler branch and I'm not sure where I would even start on a project like that. It wouldn't just be the compiler either, the VM itself would almost certainly need some changes). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Fri, Jun 25, 2010 at 7:25 PM, Daniel DELAY <danieldelay@gmail.com> wrote:
I don't know where you got the word "explicitation" -- I've never heard of it. (Maybe it's French? You sound French. :-) However, this feature existed in ABC under the name "refinement". See http://homepages.cwi.nl/~steven/abc/qr.html#Refinements -- --Guido van Rossum (python.org/~guido)

Le 26/06/2010 04:36, Guido van Rossum a écrit :
You guessed well : I'm french and a bad english speaker :-( , "explicitation" was a mistaken translation. From now I'll use the term "refinement" from ABC as this language inspired Python Thanks for the link. It's a bit difficult for me to figure out precisely how this feature works in ABC, this quick reference lacks of examples to illustrate each concept and it seems difficult to find more documentation. But I'm happy to see this feature has been implemented in few languages. Cheers -- Daniel

Daniel DELAY writes:
(Sorry if this has already been discussed earlier on this list, I have not read all the archives)
I think if you search for "first-class blocks" and "lambdas", or similar, you'll find related discussion (although not exactly the same thing). It also looks very similar to the Haskell "where", maybe searching for "Haskell where" would bring it up.
(Edited for readability; it was munged by your mail client. ;-) I'm not sure I like this better than the alternative of rewriting the outer loops explicitly. But if you're going to add syntax, I think the more verbose htmltable = ''.join('<tr>{}</tr>'.format(htmlline) for line in table) \ with htmlline = ''.join('<td>{}</td>'.format(cell) for cell in line) looks better. Note that the "with" clause becomes an optional part of an assignment statement rather than a suite controlled by the assignment, and the indentation is decorative rather than syntactic. I considered "as" instead of "=" in the with clause, but preferred the "=" because that allows nested "with" in a natural way. (Maybe, I haven't thought carefully about that at all.) Obviously "with" was chosen because it's already a keyword. I suspect this has been shot down before, though.

I suspect this has been shot down before, though.
Not so much shot down, as "never found a syntax and semantics that were sufficiently clear". Looking up 'statement local namespaces' for Python brings up some old discussions of the idea: http://www.mail-archive.com/python-list@python.org/msg07034.html (the 'new' compiler in that message is the AST compiler adopted in Python 2.5. The 'nonlocal' keyword now gives us more options in deciding how to handle assignment statements) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Fri, Jun 25, 2010 at 7:58 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Prior thread: [Python-ideas] Where-statement (Proposal for function expressions) http://mail.python.org/pipermail/python-ideas/2009-July/005114.html There certainly was criticism: http://mail.python.org/pipermail/python-ideas/2009-July/005213.html However, the BDFL seemed receptive: http://mail.python.org/pipermail/python-ideas/2009-July/005299.html Cheers, Chris -- http://blog.rebertia.com

I really dislike the idea that when I read an expression I'd have to scan to the end of the statement to figure out if it's a forward or backward reference. Is this def foo(a, b): return x * y: x : a + b y : a - b really significantly better than: def foo(a, b): x = lambda: a + b y = lambda: a - b return x() * y() Note that when I see the () there's an explicit marker that x and y are not simple variables so personally I wouldn't want to "save" those few characters. So really what you're doing is allowing me to put these in a different order and saving 7 characters. But I can reorder them easily enough if I want to: def foo(a, b): result = lambda: x() * y() x = lambda: a + b y = lambda: a - b return result() --- Bruce http://www.vroospeak.com http://jarlsberg.appspot.com On Fri, Jun 25, 2010 at 11:48 PM, geremy condra <debatem1@gmail.com> wrote:

Le 26/06/2010 04:58, Stephen J. Turnbull a écrit :
This syntax on one line is interesting if we see "refinement" as a way to make more readable a too long line. But I'm not sure wether this syntax is compatible with nesting different levels of refinement in a recursive way, as I did in an example. Using "with" as an optional part of assignment seems to me rather restrictive, as too complex expressions may appear anywhere involving an expression, not only where expressions are assigned to a variable with "=". Cheers, Daniel

Daniel DELAY writes:
I'm not sure of all the corner cases myself, but it seems to me that the above example could be extended to htmltable = ''.join(tr.format(htmlline) for line in table) \ with tr = '<tr>{}</tr>', \ htmlline = ''.join(td.format(cell) for cell in line) \ with td = '<td>{}</td>' although it's not as prettily formatted as your examples.
That was deliberate. If it's not an assignment, it's easy enough (and preserves locality) to insert an assignment to a new variable on the preceding line.

On Fri, Jun 25, 2010 at 12:08 PM, Daniel DELAY <danieldelay@gmail.com> wrote:
If we could explicitate
That's not a word. \> a too complex expression in an indented next line,
Again, not a word, and not a great name for this either, IMO.
What do you mean, you can put them right next to each other, and even better, give the expression a meaningful name: def line2html(line): return ''.join( '<td>{}</td>'.format(cell) for cell in line) htmltable = ''.join( '<tr>{}</tr>'.format(line2html(line)) for line in table)
- solution b) seems a bit better to me,
I'm gonna disagree with you there, but it is a somewhat subjective stylistic issue.
In other words, "where" clauses, à la Haskell (see http://www.haskell.org/tutorial/patterns.html section 4.5); just tweak the syntax from expr_involving_bar : bar : expr to expr_involving_bar where: bar = expr which avoids overloading colons further (i.e. constipation ;-P) and the equals sign makes more sense anyway. Having used Haskell a little bit, I can say "where" clauses can indeed make some code easier to read. However, adding them to an imperative language like Python is more problematic, since order of evaluation matters and it complicates the flow of control by causing it to go backward in "where" clauses. Basically, I don't see the problem with solution "(a)"; your general idea isn't without merit though. Cheers, Chris -- http://blog.rebertia.com

Le 26/06/2010 05:26, Chris Rebert a écrit :
If we could explicitate
That's not a word.
Sorry that was a wrong translation of french word "expliciter" (http://www.wordreference.com/fren/expliciter) wich means something like to express or to describe with more details.
explicitation(s) line(s)
Again, not a word, and not a great name for this either, IMO.
Yes perhaps "refinement line" would be better as GVR noticed this feature is named "refinement" in ABC.
Yes that's a solution if this piece of code is run only once. But if this piece of code is in a function (or a loop), your function line2html will be redefined for each function call (or iteration), wich is something I usually try to avoid.
Oh yes that seems to be the same feature.
and the equals sign makes more sense anyway.
Equals sign make sense if you see that feature as a variable assigment, but this would make more difficult to distinguish refinement lines from classical indented lines. I used ":" because I see that more like something equivalent to a {key: value} substitution wich would be done before execution. In fact there are several options about how refinement option could work a) refinement as a code substitution : In this option, refinement is just a syntactic sugar. The refinement name ("htmlline" in my example) is replaced with the refinement expression it means before execution of python code. As "htmlline" dsappear from code at the time of execution, it will not populate locals(). b) refinement as variable assignment. In this option, the values of htmlline are really stored in a local variable "htmlline", wich will remain in locals() after execution of this line. As "htmlline" is not intented to be used elsewhere in the code, I would probably prefer option a), but the pure substitution option has a disadvantage : when the refinement name "htmlline" appears twice or more in the main line, the same expression is evaluated twice or more wich is probably not what we want. That's why I would in fact prefer a third option c) a refinement expression is only evaluated once even if the refinement name (ex: "htmlline") appears twice or more, but that name is not published in locals(). Cheers -- Daniel

On Sun, Jun 27, 2010 at 12:00 AM, Daniel DELAY <danieldelay@gmail.com> wrote:
Yes, but Premature Optimization is The Root of All Evil, and you can always define line2html() at the module level; this trades distance for speed, but if you chose a good descriptive name, it should still be plenty clear. Cheers, Chris -- Again, "where" clauses are certainly an intriguing idea. http://blog.rebertia.com

On Sun, Jun 27, 2010 at 5:00 PM, Daniel DELAY <danieldelay@gmail.com> wrote:
That's definitely the same goal as the old "statement local namespace" idea. Trawling through a couple of old threads, the various keywords suggested were: - with (pro: already a keyword, con: now has completely different meaning from normal with statement) - where (pro: same name as used in Haskell, con: new keyword, also completely different meaning from the SQL meaning many programmers will find more familiar) - using (pro: completely made up name at the time, con: new keyword, now conflicts with the C# equivalents to Python's with and import statements) These days, I'm personally inclined to favour Haskell's "where" terminology, but that preference isn't particularly strong. The availability of "nonlocal" binding semantics also makes the semantics much easier to define than they were in those previous discussions (the lack of clear semantics for name binding statements with an attached local namespace was the major factor blocking creation of a reference implementation for this proposal back then). For example: c = sqrt(a*a + b*b) where: a = retrieve_a() b = retrieve_b() could translate to something like: def _anon(): # *(see below) nonlocal c a = retrieve_a() b = retrieve_b() c = sqrt(a*a + b*b) _anon() *(unlike Python code, the compiler can make truly anonymous functions by storing them solely on the VM stack. It already does this when executing class definitions): The major question mark over the idea is whether or not it would actually help or hinder readability in practice. This is a question that could be addressed by someone trawling the standard library or other large public Python code bases (e.g. things in SciPy, the assorted web frameworks, bzr, hg) for existing code that could be made clearer if less important details could easily be moved out of the way into an indented suite. It's also something that has no chance of being accepted until after the language moratorium ends. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Jun 27, 2010 at 8:25 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I like this idea, but I would tweak it slightly. Maybe we should say EXPRESSION where: BLOCK is equivalent to def _(): BLOCK return EXPRESSION _() That way, c = a where: a = 7 would be equivalent to def _(): a = 7 return a c = _() One advantage of this equivalence is it would make it easier to work around a longstanding scoping gotcha. A naïve coder might expect this code to print out numbers 0 to 4: >>> fs = [] >>> for n in range(5): ... def f(): ... print(item) ... fs.append(f) ... >>> [f() for f in fs] 4 4 4 4 4 [None, None, None, None, None] I think we all have enough experience to know this isn’t a totally unrealistic scenario. I personally stumbled into when I was trying to create a class by looping through a set of method names. To get around it, one could use a where clause like so: fs = [] for n in range(5): fs.append(f) where: shadow = n def f(): print(shadow) This would print out 0 to 4 as expected and be equivalent to >>> fs = [] >>> for n in range(5): ... def _(): ... shadow = n ... def f(): ... print(shadow) ... fs.append(f) ... _() ... >>> [f() for f in fs] 0 1 2 3 4 [None, None, None, None, None] I think a where-clause with def-like namespace semantics would be a positive addition to Python, once the moratorium is up. -- Carl Johnson

On Mon, Jul 12, 2010 at 1:39 PM, Carl M. Johnson <cmjohnson.mailinglist@gmail.com> wrote:
Implement it that way (or find someone who can), then get back to me* :) That said, my suggested semantics still have the desired effect in your use case, since your expression does not contain a name binding operation, so it makes no difference whether name binding would have been handled via a return value (your suggestion, which I tried and failed to implement last time) or via nonlocal name bindings (my suggestion this time around). Cheers, Nick. *P.S. There's a reason I stopped pushing this idea back then: the absolute nightmare that was trying to implement it without ready access to nonlocal variable definitions (trying to figure out what the return value should be and how it should be unpacked in the surrounding scope was seriously ugly). Using nonlocal semantics instead should make it relatively straightforward (fairly similar to a class definition in fact, although the compilation options for the nested code object will be different and there'll be a bit of additional dancing during the symbol pass to figure out any implicit nonlocal declarations for the inner scope). -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Bleh, I just remembered why nonlocal semantics won't work for this use case: nonlocal only looks at function namespaces, so class and module namespaces don't count. That behaviour would be unacceptable for a where clause implementation. So this suggestion going anywhere post-moratorium is firstly dependent on someone figuring out how to properly split name binding operations across the two namespaces (such that the values are generated in the inner scope, but assigned in the outer scope). As an example of the kind of thing that actually makes this a nightmare: x = b[index] = value where: index = calc_target_index() value = calc_value() It turns out that name binding is only part of the problem though. Variable *lookup* actually shares one of the problems of nonlocal name binding: it skips over class scopes, so the inner scope can't see class level names. Generator expressions and most comprehensions (all bar 2.x list comprehensions) already have this problem - at class scope, only the outermost iterator can see names defined in the class body, since everything else is in a nested scope where name lookup skips over the class due to the name lookup semantics that were originally designed for method implementations (i.e. before we had things like generator expressions that implicitly created new scopes). It took a while for all these evil variable referencing semantic problems to come back to me, but they're the kind of thing that needs to be addressed before a where clause proposal can be taken seriously. As I noted in my last message, I *did* try to implement this years ago and I now remember that the only way I can see it working is to define a completely new means of compiling a code object such that variable lookup and nonlocal namebinding can "see" an immediately surrounding class scope (as well as outer function scopes) and still fall back to global semantics if the name is not found explicitly in the symbol table. I believe such an addition would actually be beneficial in other ways, as I personally consider the current name lookup quirks in generator expressions to be something of a wart and these new semantics for implicit scopes could potentially be used to fix that (although perhaps not until Py4k). However, adding such lookup semantics is a seriously non-trivial task (I've been working with the current compiler since I helped get it ready for inclusion in 2.5 back when it was still on the ast-compiler branch and I'm not sure where I would even start on a project like that. It wouldn't just be the compiler either, the VM itself would almost certainly need some changes). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (8)
-
Bruce Leban
-
Carl M. Johnson
-
Chris Rebert
-
Daniel DELAY
-
geremy condra
-
Guido van Rossum
-
Nick Coghlan
-
Stephen J. Turnbull