I really like the proposal, although I would be interested to see if anyone can bikeshed up keywords that might be better than : or @… :-) I do have some questions about it. Does using @ instead of the name defined below make the implementation easier? In other words, would this cause a NameError on normalize because normalize isn't defined in the local namespace?-- :sorted_list = sorted(original, key=normalize) def normalize(item): … Or is the @ just for brevity? I assume the point is that it's not just brevity, but you have to use the @ in order to make the implementation straightforward. -- Carl Johnson

On Thu, Oct 13, 2011 at 2:45 PM, Carl M. Johnson wrote:

I really like the proposal, although I would be interested to see if anyone can bikeshed up keywords that might be better than : or @… :-)

I do have some questions about it. Does using @ instead of the name defined below make the implementation easier? In other words, would this cause a NameError on normalize because normalize isn't defined in the local namespace?--

:sorted_list = sorted(original, key=normalize) def normalize(item):   …

Or is the @ just for brevity? I assume the point is that it's not just brevity, but you have to use the @ in order to make the implementation straightforward.

The brevity and highlighting the "forward reference" are actually the primary motivation, making the implementation easier (which it does do) is a nice bonus. When I first started writing up the PEP, I was using a syntax more directly inspired by PEP 3150's given clause: :sorted_list = sorted(original, key=@) given (item): … There were a few problems with this: 1. It pushes the signature of the callable all the way over to the RHS 2. The callable is actually anonymous, so @.__name__ would always be "<given>". That's annoying for the same reason as it's annoying in lambda expressions. 3. It doesn't provide any real clues that the body of the statement is actually a nested function So then I thought, "well what if I use 'def' instead of 'given' as the keyword"? At that point, it was only a short leap to realising that what I wanted was really close to "arbitrary simple statement as a decorator". So I rewrote things to the format in the posted PEP: :sorted_list = sorted(original, key=@) def normalise(item): … Now that the nested function is being given a name, I realised I *could* just refer to it by that name in the block prefix. However, I left it alone for the reasons I mention above: 1. It highlights that this is not a normal name reference but something special (i.e. a forward reference to the object defined by the statement) 2. The fact that references are always a single character makes it easier to justify giving the function itself a nice name, which improves introspection support and error messages. In genuine throwaway cases, you can always use a dummy name like 'f', 'func', 'g', 'gen' or 'block' or 'attr' depending on what you're doing. 3. Multiple references don't make the block prefix unwieldy (although the use cases for those are limited) 4. It does make the implementation easier, since you don't have to worry about namespace clashes - the function remains effectively anonymous in the containing scope. It would be possible to extend the PEP to include the idea of allowing the name to be omitted in function and class definitions, but that's an awful lot of complexity when it's easy to use a throwaway name if you genuinely don't care. Just like PEP 3150, all of this is based on the premise that one of the key benefits of multi-line lambdas is that it lets you do things in the *right order* - operation first, callable second. Ordinary named functions force you to do things the other way around. Decorator abuse is also a problematic approach, since even though it gets the order right, you end up with something that looks like an ordinary function or class definition but is actually nothing of the sort. Oh, I'll also note that the class variant gives you the full power of PEP 3150 without any (especially) funky new namespace semantics: :x = property(@.get, @.set, @.delete) class scope: def get(self): return __class__.attr def set(self, val): __class__.attr = val def delete(self): del __class__.attr Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Oct 12, 2011, at 7:45 PM, David Townshend wrote:

A question: As I understand it, the function is never actually bound to its name, i.e. in your first example the name "report_destruction" doesn't exist after the statement. If this is the case, then there seems little point assigning a name at all other than for providing a description. In fact, assigning a name implies that it is reusable and that the name means something.

I'm not sure I like the idea of allowing defs without a name, but perhaps its something to think about.

-1 To me, the names are part of the documentation. The advantage of anonymous blocks is the block part, not the anonymous part.

After reading your earlier reply about the benefits of named functions, I'm fully with you! +1 to the PEP. On Thu, Oct 13, 2011 at 7:54 AM, Nick Coghlan wrote:

On Thu, Oct 13, 2011 at 3:51 PM, Carl M. Johnson wrote:

...

On Oct 12, 2011, at 7:45 PM, David Townshend wrote:

...

A question: As I understand it, the function is never actually bound to

...

...

I'm not sure I like the idea of allowing defs without a name, but

its name, i.e. in your first example the name "report_destruction" doesn't exist after the statement. If this is the case, then there seems little point assigning a name at all other than for providing a description. In fact, assigning a name implies that it is reusable and that the name means something. perhaps its something to think about.

...

-1 To me, the names are part of the documentation. The advantage of

anonymous blocks is the block part, not the anonymous part.

The "no namespace clashes" part is another benefit. PEP 403 attacks that by omitting the name binding in the current scope rather than by omitting the name entirely.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

On Oct 12, 2011, at 8:24 PM, Ka-Ping Yee wrote:

I don't see why the leading colon is necessary, actually. Isn't the presence of a bare @ enough to signal that a function or class definition must come next?

Python's parser is purposefully pretty simple, so it's not clear if that could be added without ripping up the whole language definition.

On Thu, Oct 13, 2011 at 12:30 AM, Carl M. Johnson wrote:

On Oct 12, 2011, at 8:24 PM, Ka-Ping Yee wrote:

...

I don't see why the leading colon is necessary, actually.  Isn't the presence of a bare @ enough to signal that a function or class definition must come next?

Python's parser is purposefully pretty simple, so it's not clear if that could be added without ripping up the whole language definition.

and visually the colon _does_ make the syntax clearer. It works because it's so out of place at the beginning of a line. Scanning through code you won't miss it (and misinterpret what's going on). -eric

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

On Wed, Oct 12, 2011 at 6:22 PM, Nick Coghlan wrote:

After some interesting conversations at PyCodeConf, I'm killing PEP 3150 (Statement Local Namespaces). It's too big, too unwieldy, too confusing and too hard to implement to ever be a good idea.

PEP 403 is a far simpler idea, that looks to decorators (and Ruby blocks) for inspiration. It's still a far from perfect idea, but it has a lot more going for it than PEP 3150 ever did.

The basic question to ask yourself is this: What if we had a syntax that allowed us to replace the final "name = obj" step that is implicit in function and class definitions with an alternative statement, and had a symbol that allowed us to refer to the function or class in that statement without having to repeat the name?

The new PEP is included below and is also available online: http://www.python.org/dev/peps/pep-0403/

I would *love* for people to dig through their callback based code (and any other examples of "single use" functions and classes) to see if this idea would help them.

Cheers, Nick.

PEP: 403 Title: Statement local classes and functions Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan Status: Deferred Type: Standards Track Content-Type: text/x-rst Created: 2011-10-13 Python-Version: 3.x Post-History: 2011-10-13 Resolution: TBD

Abstract ========

This PEP proposes the addition of ':' as a new class and function prefix syntax (analogous to decorators) that permits a statement local function or class definition to be appended to any Python statement that currently does not have an associated suite.

In addition, the new syntax would allow the '@' symbol to be used to refer to the statement local function or class without needing to repeat the name.

When the ':' prefix syntax is used, the associated statement would be executed *instead of* the normal local name binding currently implicit in function and class definitions.

This PEP is based heavily on many of the ideas in PEP 3150 (Statement Local Namespaces) so some elements of the rationale will be familiar to readers of that PEP. That PEP has now been withdrawn in favour of this one.

PEP Deferral ============

Like PEP 3150, this PEP currently exists in a deferred state. Unlike PEP 3150, this isn't because I suspect it might be a terrible idea or see nasty problems lurking in the implementation (aside from one potential parsing issue).

Instead, it's because I think fleshing out the concept, exploring syntax variants, creating a reference implementation and generally championing the idea is going to require more time than I can give it in the 3.3 time frame.

So, it's deferred. If anyone wants to step forward to drive the PEP for 3.3, let me know and I can add you as co-author and move it to Draft status.

Basic Examples ==============

Before diving into the long history of this problem and the detailed rationale for this specific proposed solution, here are a few simple examples of the kind of code it is designed to simplify.

As a trivial example, weakref callbacks could be defined as follows::

   :x = weakref.ref(obj, @)    def report_destruction(obj):        print("{} is being destroyed".format(obj))

This contrasts with the current repetitive "out of order" syntax for this operation::

   def report_destruction(obj):        print("{} is being destroyed".format(obj))

   x = weakref.ref(obj, report_destruction)

That structure is OK when you're using the callable multiple times, but it's irritating to be forced into it for one-off operations.

Similarly, singleton classes could now be defined as::

 :instance = @()  class OnlyOneInstance:    pass

Rather than::

 class OnlyOneInstance:    pass

 instance = OnlyOneInstance()

And the infamous accumulator example could become::

   def counter():        x = 0        :return @        def increment():            nonlocal x            x += 1            return x

Proposal ========

This PEP proposes the addition of an optional block prefix clause to the syntax for function and class definitions.

This block prefix would be introduced by a leading ``:`` and would be allowed to contain any simple statement (including those that don't make any sense in that context - while such code would be legal, there wouldn't be any point in writing it).

The decorator symbol ``@`` would be repurposed inside the block prefix to refer to the function or class being defined.

When a block prefix is provided, it *replaces* the standard local name binding otherwise implicit in a class or function definition.

Background ==========

The question of "multi-line lambdas" has been a vexing one for many Python users for a very long time, and it took an exploration of Ruby's block functionality for me to finally understand why this bugs people so much: Python's demand that the function be named and introduced before the operation that needs it breaks the developer's flow of thought. They get to a point where they go "I need a one-shot operation that does <X>", and instead of being able to just *say* that, they instead have to back up, name a function to do <X>, then call that function from the operation they actually wanted to do in the first place. Lambda expressions can help sometimes, but they're no substitute for being able to use a full suite.

Ruby's block syntax also heavily inspired the style of the solution in this PEP, by making it clear that even when limited to *one* anonymous function per statement, anonymous functions could still be incredibly useful. Consider how many constructs Python has where one expression is responsible for the bulk of the heavy lifting:

 * comprehensions, generator expressions, map(), filter()  * key arguments to sorted(), min(), max()  * partial function application  * provision of callbacks (e.g. for weak references)  * array broadcast operations in NumPy

However, adopting Ruby's block syntax directly won't work for Python, since the effectiveness of Ruby's blocks relies heavily on various conventions in the way functions are *defined* (specifically, Ruby's ``yield`` syntax to call blocks directly and the ``&arg`` mechanism to accept a block as a functions final argument.

Since Python has relied on named functions for so long, the signatures of APIs that accept callbacks are far more diverse, thus requiring a solution that allows anonymous functions to be slotted in at the appropriate location.

Relation to PEP 3150 ====================

PEP 3150 (Statement Local Namespaces) described its primary motivation as being to elevate ordinary assignment statements to be on par with ``class`` and ``def`` statements where the name of the item to be defined is presented to the reader in advance of the details of how the value of that item is calculated. This PEP achieves the same goal in a different way, by allowing the simple name binding of a standard function definition to be replaced with something else (like assigning the result of the function to a value).

This PEP also achieves most of the other effects described in PEP 3150 without introducing a new brainbending kind of scope. All of the complex scoping rules in PEP 3150 are replaced in this PEP with the simple ``@`` reference to the statement local function or class definition.

Symbol Choice ==============

The ':' symbol was chosen due to its existing presence in Python and its association with 'functions in expressions' via ``lambda`` expressions. The past Simple Implicit Lambda proposal (PEP ???) was also a factor.

The proposal definitely requires *some* kind of prefix to avoid parsing ambiguity and backwards compatibility problems and ':' at least has the virtue of brevity. There's no obious alternative symbol that offers a clear improvement.

Introducing a new keyword is another possibility, but I haven't come up with one that really has anything to offer over the leading colon.

Syntax Change =============

Current::

   atom: ('(' [yield_expr|testlist_comp] ')' |           '[' [testlist_comp] ']' |           '{' [dictorsetmaker] '}' |           NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False')

Changed::

   atom: ('(' [yield_expr|testlist_comp] ')' |           '[' [testlist_comp] ']' |           '{' [dictorsetmaker] '}' |           NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False' | '@')

New::

   blockprefix: ':' simple_stmt    block: blockprefix (decorated | classdef | funcdef)

The above is the general idea, but I suspect that change to the 'atom' definition would cause an ambiguity problem in the parser when it comes to detecting decorator lines. So the actual implementation would be more complex than that.

Grammar: http://hg.python.org/cpython/file/default/Grammar/Grammar

Possible Implementation Strategy ================================

This proposal has one titanic advantage over PEP 3150: implementation should be relatively straightforward.

Both the class and function definition statements emit code to perform the local name binding for their defined name. Implementing this PEP should just require intercepting that code generation and replacing it with the code in the block prefix.

The one potentially tricky part is working out how to allow the dual use of '@' without rewriting half the grammar definition.

More Examples =============

Calculating attributes without polluting the local namespace (from os.py)::

 # Current Python (manual namespace cleanup)  def _createenviron():      ... # 27 line function

 environ = _createenviron()  del _createenviron

 # Becomes:  :environ = @()  def _createenviron():      ... # 27 line function

Loop early binding::

 # Current Python (default argument hack)  funcs = [(lambda x, i=i: x + i) for i in range(10)]

 # Becomes:  :funcs = [@(i) for i in range(10)]  def make_incrementor(i):    return lambda x: x + i

 # Or even:  :funcs = [@(i) for i in range(10)]  def make_incrementor(i):    :return @    def incrementor(x):        return x + i

Reference Implementation ========================

None as yet.

TO DO =====

Sort out links and references to everything :)

Acknowledgements ================

Huge thanks to Gary Bernhardt for being blunt in pointing out that I had no idea what I was talking about in criticising Ruby's blocks, kicking off a rather enlightening process of investigation.

References ==========

TBD

Copyright =========

This document has been placed in the public domain.

..   Local Variables:   mode: indented-text   indent-tabs-mode: nil   sentence-end-double-space: t   fill-column: 70   coding: utf-8   End:

-- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

This is really clever. I have some comments: 1. decorators just to clarify, this is legal: :assert @(1) == 1 @lambda f: return lambda x: return x def spam(): pass but decorators above the block prefix are not (since they wouldn't be decorating a def/class statement). 2. namespaces I hate using a class definition as a plain namespace, but the following is cool (if I'm reading this right): :some_func(@.x, @.y) class _: x = 4 y = 1 and given a purer namespace (like http://code.activestate.com/recipes/577887): :some_func(**@) @as_namespace class _: x = 4 y = 1 3. does it cover all the default arguments hack use cases? If so, is it too cumbersome to be a replacement? 4. how do you introspect the statement local function/class? 5. the relationship to statement local namespaces makes me think def-from and assignment decorators, but this seems like something else. Some of these are probably pretty obvious, but it's getting late and my brain's a little fuzzy. :) I'll think about these some more tomorrow. All in all, this is a pretty sweet idea! -eric

On Oct 12, 2011, at 9:32 PM, Eric Snow wrote:

1. decorators

just to clarify, this is legal:

:assert @(1) == 1 @lambda f: return lambda x: return x def spam(): pass

Nitpicker's note: lambda as a decorator isn't legal today (although I argued in the past maybe it should be), and I don't think PEP 403 would change it, so that won't work. Nevertheless, this would be legal if I understand correctly: _ = lambda arg: arg :assert @(1) == 1 @_(lambda f: return lambda x: return x) def spam(): pass

On 13 October 2011 09:14, Nick Coghlan wrote:

That's basically why I went with the leading ':' - it's jarring enough to get your attention, without being so ugly as to be intolerable. However, Georg's "this doesn't look like Python any more" criticism has serious merit - while style guidelines can mitigate that to some degree (just as they advise against gratuitous use of lambda expressions when a named function would be better), there's an inherent ugliness to the syntax in the first draft of the PEP that may make it irredeemable.

Personally, I find the leading ":" too light (to my ageing eyes :-)) so that it gets lost. Also, I am now trained by decorators to see lines starting with @ as "attached" to the following definition, in a way that other syntax isn't. As an alternative bikeshed colour, would @: work rather than plain @?

I already think there are two simplifying assumptions that should be made at least for the first iteration of the idea:

1. Leave classes out of it, at least for now. We did that with decorators, and I think it's a reasonable approach to follow.

While I don't disagree per se, I suspect that statement-local classes will be on the enhancement list from day 1 - the trick of using a class as a local namespace is just too compelling. So deferring that option may be a false economy.

2. The initial version should be an alternative to decorator syntax, not an addition to it. That is, you wouldn't be able to mix the first incarnation of a block prefix with ordinary decorators.

Agreed, and with this one I'm not sure it shouldn't stay a limitation forever. Mixing the two seems like a step too far. (And if you really need it, just call the decorator directly as part of the @: statement).

If we can find an approach that works for the basic case (i.e. naked function definition) and doesn't make people recoil in horror (esp. Guido), then we can look at expanding back to cover these two cases.

I'll note that the only further syntax idea I've had is to replace the ':' with 'postdef' and the '@' with 'def', so some of the examples kicking around would become:

   postdef x = weakref.ref(obj, def)    def report_destruction(obj):        print("{} is being destroyed".format(obj))

   postdef funcs = [def(i) for i in range(10)]    def make_incrementor(i):        postdef return def        def incrementor(x):            return x + i

   postdef sorted_list = sorted(original, key=def)    def normalise(item):        …

That actually looks quite readable to me, and is fairly explicit about what it does: here's a piece of code to run after the following function has been defined. I definitely like it better than what I have in the PEP.

My instinct still prefers a form including a leading @, but I definitely like this better than the bare colon. And given that the "@ attaches to the next statement" instinct is learned behaviour, I'm sure I could learn to recognise something like this just as easily. I *don't* like using def to mark the placeholder, though - too easily lost.

With this variant, I would suggest that any postdef clause be executed *in addition* to the normal name binding. Colliding on dummy names like "func" would then be like colliding on loop variables like "i" - typically harmless, because you don't use the names outside the constructs that define them anyway.

I'm not sure why you feel that using a keyword implies that the binding behaviour should change - but as you say typically it's not likely to matter. Some other minor comments: - I am +1 on this idea, but don't really have any particular use cases so that's on a purely theoretical basis. - The parallel with decorators makes this much easier to integrate mentally than PEP 3150. - Using _ as a throwaway name seems fine to me, and means there's little need to worry about allowing the name to be omitted (although I do think the unused name-as-documentation-only is a little jarring) - The @ as reference to the local function is mildly ugly, in a way that @ at the start of a line isn't. But I don't have any really good alternatives to offer (other than maybe a special name with double underscores, such as __this__, but I'm not convinced by that - hmm, what about a bare double underscore, __? Too "cute"?). - Georg's point about this not looking like Python any more is good. I don't completely agree, and in particular I think that the semantics are sufficiently Pythonic, it's just the syntax that is jarring, but it *is* something to be careful of. Decorators felt much the same when they were introduced, though... Paul.

On Thu, Oct 13, 2011 at 7:10 PM, Paul Moore wrote:

On 13 October 2011 09:14, Nick Coghlan wrote:

...

That's basically why I went with the leading ':' - it's jarring enough to get your attention, without being so ugly as to be intolerable. However, Georg's "this doesn't look like Python any more" criticism has serious merit - while style guidelines can mitigate that to some degree (just as they advise against gratuitous use of lambda expressions when a named function would be better), there's an inherent ugliness to the syntax in the first draft of the PEP that may make it irredeemable.

Personally, I find the leading ":" too light (to my ageing eyes :-)) so that it gets lost. Also, I am now trained by decorators to see lines starting with @ as "attached" to the following definition, in a way that other syntax isn't.

As an alternative bikeshed colour, would @: work rather than plain @?

I'm much happier with the keyword - "postdef" provides a strong hint to what it does, and also ties it in with the following function definition.

...

I already think there are two simplifying assumptions that should be made at least for the first iteration of the idea:

1. Leave classes out of it, at least for now. We did that with decorators, and I think it's a reasonable approach to follow.

While I don't disagree per se, I suspect that statement-local classes will be on the enhancement list from day 1 - the trick of using a class as a local namespace is just too compelling. So deferring that option may be a false economy.

It doesn't make a huge difference, as 'postdef' also works sufficiently well as the keyword for classes as well (since the class statement if more commonly known as a class definition). Including classes may nudge the forward reference syntax back towards the semantically neutral '@', though. The alternative would be to use 'def' for functions and 'class' for classes, which would be a little ugly.

...

2. The initial version should be an alternative to decorator syntax, not an addition to it. That is, you wouldn't be able to mix the first incarnation of a block prefix with ordinary decorators.

Agreed, and with this one I'm not sure it shouldn't stay a limitation forever. Mixing the two seems like a step too far. (And if you really need it, just call the decorator directly as part of the @: statement).

With an explicit keyword, mixing them may be OK, though. For decorator factories, the explicit calling syntax is a little clumsy.

...

That actually looks quite readable to me, and is fairly explicit about what it does: here's a piece of code to run after the following function has been defined. I definitely like it better than what I have in the PEP.

My instinct still prefers a form including a leading @, but I definitely like this better than the bare colon. And given that the "@ attaches to the next statement" instinct is learned behaviour, I'm sure I could learn to recognise something like this just as easily. I *don't* like using def to mark the placeholder, though - too easily lost.

It's easier to see with syntax highlighting, but yeah, I'm not as sold on the idea of using def for forward reference rather than '@' as I am on the switch from a bare colon to the postdef keyword.

...

With this variant, I would suggest that any postdef clause be executed *in addition* to the normal name binding. Colliding on dummy names like "func" would then be like colliding on loop variables like "i" - typically harmless, because you don't use the names outside the constructs that define them anyway.

I'm not sure why you feel that using a keyword implies that the binding behaviour should change - but as you say typically it's not likely to matter.

It's actually more that I wasn't entirely comfortable with suppressing the name binding in the first place and changing to a keyword really emphasised the "do the function definition as normal, but then run this extra piece of code afterwards" aspect. Exposing the function names by default can also help with testability of code that overuses the new construct.

- The @ as reference to the local function is mildly ugly, in a way that @ at the start of a line isn't. But I don't have any really good alternatives to offer (other than maybe a special name with double underscores, such as __this__, but I'm not convinced by that - hmm, what about a bare double underscore, __? Too "cute"?).

No, I think we want a symbol or a real keyword here. Getting too cute with names may actually make it harder to implement and understand rather than easier.

- Georg's point about this not looking like Python any more is good. I don't completely agree, and in particular I think that the semantics are sufficiently Pythonic, it's just the syntax that is jarring, but it *is* something to be careful of. Decorators felt much the same when they were introduced, though...

The leading colon to introduce the new clause was definitely far too cryptic, so I'm a lot happier with the explicit 'postdef' keyword idea. For the rest, I think it falls into the same category as lambda abuse - overusing the post definition clause would be a code smell, to be fought by the forces of style guides, code reviews and an emphasis on writing testable code. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Oct 13, 2011, at 1:50 AM, Nick Coghlan wrote:

I'm much happier with the keyword - "postdef" provides a strong hint to what it does, and also ties it in with the following function definition.

My 2 cents: postdef and @ seem to be a good pair, unless someone has a better name. I would also be surprised if "postdef" as a keyword broke very much code.

On Thu, Oct 13, 2011 at 10:02 PM, Carl M. Johnson wrote:

On Oct 13, 2011, at 1:50 AM, Nick Coghlan wrote:

...

I'm much happier with the keyword - "postdef" provides a strong hint to what it does, and also ties it in with the following function definition.

My 2 cents: postdef and @ seem to be a good pair, unless someone has a better name.

I also think that could work, but I do find 'def' an appealing alternative to '@' because it's a much better intuition pump that the thing it refers to is the object currently being defined. The '@', however, wouldn't be hard to remember once you knew about it, works nicely for both functions and classes, stands out visually even without syntax highlighting and is hard to beat for brevity. I suspect this is a case where a bigger set of example use cases and lining up the two alternatives for each one would be useful. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On 13 October 2011 13:06, Nick Coghlan wrote:

Actually, here's an interesting example based on quickly firing up a worker thread:

postdef t = threading.Thread(target=def); t.start() def pointless():    """A pointless worker thread that does nothing except serve as an example"""    print("Starting")    time.sleep(3)    print("Ending")

You used 2 statements on the postdef line. That's not in the PEP (the original, I haven't read your revisons yet) and is probably fairly difficult to implement as well. Paul.

On Thu, 13 Oct 2011 15:41:45 +0200 Antoine Pitrou wrote:

On Thu, 13 Oct 2011 22:06:00 +1000 Nick Coghlan wrote:

...

Actually, here's an interesting example based on quickly firing up a worker thread:

postdef t = threading.Thread(target=def); t.start() def pointless(): """A pointless worker thread that does nothing except serve as an example""" print("Starting") time.sleep(3) print("Ending")

I think the problem is still that the syntax isn't nice or obvious. Until the syntax is nice and obvious, I don't think there's any point adding it. (by contrast, decorators *are* nice and obvious: writing "@classmethod" before a method was tons better than the previous redundant idiom)

So how about re-using the "lambda" keyword followed by the anonymous function's parameters (if any)? x = weakref.ref(target, lambda obj): print("{} is being destroyed".format(obj)) t = threading.Thread(target=lambda): """A pointless worker thread that does nothing except serve as an example""" print("Starting") time.sleep(3) print("Ending") t.start()

Hi, I dislike "postdef"; it and the leading colon are both mysterious. I guess that's because I find both of them visually misleading. In the case of ":", the colon visually binds to the first token, so :x = whatever(@...) looks like an assignment to ":x" and my brain immediately goes "What's :x?" I have to really work to force myself to make the tiny little ":" bind looser than everything else on the line. In the case of "postdef", the Python syntactic tradition is that a keyword at the beginning always means "this is a statement" and always describes the type of statement. "postdef" breaks both of these rules. This: postdef return whatever(@...) is NOT a postdef statement, it's a return statement. And this: postdef button.add_handler(@...) is not even a statement! It's an expression. ==== Finally, there's a grammar problem that hasn't been addressed yet. What about multi-line statements? Can you write: :if do_something(sorted(items, key=@)): def sort_key(item): # ??? where to indent this? .... # sort_key body ... # if body I sure hope not! ==== All of the above is making me really like the syntactic variations that put the ":" at the end of a line. This is visually natural to me: a colon introduces a block, and that's a good approximation of what is happening here. It also clearly demands one level of indentation, which seems like a good idea. And it has this nice property that it signals when you are allowed to use it: on any line that doesn't *already* end with a colon. If we made that the rule, then post-definitions would be allowed in these kinds of statements, which seems reasonable: assert assignment del exec expression global nonlocal print raise return yield --Ping

Nick Coghlan wrote:

postdef x = weakref.ref(obj, def) def report_destruction(obj): print("{} is being destroyed".format(obj))

postdef funcs = [def(i) for i in range(10)] def make_incrementor(i): postdef return def def incrementor(x): return x + i

postdef sorted_list = sorted(original, key=def) def normalise(item): …

That actually looks quite readable to me, and is fairly explicit about what it does:

Sorry, but I think it only looks that way to you because you invented it. To me, all of these look like two completely separate statements: some weird thing starting with "postdef", and then a function definition. It might be slightly better if the subsequent def were indented and made into a suite: postdef x = weakref.ref(obj, def): def report_destruction(obj): print("{} is being destroyed".format(obj)) Now it's very clear that the def is a subordinate clause.

With this variant, I would suggest that any postdef clause be executed *in addition* to the normal name binding. Colliding on dummy names like "func" would then be like colliding on loop variables like "i" - typically harmless, because you don't use the names outside the constructs that define them anyway.

That sounds reasonable. However, if we're binding the name anyway, why not use it in the main clause instead of introducing something weird like a lone 'def'? postdef x = weakref.ref(obj, report_destruction): def report_destruction(obj): print("{} is being destroyed".format(obj)) This makes it even more obvious what's going on. Furthermore, there's no longer any need to restrict ourselves to a single 'def' in the body, or even any need for the defining statement to be a 'def' -- anything that binds a name would do. We've now arrived at something very like PEP 3150, but without the local-namespace idea that was causing all the difficulties. -- Greg

Greg Ewing writes:

Nick Coghlan wrote:

...

postdef x = weakref.ref(obj, def) def report_destruction(obj): print("{} is being destroyed".format(obj))

postdef funcs = [def(i) for i in range(10)] def make_incrementor(i): postdef return def def incrementor(x): return x + i

postdef sorted_list = sorted(original, key=def) def normalise(item): …

That actually looks quite readable to me, and is fairly explicit about what it does:

Sorry, but I think it only looks that way to you because you invented it. To me, all of these look like two completely separate statements: some weird thing starting with "postdef", and then a function definition.

I have to agree. The decorator syntax was hotly debated for (in part) the very same reason: when looking at the definition of a function, a statement *preceding* the definition is not obviously connected. -- \ “Everyone is entitled to their own opinions, but they are not | `\ entitled to their own facts.” —US Senator Pat Moynihan | _o__) | Ben Finney

On Fri, 2011-10-14 at 10:01 +1000, Nick Coghlan wrote:

On Fri, Oct 14, 2011 at 7:01 AM, Greg Ewing wrote:

...

Withdrawing PEP 3150 altogether seems like an over- reaction to me. A lot of its problems would go away if the idea of trying to make the names local to the suite were dropped. That part doesn't seem particularly important to me -- we manage to live without the for-loop putting its variable into a local scope, even though it would be tidier if it did.

So, keep the PEP 3150 syntax, but don't make the inner suite special aside from the out of order execution?

While that would work, it still feels overly heavy for what I consider the primary use case of the construct:

sorted_list = sorted(original, key=key_func) given: def key_func(item): return item.attr1, item.attr2

The heart of the problem is that the name 'key_func' is repeated twice, encouraging short, cryptic throwaway names. Maybe I'm worrying too much about that, though - it really is the out of order execution that is needed in order to let the flow of the Python code match the way the developer is thinking about their problem.

Yes, you are worrying too much about this. :-) I like things that can be taken apart and put together again in a program and by a program. And even put together in new ways by a program. The special syntax and requirement that they be localized together doesn't fit that. Which means each time you want something similar to it, but with maybe with a small variation, you must rewrite the whole thing by hand. (Yes, we saved a few keystrokes each time, but...) To me that is going backwards and is a counter point to the purpose of programming. Hey lets keep those programmers working! JK ;-)

I'll note that the evolution from PEP 3150 (as shown above) to PEP 403 went as follows:

1. Make the inner suite a true anonymous function with the signature on the header line after the 'given' clause. Reference the function via '@' since it is otherwise inaccessible.

sorted_list = sorted(original, key=@) given (item): return item.attr1, item.attr2

2. Huh, that 'given' keyword doesn't scream 'anonymous function'. How about 'def' instead?

sorted_list = sorted(original, key=@) def (item): return item.attr1, item.attr2

3. Huh, that looks almost exactly like decorator prefix syntax. And the callable signature is way over on the RHS. What if we move it to the next line?

sorted_list = sorted(original, key=@) def (item): return item.attr1, item.attr2

4. We may as well let people add a name for debugging purposes, and it's less work to just make it compulsory to match existing syntax. By keeping the shorthand symbolic reference, we get the best of both worlds: a descriptive name for debugging purposes, a short local name for ease of use.

sorted_list = sorted(original, key=@) def key_func(item): return item.attr1, item.attr2

5. Well, the parser won't like that and it's backwards incompatible anyway. We need something to flag the prefix line as special. ':' will do.

:sorted_list = sorted(original, key=@) def key_func(item): return item.attr1, item.attr2

6. Keywords are better than symbols, so let's try that instead

postdef sorted_list = sorted(original, key=def) def key_func(item): return item.attr1, item.attr2

PEP 403 really is just an extension of the principles behind decorators at its heart, so I think it makes sense for those semantics to have a decorator-style syntax. If we want to revert back to using an indented suite, than I think it makes more sense to go all the way back to PEP 3150 and discuss the relative importance of "out of order execution" and "private scope to avoid namespace pollution".

Decorators can be moved out of the way and reused multiple times. I don't think this syntax can do that. It locks the two pieces together so they can't be used separately. Using your sort example. Lets say we rewrite that concept so it's more general using a decorator that can be reused. In this case, the key isn't the reusable part. It's the part that changes when we do sorts, so it's the part we put the decorator on to create a new sorted variation. def sorted_with(key): def _(seq): return sorted(seq, key=key) return _ @sorted_with def key_sorted(item): return item.attr1, item.attr2 new_list = key_sorted(original_list) We can reuse the sorted_with decorator with as may key functions as we want. That reuse is an important feature of decorators. Cheers, Ron

On Oct 13, 2011, at 2:01 PM, Nick Coghlan wrote:

On Fri, Oct 14, 2011 at 7:01 AM, Greg Ewing wrote:

...

Withdrawing PEP 3150 altogether seems like an over- reaction to me. A lot of its problems would go away if the idea of trying to make the names local to the suite were dropped. That part doesn't seem particularly important to me -- we manage to live without the for-loop putting its variable into a local scope, even though it would be tidier if it did.

So, keep the PEP 3150 syntax, but don't make the inner suite special aside from the out of order execution?

To me the limitations of 403 are its strength. I don't want to see people doing crazy inside-out code. Let's say we had a keyword OoO for "out of order": OoO result = flange(thingie): OoO thing = doohickie(majiger): part_a = source / 2 part_b = source ** 2 majiger = blender(part_a, part_b) This violates the spirit of One Obvious Way from my perspective. If you have an OoO keyword, for every sequence of code you end up asking yourself, "Hmm, would this make more sense if I did it backwards or forwards?…" That will lead to a bunch of style guide wars and in many cases result, worse readability. By comparison, PEP 403 allows you to do one thing (or two if classes are allowed). You can place a callback underneath the receiver of the callback. That's more or less it. The answer to "when should I use PEP 403?" is very clear: "When you're never going to pass a callback function to anything other than the one receiver." In the same way, "When do I use an @decorator?" has a clear answer: "When you're never going to want to use the undecorated form." I can imagine a little bit of tweaking around the edges for this (see below), but otherwise, I think the core theory of PEP 403 is just right. Slight tweak: postdef items = sorted(items, key=@keyfunc(item)): item = item.lower() item = item.replace(" ", "") ... return item In this case "keyfunc" is the name of the function (not that anyone will ever see it) and (item) is the argument list. I'm not convinced this tweak is better than having a full def on the line below, but it's worth considering.

On Fri, Oct 14, 2011 at 1:07 PM, Carl M. Johnson wrote:

To me the limitations of 403 are its strength. I don't want to see people doing crazy inside-out code.

This has long been my fear with PEP 3150, but I'm beginning to wonder if that fear is overblown.

Let's say we had a keyword OoO for "out of order":

OoO result = flange(thingie):        OoO thing = doohickie(majiger):                part_a = source / 2                part_b = source ** 2                majiger = blender(part_a, part_b)

But now consider this: part_a = source / 2 part_b = source ** 2 majiger = blender(part_a, part_b) thingie = doohickie(majiger) result = flange(thingie) And now with a couple of operations factored out into functions: def make_majiger(source): part_a = source / 2 part_b = source ** 2 return blender(part_a, part_b) def make_thingie(source): return doohickie(make_majigger(source)) result = flange(make_thingie(source)) All PEP 3150 is allowing you to do is indent stuff that could potentially be factored out into a function at some point, without forcing you to factor it out *right now*: result = flange(thingie) given: thingie = doohickie(majiger) given: part_a = source / 2 part_b = source ** 2 majiger = blender(part_a, part_b) So is the "inline vs given statement" question really any more scary than the "should I factor this out into a function" question? I expect the style guidelines will boil down to: - if you're writing a sequential process "first we do A, then we do B, then we do C, etc", use normal inline code - if you're factoring out subexpressions from a single step in an algorithm, use a given statement - as a general rule, given statements should be used for code that could reasonably be factored out into its own function Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

Nick Coghlan wrote:

So is the "inline vs given statement" question really any more scary than the "should I factor this out into a function" question?

Especially since the factored-out functions could be written either before or after the place where they're used, maybe not even on the same page, and maybe not even in the same source file...

I expect the style guidelines will boil down to: ... - as a general rule, given statements should be used for code that could reasonably be factored out into its own function

Also perhaps: - If you're assigning some things to local names and then using them just once in a subsequent expression, consider using a 'given' statement. I would also be inclined to add that if you're using the 'given' style in a particular function, you should use it *consistently*. For instance, consider the earlier example

result = flange(thingie) given: thingie = doohickie(majiger) given: part_a = source / 2 part_b = source ** 2 majiger = blender(part_a, part_b)

Why have we used the 'given' style for 'thingie' and 'majiger', but fallen back on the sequential style for 'part_a' and 'part_b'? It would be more consistent to write it as result = flange(thingie) given: thingie = doohickie(majiger) given: majiger = blender(part_a, part_b) given: part_a = source / 2 part_b = source ** 2 The flow of data from one place to another within the function is now very clear: source -----> part_a -----> blender --> doohickie --> flange | | \--> part_b --/ -- Greg

Nick Coghlan wrote:

On Fri, Oct 14, 2011 at 11:10 AM, Ron Adam wrote:

...

def sorted_with(key): def _(seq): return sorted(seq, key=key) return _

@sorted_with def key_sorted(item): return item.attr1, item.attr2

Seriously? You're suggesting that mess of symbols and indentation as a good answer to the programming task "I want to sort the items in this sequence according to the values of attr1 and attr2"?

I don't think that example is as well-constructed as it could be. The key_sorted function would be better named something like 'sort_by_attr1_and_attr2'. -- Greg

On Oct 13, 2011, at 8:56 PM, Greg Ewing wrote:

I would also be inclined to add that if you're using the 'given' style in a particular function, you should use it *consistently*. For instance, consider the earlier example

...

result = flange(thingie) given: thingie = doohickie(majiger) given: part_a = source / 2 part_b = source ** 2 majiger = blender(part_a, part_b)

Why have we used the 'given' style for 'thingie' and 'majiger', but fallen back on the sequential style for 'part_a' and 'part_b'?

That's more-or-less my point. The ability to write arbitrary code with given will create a lot of arguments about style. One person will use it in one place but not another, then someone will say, hey, let's make it more consistent, but then that's going too far… and it becomes an attractive nuisance for pointless rewrites.

Nick Coghlan wrote:

So, keep the PEP 3150 syntax, but don't make the inner suite special aside from the out of order execution?

That's right. If we're willing to accept the idea of the def in a postdef statement binding a name in the surrounding scope, then we've already decided that we don't care about polluting the main scope -- and it would make PEP 3150 a heck of a lot easier to both specify and implement. Having said that, I think there might be a way of implementing PEP 3150 with scoping and all that isn't too bad. The main difficulties seem to concern class scopes. Currently they're kept in a dict while they're being built, like function local scopes used to be before they were optimised. We could change that so that they're compiled in the same way as a normal function scope, and then use the equivalent of locals() at the end to build the class dict. The body of a 'given' statement could then be compiled as a separate function with access to the class scope. Nested 'def' and 'class' statements, on the other hand, would be compiled with the surrounding scope deliberately excluded. -- Greg

On Fri, 2011-10-14 at 20:03 +1300, Greg Ewing wrote:

Ron Adam wrote:

...

Class's are constructed top down. You create the framework and then fill in the parts.

Actually, they're not -- the class object is created *after* executing the class body! So this is a (small) precedent for writing things out of order when it helps.

I fairly often will write classes by writing the class header, then write the methods headers, and then go back and fill in the method bodies. The mental model for solving a problem doesn't have to match the exact computational order. Cheers, Ron

Nick Coghlan wrote:

The update to the PEP that I just pushed actually drops class statement support altogether (at least for now), but if it was still there, the above example would instead look more like:

postdef x = property(class.get, class.set, class.delete) class scope: def get(self): return __class__.attr def set(self, val): __class__.attr = val def delete(self): del __class__.attr

I like Greg's proposal to use the throw-away name: postdef x = property(scope.get, scope.set, scope.delete) class scope: def get(self): return __class__.attr def set(self, val): __class__.attr = val def delete(self): del __class__.attr ~Ethan~

On Fri, Oct 14, 2011 at 1:47 AM, Greg Ewing wrote:

Nick Coghlan wrote:

...

So, keep the PEP 3150 syntax, but don't make the inner suite special aside from the out of order execution?

That's right. If we're willing to accept the idea of the def in a postdef statement binding a name in the surrounding scope, then we've already decided that we don't care about polluting the main scope -- and it would make PEP 3150 a heck of a lot easier to both specify and implement.

Having said that, I think there might be a way of implementing PEP 3150 with scoping and all that isn't too bad.

The main difficulties seem to concern class scopes. Currently they're kept in a dict while they're being built, like function local scopes used to be before they were optimised.

We could change that so that they're compiled in the same way as a normal function scope, and then use the equivalent of locals() at the end to build the class dict. The body of a 'given' statement could then be compiled as a separate function with access to the class scope. Nested 'def' and 'class' statements, on the other hand, would be compiled with the surrounding scope deliberately excluded.

-- Greg _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

Perhaps we should avenge 3150's mortal wounding then. <wink> I'd vote for this in-order usage: given: a = 1 b = 2 def f(a=given.a, b=given.b): ... Notice that the contents of the "given" block are exposed on a special "given" name; the name would only be available in the statement attached to the given clause. The idea of a one-shot anonymous block is a part of the PEP 403 idea that really struck me. (Perhaps it was already implicit in the 3150 concept, but this thread sold it for me.) It seems like the CPython implementation wouldn't be that tricky but I'm _sure_ I'm missing something. Make the given block compile like a class body, but stop there and locally bind the temporary "given" to that resulting dictionary. Any use of "given.<name>" would be treated as a key lookup on the dictionary. In the case of function definitions with a given clause, don't close on "given". Instead, make a cell for each "given.<name>" used in the function body and tie each use to that cell. This will help to avoid superfluous lookups on "given". I like the in-order variant because it's exactly how you would do it now, without the cleanup after: a = 1 b = 2 def f(a=a, b=b): ... del a, b This reminds me of how the with statement pulled the "after" portion out, and of decorators. Not only that, but when you write a module, don't you write the most dependent statements (definitions even) at the bottom? First you'll write that part you care about. Then you'll write the dependent code _above_, often because of execution (or definition) order dependencies. It's like Greg said (below), though my understanding is that the common convention is to put the factored out code above, not below. If I'm wrong then I would gladly hear about it. Greg Ewing said:

Nick Coghlan wrote:

...

So is the "inline vs given statement" question really any more scary than the "should I factor this out into a function" question?

Especially since the factored-out functions could be written either before or after the place where they're used, maybe not even on the same page, and maybe not even in the same source file...

Also, the in-order given statement is easy to following when reading code, while the post-order one is less so. Here are some examples refactored from other posts in this thread (using in-order): given: def report_destruction(obj): print("{} is being destroyed".format(obj)) x = weakref.ref(obj, given.report_destruction) given: def pointless(): """A pointless worker thread that does nothing except serve as an example""" print("Starting") time.sleep(3) print("Ending") t = threading.Thread(target=given.pointless); t.start() given: len = len def double_len(x): print(x) return 2 * given.len(x) given: part_a = source / 2 part_b = source ** 2 majiger = blender(part_a, part_b) thingie = doohickie(majiger) result = flange(given.thingie) # or given: given: given: part_a = source / 2 part_b = source ** 2 majiger = blender(given.part_a, given.part_b) thingie = doohickie(given.majiger) result = flange(given.thingie) When writing these, you would normally just write the statement, and then fill in the blanks above it. Using "given", if you want to anonymize any statement on which the original depends, you stick it in the given clause. It stays in the same spot that you had it before you put a given clause. It's where I would expect to find it: before it gets used. Here are some unknowns that I see in the idea, which have been brought up before: 1. How would decorators mix with given clauses on function/class definitions? (maybe disallow?) 2. How could you introspect the code inside the given clause? (same as code in a function body?) 3. Would it make sense to somehow inspect the actual anonymous namespace that results from the given clause? 4. For functions, would there be an ambiguity to resolve? For that last one, take a look at this example: given: x = 1 def f(): given: x = 2 return given.x My intuition is that the local given clause would supersede the closed one (i.e. the outer one would be rendered unused code and no closure would have been compiled for it). However, I could also see this as resulting in a SyntaxError. Nick Coghlan wrote:

If we want to revert back to using an indented suite, than I think it makes more sense to go all the way back to PEP 3150 and discuss the relative importance of "out of order execution" and "private scope to avoid namespace pollution".

I'm just not seeing that relative importance. Nick, you've mentioned it on several occasions. Is it the following, which you mentioned in the PEP? Python's demand that the function be named and introduced before the operation that needs it breaks the developer's flow of thought. I know for a fact that Nick knows a lot more than me (and has been at this a lot longer), so I assume that I'm missing something here. The big advantage of the post-order given statement, that I see, is that you can do a one-liner: x = [given.len(i) for i in somebiglist] given: len = len vs. given: len = len x = [given.len(i) for i in somebiglist] Nick said:

Interestingly, the main thing I'm getting out of this discussion is more of an idea of why PEP 3150 has fascinated me for so long. I expect the outcome is going to be that 403 gets withdrawn and 3150 resuscitated :)

So far this thread has done the same for me. I like where 403 has taken us. The relationship between decorators and the PEP 403 syntax is also a really clever connection! Very insightful too (a frame's stack as a one-off pseudo-scope). And I agree with the sentiments of Nick's expectation. :) My preference is for PEP 3150, a case for which I hope I've made above. And, after thinking about it, I like the simplicity of PEP 3150 better. It has more of that "executable pseudo-code" feel. While explaining PEP 403 to a co-worker (without much Python under his belt), I had to use the 3150 syntax to explain to him how the 403 syntax worked and what it meant. He found the 3150 syntax much easier to read and understand. "Why don't you just use _that_?", he asked. Incidentally, he said the in-order variant is easier to read. ;) -eric

On Fri, Oct 14, 2011 at 1:01 PM, Jim Jewett wrote:

On Fri, Oct 14, 2011 at 2:28 PM, Eric Snow wrote:

...

On Fri, Oct 14, 2011 at 1:47 AM, Greg Ewing wrote:

...

Nick Coghlan wrote:

...

...

...

So, keep the PEP 3150 syntax, but don't make the inner suite special aside from the out of order execution?

...

  given:       a = 1       b = 2   def f(a=given.a, b=given.b):       ...

A few years ago, I would have liked this (except for the awkward repetition of given.*), because it would have felt like the function was cleaning up after itself.

Even now, I like it, because the extra indentation make it clear that a and b group with f.  Today, I do (or fail to do) that with vertical whitespace.

But I'm not sure it answers the original concerns.

...

...   First you'll write that part you care about.  Then you'll write the dependent code _above_,

It is precisely that going-back-up that we want to avoid, for readers as well as for writers.

I see what you mean. And when you are interacting with an object from an outer scope then it doesn't matter where the object was defined, as long as it happened execution-wise before you go to use it. However, in the same block, the object you are going to use must be defined _above_, since execution flows down. For example, a module will have "if __name__ == "__main__" at the bottom, if it has one. I guess I'm just used to code generally following that same pattern, even when not semantically necessary. In-order just _seems_ so natural and easy to understand. But on the other hand, I was able to figure out other out-of-order syntax in Python... Perhaps I simply need to adjust to out-of-order syntax. Out-of-order doesn't feel nearly as natural to me, but that may only be the consequence of my experience and not reflect its value. Presently I can only defer to others with more experience here to make a case for or against it. In the meantime, the in-order variant seems to me to be the better choice.

...

Also, the in-order given statement is easy to following when reading code, while the post-order one is less so.

I like a Table of Contents.  Sometimes I like an outline.  I can follow a bottom-up program, but it is usually better if I can first see "Here's the main point", *followed*by* "Here's how to do it."

...

1. How would decorators mix with given clauses on function/class definitions?  (maybe disallow?)

I would assume that they work just like they do now -- put them on the line right before the definition, at the same indent level.  That would be after the given clause semantically, and also literally if the given clause happens before the def.

Having decorators come outside the given clause would be a problem, because the given statements don't always return anything, let alone the right function.

Banning decorators would seem arbitrary enough that I would want to say "hmm, this isn't ready yet."

...

2. How could you introspect the code inside the given clause? (same as code in a function body?)

Either add a .given node to the function object, or treat it like any other module-level (or class-level) code outside a function.

...

   Python's demand that the function be named and introduced    before the operation that needs it breaks the developer's flow    of thought.

I think of this a bit like C function prototypes and header files. You get used to the hassle of having to repeat the prototypes, but it is still a bad thing.  Having the prototypes as a quick Table Of Contents or Index, on the other hand, is a good thing.

Code as a table of contents...hadn't considered it. It's an interesting idea. For large modules I've seen it mostly done in the module docstring, rather than in code. When I need a table of contents for some code I'll usually use dir(), help(), dedicated documentation, or docstrings directly (in that order). When I code I usually keep the number of objects in any given block pretty small, thus it's easy to scan through. But with other (equally/more valid) coding styles than mine (and other languages) I can see where code as a table of contents at the beginning would be helpful.

...

big advantage of the post-order given statement, that I see, is that you can do a one-liner:

Nah; if it really is a one-liner, then moving the definition up a line isn't that expensive.  The big advantage is when something is a level of detail that you don't want to focus on yet, but it is also (textually) big enough to cause a mental interruption.

That mental interruption is definitely to be avoided. In those cases I typically factor that code out into a function. Then the high-level execution is obvious in one place. I'm not seeing how a given statement (of either variant) would be helpful there; and I don't see how it would be an interruption in places I would use it. Thanks for the insights. I definitely want to get on the same page (either way) with those promoting the post-order given statement. I'm glad that Nick wrote PEP 403 because it's gotten me thinking about this. Regardless of the variant, I think the idea of one-off anonymous namespaces is worth figuring out. -eric

-jJ

On Fri, 2011-10-14 at 13:38 +1000, Nick Coghlan wrote:

All PEP 3150 is allowing you to do is indent stuff that could potentially be factored out into a function at some point, without forcing you to factor it out *right now*:

result = flange(thingie) given: thingie = doohickie(majiger) given: part_a = source / 2 part_b = source ** 2 majiger = blender(part_a, part_b)

I don't see it as much advantage for just a few lines of code where you can see the whole thing at once both visually and mentally. # determine source ... # Get flange result part_a = source / 2 part_b = source ** 2 majiger = blender(part_a, part_b) thingie = doohickie(majiger) result = flange(thingie) # do something with flange result ... With comments, I find that it's just as readable, mostly because it's a small enough chunk that the order isn't all that important. If the block of code was one used over again outside that context it would be factored out and you would have this. You could factor it out anyway if it helps make the surrounding code more efficient. # Determine source. ... # Get flange result = get_flange(source) # Do something with flange result ... For very large blocks of code, you might end up with many levels of extra indentation depending on how much nesting there is. To be clear. It's not a bad concept. It's a very nice way to express some problems. I'm just not sure it's needed in Python. On the other hand... The waiting for a block to complete to continue a statement reminds me of futures. http://en.wikipedia.org/wiki/Future_%28programming% 29#Implicit_vs_explicit It would be nice if something like this was also compatible with micro threading. I bet it would be a sure win-win if we could get that from this. (or at least a possibility of it.) Cheers, Ron

On Sun, Oct 16, 2011 at 9:01 PM, Steven D'Aprano wrote:

If it's not tested, how do you know it does what you need?

Because my script does what I want. Not every Python script is a production application. We lose sight of that point at our peril. Systems administrators, lawyers, financial analysts, mathematicians, scientists... they're not writing continuously integrated production code, they're not writing reusable modules, they're writing scripts to *get things done*. Python's insistence that every symbol be defined before it can be referenced gets in the way of that. Imagine how annoying Python would be if we got rid of while loops and insisted that anyone want to implement a while loop instead write a custom iterator to use with a for loop instead. It's taken me a long time to realise it, but that's exactly how the lack of multi-line lambda support can feel. PEP 403 was an attempt to address *just* that problem with an approach inspired by Ruby's blocks. That discussion convinced me that such a terse, unintuitive idiom could never be made "Pythonic" - but there's still hope for the more verbose, yet also more expressive, statement local namespace concept. Will there be times where someone writes a piece of code using a given statement, realises they need access to the internals in order to test them in isolation, and refactors the code accordingly? Absolutely, just as people take "if __name__ == '__main__'" blocks and move them into main functions to improve testability, break up large functions into smaller ones, add helper methods to classes, or take repeated list comprehensions and make a named function out of them. The problem with the status quo is that there are plenty of people (especially mathematicians and scientists) that *do* think declaratively, just as there are people that think in terms of functional programming rather than more typical imperative styles. Python includes constructs that are technically redundant with each other in order to allow people to express concepts in a way that makes sense to *them* (e.g. keeping functools.reduce around). Unless, that is, you think declaratively. Then Python isn't currently the language for you - go use Ruby or some other language with a construct that lets you supply additional details after the operation that needs them. To attack the question from a different perspective, how do I know that any loops in my code work? After all, my test code can't see my loops - it can only see the data structures they create and the functions that contain them. So the answer is because I tested data structures and I tested those functions. How could I test that a weakref callback did the right thing without direct access to the callback function? The same way I'd have to test it *anyway*, even if I *did* have direct access: by allocating an object, dropping all references to it, forcing a GC cycle. If we're not in the middle of deleting the object, testing the callback actually tells me nothing useful, since the application state is completely wrong. All that said... is your objection *only* to the "statement local" part of the proposal? That part is actually less interesting to me than the out of order execution part. However, our past experience with list comprehensions suggests that exposing the names bound inside the suite in the containing scope would be confusing in its own way. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Sun, Oct 16, 2011 at 8:30 PM, Steven D'Aprano wrote:

Nick Coghlan wrote:

...

On Sun, Oct 16, 2011 at 9:01 PM, Steven D'Aprano wrote:

...

...

If it's not tested, how do you know it does what you need?

...

Because my script does what I want. ... they're writing scripts to *get things done*.

"And it doesn't matter whether it's done correctly, so long as SOMETHING gets done!!!" <wink>

So they have only system tests, and not unit tests. (Or at least not to the level you would recommend.) But if the system tests pass, that really is enough. (And yes, there could be bugs exposed later by data that didn't show up in the system tests -- but that is still better than most of the other software in that environment.)

I'm not convinced that requiring coders to write:

given a, b c do f(a, b, c)

instead of

do f(a, b, c) given a, b, c

gets in the way of getting things done.

Nah; people can interrupt their train of thought to scroll up. Doing so doesn't improve their code, but it isn't a deal-breaker. Requiring that order does get in the way of reuse, because burying the lede [ do f(a,b,c) ] makes it a bit harder to find, so there is a stronger temptation to just fork another copy. -jJ

Am 17.10.2011 02:30, schrieb Steven D'Aprano:

Nick Coghlan wrote:

...

On Sun, Oct 16, 2011 at 9:01 PM, Steven D'Aprano wrote:

...

If it's not tested, how do you know it does what you need?

Because my script does what I want. Not every Python script is a production application. We lose sight of that point at our peril.

Systems administrators, lawyers, financial analysts, mathematicians, scientists... they're not writing continuously integrated production code, they're not writing reusable modules, they're writing scripts to *get things done*.

"And it doesn't matter whether it's done correctly, so long as SOMETHING gets done!!!" <wink>

Sorry, but even with the wink this is a dangerous statement. When I write a 40-line script to read, fit and plot a dataset, I don't add a unit test for that. On the contrary, I want to write that script as conveniently as possible. Python is as popular as it is in science *because* it allows that. People hate writing boilerplate just to do a small job.

I'm not convinced that requiring coders to write:

given a, b c do f(a, b, c)

instead of

do f(a, b, c) given a, b, c

gets in the way of getting things done.

I guess this means -0 from my side given: That said: I'm not sure this specific proposal would help. I agree with you and can't see that defining the function before using it is the wrong order. What does appeal to me is the potential for cleaner namespaces; while this doesn't matter so much in the function-locals (there it's mostly the OCD speaking), it would definitely helpful for module and class namespaces (where we routinely see "del" statements used to remove temporary names). I can believe that others see the "given" semantics as the natural order of things. This is probably why Haskell has both "let x = y in z" and "z where x = y" constructs to locally bind names, but there it fits well with the syntax, while in the case of Python, the "given: suite" feels somewhat out of place; we really have to think hard before adding new syntax -- one more thing that the "casual" users Nick mentioned have to grok. cheers, Georg

I don't understand all this angst about testing at all. We don't currently expect to be able to reach inside a function and test parts of it in isolation. PEP 3150 does nothing to change that. -- Greg

On Mon, Oct 17, 2011 at 5:52 PM, Greg Ewing wrote:

Nick Coghlan wrote:

...

Yeah, that's a large part of why I now think the given clause needs to be built on the same semantics that we already use internally for implicit out of order evaluation (i.e. decorators, comprehensions and generator expressions), such that it merely exposes the unifying mechanic underlying existing constructs rather than creating a completely new way of doing things.

I'm not sure what you mean by that. If you're talking about the implementation, all three of those use rather different underlying mechanics. What exactly do you see about these that unifies them?

Actually, comprehensions and generator expressions are almost identical in 3.x (they only differ in the details of the inner loop in the anonymous function). For comprehensions, the parallel with the proposed given statement would be almost exact: seq = [x*y for x in range(10) for y in range(5)] would map to: seq = _list_comp given _outermost_iter = range(10): _list_comp = [] for x in _outermost_iter: for y in range(5): _list_comp.append(x*y) And similarly for set and dict comprehensions: # unique = {x*y for x in range(10) for y in range(5)} unique = _set_comp given _outermost_iter = range(10): _set_comp = set() for x in _outermost_iter: for y in range(5): _set_comp.add(x*y) # map = {(x, y):x*y for x in range(10) for y in range(5)} map = _dict_comp given _outermost_iter = range(10): _anon = {} for x in _outermost_iter: for y in range(5): _anon[x,y] = x*y Note that this lays bare some of the quirks of comprehension scoping - at class scope, the outermost iterator expression can sometimes see names that the inner iterator expressions miss. For generator expressions, the parallel isn't quite as strong, since the compiler is able to avoid the redundant anonymous function involved in the given clause and just emit an anonymous generator directly. However, the general principle still holds: # gen_iter = (x*y for x in range(10) for y in range(5)) gen_iter = _genexp() given _outermost_iter = range(10): def _genexp(): for x in _outermost_iter: for y in range(5): yield x*y For decorated functions, the parallel is actually almost as weak as it is for classes, since so many of the expressions involved (decorator expressions, default arguments, annotations) get evaluated in order in the current scope and even a given statement can't reproduce the actual function statement's behaviour of not being bound at *all* in the current scope while decorators are being applied, even though the function already knows what it is going to be called:

...

...

def call(f): ... print(f.__name__) ... return f() ... @call ... def func(): ... return func.__name__ ... func Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in call File "<stdin>", line 3, in func NameError: global name 'func' is not defined

So it's really only the machinery underlying comprehensions that is being exposed by the PEP rather than anything more far reaching. Exposing the generator expression machinery directly would require the ability to turn the given clause into a generator (via a top level yield expression) and then a means to reference that from the header line, which gets us back into cryptic and unintuitive PEP 403 territory. Better to settle for the named alternative. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Mon, Oct 17, 2011 at 11:00 PM, Paul Moore wrote:

I'd expect the given suite to only contain name-definition statements (assignments, function and class definitions). Anything else should be at least bad practice, if not out and out illegal...

It would be perfectly legal, just inadvisable most of the time (as the suggested PEP 8 addition says, if your given clause gets unwieldy, it's probably a bad idea). However, those expansions are almost exactly what 3.x comprehensions look like from the interpreter's point of view. And the assignment between the given statement and the colon wasn't a typo - that was the explicit early binding of an expression evaluated in the outer scope. It's the way comprehensions behave, and I suspect it may be a good compromise for given statements as well (to make their scoping rules more consistent with the rest of the language, while still having them be vaguely usable at class scope) Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Mon, 2011-10-17 at 22:05 +1000, Nick Coghlan wrote:

On Mon, Oct 17, 2011 at 5:52 PM, Greg Ewing wrote:

...

Nick Coghlan wrote:

...

Yeah, that's a large part of why I now think the given clause needs to be built on the same semantics that we already use internally for implicit out of order evaluation (i.e. decorators, comprehensions and generator expressions), such that it merely exposes the unifying mechanic underlying existing constructs rather than creating a completely new way of doing things.

I'm not sure what you mean by that. If you're talking about the implementation, all three of those use rather different underlying mechanics. What exactly do you see about these that unifies them?

Actually, comprehensions and generator expressions are almost identical in 3.x (they only differ in the details of the inner loop in the anonymous function).

For comprehensions, the parallel with the proposed given statement would be almost exact:

seq = [x*y for x in range(10) for y in range(5)]

would map to:

seq = _list_comp given _outermost_iter = range(10): _list_comp = [] for x in _outermost_iter: for y in range(5): _list_comp.append(x*y)

Ok, here's a way to look at this that I think you will find interesting. It looks to me that the 'given' keyword is setting up a local name_space in the way it's used. So rather than taking an expression, maybe it should take a mapping. (which could be from an expression) mapping = dict(iter1=range(10), iter2=range(5)) given mapping: # mapping as local scope list_comp=[] for x in iter1: for y in iter2: list_comp.append(x*y) seq = mapping['list_comp'] (We could stop here.) This doesn't do anything out of order. It shows that statement local name space, and the out of order assignment are two completely different things. But let's continue... Suppose we use a two suite pattern to make getting values out easier. mapping = dict(iter1=range(10), iter2=range(5)) given mapping: list_comp=[] for x in iter1: for y in iter2: list_comp.append(x*y) get: list_comp as seq # seq = mapping['list_comp'] That saves us from having to refer to 'mapping' multiple times, especially if we need to get a lot of values from it. So now we can change the above to ... given dict(iter1=range(10), iter2=range(5)): list_comp=[] for x in iter1: for y in iter2: list_comp.append(x*y) get: list_comp as seq And then finally put the 'get' block first. get: list_comp as seq given dict(iter1=range(10), iter2=range(5)): list_comp=[] for x in iter1: for y in iter2: list_comp.append(x*y) Which is very close to the example you gave above, but more readable because it puts the keywords in the front. That also makes it more like a statement than an expression. Note, that if you use a named mapping with given, you can inspect it after the given block is done, and/or reuse it multiple times. I think that will be very useful for unittests. This creates a nice way to express some types of blocks that have local only names in pure python rather than just saying it's magic dust sprinkled here and there to make it work like that. (That doesn't mean we should actually change those, but the semantics could match.)

And similarly for set and dict comprehensions:

# unique = {x*y for x in range(10) for y in range(5)} unique = _set_comp given _outermost_iter = range(10): _set_comp = set() for x in _outermost_iter: for y in range(5): _set_comp.add(x*y)

get: set_comp as unique given dict(iter1=range(10), iter2=range(5)): set_comp = set() for x in iter1: for y in iter2: set_comp.add(x, y)

# map = {(x, y):x*y for x in range(10) for y in range(5)} map = _dict_comp given _outermost_iter = range(10): _anon = {} for x in _outermost_iter: for y in range(5): _anon[x,y] = x*y

get: dict_comp as map given dict(iter1=range(10), iter2=range(5)): dict_comp = {} for x in iter1: for y in iter2: dict_comp[x] = y I'm not sure if I prefer the "get" block first or last. given dict(iter1=range(10), iter2=range(5)): dict_comp = {} for x in iter1: for y in iter2: dict_comp[x] = y get: dict_comp as map But the order given/get order is a detail you can put to a final vote at some later time.

Note that this lays bare some of the quirks of comprehension scoping - at class scope, the outermost iterator expression can sometimes see names that the inner iterator expressions miss.

For generator expressions, the parallel isn't quite as strong, since the compiler is able to avoid the redundant anonymous function involved in the given clause and just emit an anonymous generator directly. However, the general principle still holds:

# gen_iter = (x*y for x in range(10) for y in range(5)) gen_iter = _genexp() given _outermost_iter = range(10): def _genexp(): for x in _outermost_iter: for y in range(5): yield x*y

given dict(iter1=range(10), iter2=range(5)): def genexp(): for x in iter1: for y in iter2: yield x*y get: genexp as gen_iter Interestingly, if we transform the given blocks a bit more we get something that is nearly a function. given Signature(<signature>).bind(mapping): ... function body ... get: ... return values ... ('def' would wrap it in an object, and give it a name.) So it looks like it has potential to unify some underlying mechanisms as well as create a nice local only statement space. What I like about it is that it appears to complement python very well and doesn't feel like it's something tacked on. I think having given take a mapping is what did that for me. Cheers, Ron

For decorated functions, the parallel is actually almost as weak as it is for classes, since so many of the expressions involved (decorator expressions, default arguments, annotations) get evaluated in order in the current scope and even a given statement can't reproduce the actual function statement's behaviour of not being bound at *all* in the current scope while decorators are being applied, even though the function already knows what it is going to be called:

It's hard to beat a syntax that is only one character long. ;-)

...

...

...

def call(f): ... print(f.__name__) ... return f() ... @call ... def func(): ... return func.__name__ ... func Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in call File "<stdin>", line 3, in func NameError: global name 'func' is not defined

So it's really only the machinery underlying comprehensions that is being exposed by the PEP rather than anything more far reaching.

Exposing the generator expression machinery directly would require the ability to turn the given clause into a generator (via a top level yield expression) and then a means to reference that from the header line, which gets us back into cryptic and unintuitive PEP 403 territory. Better to settle for the named alternative.

Cheers, Nick.

Regarding out of order execution: how is pep 3150 any worse than "x = y if z else w"? Apart from the fact than this sort of statement is usually quite short, the middle part still gets executed before the start. Either way, out of order execution hurts readability. There's a reason that

even mathematicians usually define terms before using them.

On the other hand, I have read countless scientific papers which define functions along the lines of "this phenomenon can be represented by the equation x=2sin(yz), where x is the natural frequency, y is the height of the giraffe and z is the number of penguins". Just saying... Changing the subject slightly, I haven't studied the details of the proposed grammar, but if given can be used in simple statements, this implies that the following would be possible: def function_with_annotations() -> x given: x = 4: function_body() I assume I'm not alone in thinking that this is a really bad idea (if the current grammar does actually allow it). However, "given" in def statements would be a great way of dealing with the recently discussed question of how to define function variables at definition time: def function() given (len=len): ... David

On Wed, Oct 19, 2011 at 7:43 PM, Terry Reedy wrote:

On 10/19/2011 2:44 AM, David Townshend wrote:

...

Regarding out of order execution: how is pep 3150 any worse than "x = y if z else w"?

A lot of people did not like that version of a conditional expression and argued against it. If it is used to justify other backwards constructs, I like it even less.

I happened to be doing a lot of string formatting today, and it occurred to me that because Python doesn't allow implicit string interpolation, *all* string formatting is based on forward references - we define placeholders in the format strings and only fill them in later during the actual formatting call. PEP 3150 is essentially just about doing that for arbitrary expressions by permitting named placeholders that are filled in from the following suite. I agree that using any parallels with conditional expressions as justification for out of order execution is a terrible idea, though. The chosen syntax was definitely a case of "better than the error prone status quo and not as ugly as the other alternatives proposed" rather than "oh, this is a wonderful way to do it" :P Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia