[Python-ideas] ScopeGuardStatement/Defer Proposal

[Guido van Rossum]

On Sat, Feb 18, 2012 at 5:27 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Sun, Feb 19, 2012 at 9:57 AM, Nathan Rice
> <nathan.alexander.rice at gmail.com> wrote:
>> I enjoy writing python a lot, and would prefer to use it rather than
>> ruby/lisp/java/etc in most cases. My suggestions come from
>> frustrations that occur when using python in areas where the right
>> answer is probably just to use a different language.  If I knew that
>> what I wanted was at odds with the vision for python, I would have
>> less of an issue just accepting circumstances, and would just get to
>> work rather than sidetracking discussions on this list.
>
> The core problem comes down to the differences between Guido's
> original PEP 340 idea (which was much closer in power to Ruby's
> blocks, since it was a new looping construct that allowed 0-or-more
> executions of the contained block) and the more constrained with
> statement that is defined in PEP 343 (which will either execute the
> body once or throw an exception, distinguishing it clearly from both
> the looping constructs and if statements).
>
> The principle Guido articulated when making that decision was:
> "different forms of flow control should look different at the point of
> invocation".
>
> So, where a language like Ruby just defines one protocol (callbacks,
> supplemented by anonymous blocks that run directly in the namespace of
> the containing function) and uses it for pretty much *all* flow
> control (including all their loop constructs), Python works the other
> way around, defining *different* protocols for different patterns of
> invocation.
>
> This provides a gain in readability on the Python side. When you see
> any of the following in Python:
>
>   @whatever()
>   def f():
>        pass
>
>    with whatever():
>        # Do something!
>
>    for x in whatever():
>        # Do something!
>
> It places a lot of constraints on the nature of the object returned by
> "whatever()" - even without knowing anything else about it, you know
> the first must return a decorator, the second a context manager, and
> the third an iterable. If that's all you need to know at this point in
> time, you don't need to worry about the details - the local syntax
> tells you the important things you need to know about the flow
> control.
>
> In Ruby, though, all of them (assuming it isn't actually important
> that the function name be bound locally) could be written like this:
>
>    whatever() do:
>        # Do something!
>    end
>
> Is it a context manager? An iterable? Some other kind of callback?
> There's nothing in the syntax to tell you that - you're relying on
> naming conventions to provide that information (like the ".foreach"
> convention for iteration methods). That approach can obviously work
> (otherwise Ruby wouldn't be as popular as it is), but it *does* make
> it harder to pick up a piece of code and understand the possible
> control flows without looking elsewhere.
>
> However, this decision to be explicit about flow control for the
> benefit of the *reader* brings with it a high *cost* on the Python
> side for the code *author*: where Ruby works by defining a nice syntax
> and semantics for callback based programming and building other
> language constructs on top of that, Python *doesn't currently have* a
> particularly nice general purpose native syntax for callback based
> programming.
>
> Decorators do work in many cases (especially simple callback
> registration), but they sometimes feel wrong because they're mainly
> designed to modify how a function is defined, not implement key
> program flow control constructs. However, their flexibility shouldn't
> be underestimated, and the CallbackStack API is designed to help
> Python developers push decorators and context managers closer to those
> limits *without* needing new language constructs. By decoupling the
> callback stack from the code layout, it gives you full *programmatic*
> control of the kinds of things context managers can help with when you
> know in advance exactly what you want to do.
>
> *If* CallbackStack proves genuinely popular (and given the number of
> proposals I have seen along these lines, and the feedback I have
> received on ContextStack to date, I expect it will), and people start
> to develop interesting patterns for using it, *then* we can start
> looking at the possibility of dedicated syntax to streamline
> particular use cases (just as the with statement itself was designed
> to streamline various use cases of the more general try statement).

Very lucid explanation, Nick. (I also liked your blog post that you
referenced in a previous message, which touches upon the same issues.)

Apparently I don't seem to like flow control constructs formed by
"quoting" (in Lisp terms) a block of code and leaving its execution to
some other party, with the exception of explicit function definitions.
Maybe a computer-literate psychoanalyst can do something with this...

To this day I am having trouble liking event-based architectures -- I
do see a need for them, but I immediately want to hide their
mechanisms and offer a *different* mechanism for most use cases. See
e.g. the (non-thread-based) async functionality I added to the new App
Engine datastore client, NDB:
https://docs.google.com/document/pub?id=1LhgEnZXAI8xiEkFA4tta08Hyn5vo4T6HSGLFVrP0Jag
. Deep down inside it has an event loop, but this is hidden by using
Futures, which in turn are mostly wrapped in tasklets , i.e.
yield-based coroutines. I expect that if I were to find a use for
Twisted, I'd do most of my coding using its so-called inlineCallbacks
mechanism (also yield-based coroutines). When I first saw Monocle,
which offers a simplified coroutine-based API on top of (amongst
others) Twisted, I thought it was a breath of fresh air (NDB is
heavily influenced by it).

I've probably (implicitly) trained most key Python developers and
users to think similarly, and Python isn't likely to morph into Ruby
any time soon. It's easy enough to write an event-based architecture
in Python (see Twisted and Tornado); but an event loop is never going
to be the standard way to solve all your programming problems in
Python.

I do kind of like the 'defer' idea that started this thread (even if I
had syntactic quibbles with it that already came up before the thread
was derailed), but I notice that it is a far cry from an event-driven
architecture -- like the referenced counterparts in Go and D, 'defer'
blocks are not anonymous functions that can be passed off to arbitrary
other libraries for possibly later and/or repeated execution -- they
are a way to specify out-of-order execution within the current scope,
which "tames" them enough to be acceptable from my perspective. Though
they may also not be powerful enough to be convincing as a new
feature, since you can do everything they can do by rearranging the
code of your function somewhat and carefully using try/finally.

-- 
--Guido van Rossum (python.org/~guido)