[Python-ideas] Statement local functions and classes (aka PEP 3150 is dead, say 'Hi!' to PEP 403)

[Eric Snow]

On Fri, Oct 14, 2011 at 1:47 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Nick Coghlan wrote:
>
>> So, keep the PEP 3150 syntax, but don't make the inner suite special
>> aside from the out of order execution?
>
> That's right. If we're willing to accept the idea of the
> def in a postdef statement binding a name in the surrounding
> scope, then we've already decided that we don't care about
> polluting the main scope -- and it would make PEP 3150 a
> heck of a lot easier to both specify and implement.
>
> Having said that, I think there might be a way of
> implementing PEP 3150 with scoping and all that isn't too
> bad.
>
> The main difficulties seem to concern class scopes.
> Currently they're kept in a dict while they're being
> built, like function local scopes used to be before
> they were optimised.
>
> We could change that so that they're compiled in the
> same way as a normal function scope, and then use the
> equivalent of locals() at the end to build the class
> dict. The body of a 'given' statement could then be
> compiled as a separate function with access to the
> class scope. Nested 'def' and 'class' statements, on
> the other hand, would be compiled with the surrounding
> scope deliberately excluded.
>
> --
> Greg
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>

Perhaps we should avenge 3150's mortal wounding then.  <wink>  I'd
vote for this in-order usage:

   given:
       a = 1
       b = 2
   def f(a=given.a, b=given.b):
       ...

Notice that the contents of the "given" block are exposed on a special
"given" name; the name would only be available in the statement
attached to the given clause.  The idea of a one-shot anonymous block
is a part of the PEP 403 idea that really struck me.  (Perhaps it was
already implicit in the 3150 concept, but this thread sold it for me.)

It seems like the CPython implementation wouldn't be that tricky but
I'm _sure_ I'm missing something.  Make the given block compile like a
class body, but stop there and locally bind the temporary "given" to
that resulting dictionary.  Any use of "given.<name>" would be treated
as a key lookup on the dictionary.

In the case of function definitions with a given clause, don't close
on "given".  Instead, make a cell for each "given.<name>" used in the
function body and tie each use to that cell.  This will help to avoid
superfluous lookups on "given".

I like the in-order variant because it's exactly how you would do it
now, without the cleanup after:

   a = 1
   b = 2
   def f(a=a, b=b):
       ...
   del a, b

This reminds me of how the with statement pulled the "after" portion
out, and of decorators.  Not only that, but when you write a module,
don't you write the most dependent statements (definitions even) at
the bottom?   First you'll write that part you care about.  Then
you'll write the dependent code _above_, often because of execution
(or definition) order dependencies.  It's like Greg said (below),
though my understanding is that the common convention is to put the
factored out code above, not below.  If I'm wrong then I would gladly
hear about it.

Greg Ewing said:
> Nick Coghlan wrote:
>> So is the "inline vs given statement" question really any more scary
>> than the "should I factor this out into a function" question?
>
> Especially since the factored-out functions could be
> written either before or after the place where they're
> used, maybe not even on the same page, and maybe not
> even in the same source file...

Also, the in-order given statement is easy to following when reading
code, while the post-order one is less so.

Here are some examples refactored from other posts in this thread
(using in-order):

   given:
      def report_destruction(obj):
          print("{} is being destroyed".format(obj))
   x = weakref.ref(obj, given.report_destruction)

   given:
      def pointless():
         """A pointless worker thread that does nothing except serve
as an example"""
         print("Starting")
         time.sleep(3)
         print("Ending")
   t = threading.Thread(target=given.pointless); t.start()

   given: len = len
   def double_len(x):
       print(x)
       return 2 * given.len(x)

   given:
       part_a = source / 2
       part_b = source ** 2
       majiger = blender(part_a, part_b)
       thingie = doohickie(majiger)
   result = flange(given.thingie)

   # or

   given:
       given:
           given:
               part_a = source / 2
               part_b = source ** 2
           majiger = blender(given.part_a, given.part_b)
       thingie = doohickie(given.majiger)
   result = flange(given.thingie)

When writing these, you would normally just write the statement, and
then fill in the blanks above it.  Using "given", if you want to
anonymize any statement on which the original depends, you stick it in
the given clause.  It stays in the same spot that you had it before
you put a given clause.  It's where I would expect to find it: before
it gets used.

Here are some unknowns that I see in the idea, which have been brought
up before:

1. How would decorators mix with given clauses on function/class
definitions?  (maybe disallow?)
2. How could you introspect the code inside the given clause? (same as
code in a function body?)
3. Would it make sense to somehow inspect the actual anonymous
namespace that results from the given clause?
4. For functions, would there be an ambiguity to resolve?

For that last one, take a look at this example:

   given:
       x = 1
   def f():
       given:
           x = 2
       return given.x

My intuition is that the local given clause would supersede the closed
one (i.e. the outer one would be rendered unused code and no closure
would have been compiled for it).  However, I could also see this as
resulting in a SyntaxError.


Nick Coghlan wrote:
> If we want to revert back to using
> an indented suite, than I think it makes more sense to go all the way
> back to PEP 3150 and discuss the relative importance of "out of order
> execution" and "private scope to avoid namespace pollution".

I'm just not seeing that relative importance.  Nick, you've mentioned
it on several occasions.  Is it the following, which you mentioned in
the PEP?

    Python's demand that the function be named and introduced
    before the operation that needs it breaks the developer's flow
    of thought.

I know for a fact that Nick knows a lot more than me (and has been at
this a lot longer), so I assume that I'm missing something here.  The
big advantage of the post-order given statement, that I see, is that
you can do a one-liner:

   x = [given.len(i) for i in somebiglist] given: len = len

vs.

   given: len = len
   x = [given.len(i) for i in somebiglist]


Nick said:
> Interestingly, the main thing I'm getting out of this discussion is
> more of an idea of why PEP 3150 has fascinated me for so long. I
> expect the outcome is going to be that 403 gets withdrawn and 3150
> resuscitated :)

So far this thread has done the same for me.  I like where 403 has
taken us.  The relationship between decorators and the PEP 403 syntax
is also a really clever connection!  Very insightful too (a frame's
stack as a one-off pseudo-scope).  And I agree with the sentiments of
Nick's expectation.  :)  My preference is for PEP 3150, a case for
which I hope I've made above.

And, after thinking about it, I like the simplicity of PEP 3150
better.  It has more of that "executable pseudo-code" feel.  While
explaining PEP 403 to a co-worker (without much Python under his
belt), I had to use the 3150 syntax to explain to him how the 403
syntax worked and what it meant.  He found the 3150 syntax much easier
to read and understand.  "Why don't you just use _that_?", he asked.
Incidentally, he said the in-order variant is easier to read.  ;)

-eric