[Python-Dev] anonymous blocks as scope-collapse: detailed proposal

[Jim Jewett]

Based on Guido's opinion that caller and callee should both be
marked, I have used keywords 'include' and 'chunk'.  I therefore
call them "Chunks" and "Includers".

Examples are based on

(1)  The common case of a simple resource manager.  e.g.
http://mail.python.org/pipermail/python-dev/2005-April/052751.html

(2)  Robert Brewer's Object Relational Mapper
http://mail.python.org/pipermail/python-dev/2005-April/052924.html
which uses several communicating Chunks in the same Includer, and
benefits from Includer inheritance.

Note that several cooperating Chunks may use the same name
(e.g. old_children) to refer to the same object, even though
that object is never mentioned by the Includer.

It is possible for the same code object to be both a Chunk and
an Includer.  Its own included sub-Chunks also share the top
Includer's namespace.

Chunks and Includers must both be written in pure python,
because C frames cannot be easily manipulated.  They can
of course call or be called (as a unit) by extension modules.

I have assumed that Chunks should not take arguments.  While
arguments are useful ("Which pattern should I match against
on this inclusion?"), the same functionality *can* be had by
binding a known name in the Includer.  When that starts to get
awkward, it is a sign that you should be using separate
namespaces (and callbacks, or value objects).

"self" and "cls" are just random names to a Chunk, though
using them for any but the conventional meaning will be as
foolhardy as it is in a method.

Chunks are limited to statement context, as they do not return
a value.  

Includers must provide a namespace.  Therefore a single inclusion
will turn the entire nearest enclosing namespace into an Includer.
    ?  Should this be limited to nearest enclosing function or
       method?  I can't think of a good use case for including
       directly from class definition or module toplevel, except
       registration.  And even then, a metaclass might be better.

Includers may only be used in a statement context, as the Chunks
must be specified in a following suite.  (It would be possible to
skip the suite if all Chunk names are already bound, but I'm not
sure that is a good habit to encourage -- so initially forbid it.)

Chunks are defined without a (), in analogy to parentless classes.
They are included (called) with a (), so that they can remain first
class objects.

Example Usage
=============

def withfile(filename, mode='r'):
    """Close the file as soon we're done.

    This frees up file handles sooner.  This is particularly important
    under Jython, or if you are using files in cyclic structures."""
    openfile = open(filename, mode)
    try:
        include fileproc()  # keyword 'include' prevents XXX_FAST optimization
    finally:
        openfile.close()

chunk nullreader:           # callee Chunk defined for reuse
    for line in openfile:
        pass

withfile("testr.txt"):      # Is this creation of a new block-starter a problem?
    fileproc=nullreader     # Using an external Chunk object

withfile("testw.txt", "w"):
    chunk fileproc:         # Providing an "inline" Chunk
        openfile.write("Line 1")

#   If callers must be supported in expression context
#fileproc=nullreader
#withfile("tests.txt")      # Resolve Chunk name from caller's default
                            # binding, which in this case defaults back
                            # to the current globals.
                            # Is this just asking for trouble?
                        

            
class ORM(object):

    chunk nullchunk:                # The extra processing is not always needed.
        pass
    begin=pre=post=end=nullchunk    # Default to no extra processing

    def __set__(self, unit, value):
        include self.begin()
        if self.coerce:
            value = self.coerce(unit, value)
        oldvalue = unit._properties[self.key]
        if oldvalue != value:
            include self.pre()
            unit._properties[self.key] = value
            include self.post()
        include self.end()

class TriggerORM(ORM):
    chunk pre:
        include super(self,TriggerORM).pre()    # self was bound by __set__
        old_children = self.children()          # inject new variable
    
    chunk post:
        include super(self,TriggerORM).post()
        for child in self.children():
            if child not in old_children:       # will see pre's binding
                notify_somebody("New child %s" % child)


As Robert Brewer said, 

> The above is quite ugly written with callbacks (due to
> excessive argument passing), and is currently fragile
> when overriding __set__ (due to duplicated code).

How to Implement
----------------

The Includer cannot know which variables a Chunk will use (or
inject), so the namespace must remain a dictionary.  This precludes
use of the XXX_FAST bytecodes.  But as Robert pointed out, avoiding
another frame creation/destruction will compensate somewhat.

Two new bytecodes will be needed to handle the jump and return to
a different bytecode string without setting up or tearing down a
new frame.  Position in the Includer bytecode will need to be kept
in a stack, though it might make sense to use a frame variable
instead of the execution stack.

With those two exceptions, the Includer and Chunk are both
composed entirely of valid statements that can already be
compiled to ordinary bytecode.

-jJ