[Python-Dev] Another Anonymous Block Proposal

[Jason Diamond]

Hi.

I hope you don't mind another proposal. Please feel free to tear it apart.

A limitation of both Ruby's block syntax and the new PEP 340 syntax is 
the fact that they don't allow you to pass in more than a single 
anonymous block parameter. If Python's going to add anonymous blocks, 
shouldn't it do it better than Ruby?

What follows is a proposal for a syntax that allows passing multiple, 
anonymous callable objects into another callable. No new protocols are 
introduced and none of it is tied to iterators/generators which makes it 
much simpler to understand (and hopefully simpler to implement).

This is long and the initial syntax isn't ideal so please bear with me 
as I move towards what I'd like to see.

The Python grammar would get one new production:

    do_statement ::=
        "do" call ":" NEWLINE
        ( "with" funcname "(" [parameter_list] ")" ":" suite )*

Here's an example using this new "do" statement:

    do process_file(path):
    with process(file):
        for line in file:
            print line

That would translate into:

    def __process(file):
        for line in file:
            print line
    process_file(path, process=__process)

Notice that the name after each "with" keyword is the name of a 
parameter to the function being called. This will be what allows 
multiple block parameters.

The implementation of `process_file` could look something like:

    def process_file(path, process):
        try:
            f = file(path)
            process(f)
        finally:
            if f:
                f.close()

There's no magic in `process_file`. It's just a function that receives a 
callable named `process` as a parameter and it calls that callable with 
one parameter.

There's no magic in the post-translated code, either, except for the 
temporary `__process` definition which shouldn't be user-visible.

The magic comes when the pre-translated code gets each "with" block 
turned into a hidden, local def and passed in as a parameter to 
`process_file`.

This syntax allows for multiple blocks:

    do process_file(path):
    with process(file):
        for line in file:
            print line
    with success():
        print 'file processed successfully!'
    with error(exc):
        print 'an exception was raised during processing:', exc

That's three separate anonymous block parameters with varying number of 
parameters in each one.

This is what `process_file` might look like now:

    def process_file(path, process, success=None, error=None):
        try:
            try:
                f = file(path)
                process(f)
                if success:
                    success(()
            except:
                if error:
                    error(sys.exc_info())
                raise
        finally:
            if f:
                f.close()

I'm sure that being able to pass in multiple, anonymous blocks will be a 
huge advantage.

Here's an example of how Twisted might be able to use multiple block 
parameters:

    d = do Deferred():
    with callback(data): ...
    with errback(failure): ...

(After typing that in, I realized the do_statement production needs an 
optional assignment part.)

There's nothing requiring that anonymous blocks be used for looping. 
They're strictly parameters which need to be callable. They can, of 
course, be called from within a loop:

    def process_lines(path, process):
        try:
            f = file(path)
            for line in f:
                process(line)
        finally:
            if f:
                f.close()

    do process_lines(path):
    with process(line):
        print line

Admittedly, this syntax is pretty bulky. The "do" keyword is necessary 
to indicate to the parser that this isn't a normal call--this call has 
anonymous block parameters. Having to prefix each one of these 
parameters with "with" is just following the example of "if/elif/else" 
blocks. An alternative might be to use indentation the way that class 
statements "contain" def statements:

    do_statement ::=
        "do" call ":" NEWLINE
        INDENT
            ( funcname "(" [parameter_list] ")" ":" suite )*
        DEDENT

That would turn our last example into this:

    do process_lines(path):
        process(line):
            print line

The example with the `success` and `error` parameters would look like this:

    do process_file(path):
        process(file):
            for line in file:
                print line
        success():
            print 'file processed successfully!'
        error(exc):
            print 'an exception was raised during processing:', exc

To me, that's much easier to see that the three anonymous block 
statements are part of the "do" statement.

It would be ideal if we could even lose the "do" keyword. I think that 
might make the grammar ambiguous, though. If it was possible, we could 
do this:

    process_file(path):
        process(file):
            for line in file:
                print line
        success():
            print 'file processed successfully!'
        error(exc):
            print 'an exception was raised during processing:', exc

Now the only difference between a normal call and a call with anonymous 
block parameters would be the presence of the trailing colon. I could 
live with the "do" keyword if this can't be done, however.

The only disadvantage to this syntax that I can see is that the simple 
case of opening a file and processing it is slightly more verbose than 
it is in Ruby. This is Ruby:

    File.open_and_process("testfile", "r") do |file|
        while line = file.gets
            puts line
        end
    end

This would be the Python equivalent:

    do open_and_process("testfile", "r"):
        process(file):
            for line in file:
                print line

It's one extra line in Python (I'm not counting lines that contain 
nothing but "end" in Ruby) because we have to specify the name of the 
block parameter. The extra flexibility that the proposed syntax has 
(being able to pass in multiple blocks) is worth this extra line, in my 
opinion.

If we wanted to optimize even further for this case, however, we could 
allow for an alternate form of the "do" statement that lets you only 
specify one anonymous block parameter. Maybe it would look like this:

    do open_and_process("testfile", "r") process(file):
        for line in file:
            print line

I don't really think this is necessary. I don't mind being verbose if it 
makes things clearer and simpler.

Here's some other ideas: use "def" instead of "with". They'd have to be 
indented to avoid ambiguity, though:

    do process_file(path):
        def process(file):
            for line in file:
                print line
        def success():
            print 'file processed successfully!'
        def error(exc):
            print 'an exception was raised during processing:', exc

The presence of the familiar def keyword should help people understand 
what's happening here.

Note that I didn't include an example but there's no reason why an 
anonymous block parameter couldn't return a value which could be used in 
the function calling the block.

Please, be gentle.

-- 
Jason