[Python-ideas] With clauses for generator expressions

[Andrew Barnert]

So far, nearly everyone is discussing things which are tangential, or arguing 
that one of the optional variants is bad. So let me strip down the proposal, 
without any options in it, and expand on a use case. The syntax is:


    (foo(line) with open('bar') as f for line in baz(f))

This translates to calling this function:

    def gen():
        with open('bar') as f:
            for line in baz(f):
                yield foo(line)

The translation for with clauses is identical to for and if clauses, and nesting 
works in the obvious way.

So, why do I want to create a generator that wraps a file or other generator 
inside a with clause?

There are a wide range of modules that have functions that can take a generator 
of strings in place of a file. Some examples off the top of my head include 
numpy.loadtxt, poster.multipart_encode, and line_protocol.connection.send. Many 
of these are asynchronous, so I can't just wrap the call in a with statement; I 
have to send a generator that will close the wrapped file (or other generator) 
when it's exhausted or closed, instead of when the function returns.

So, imagine a simple "get" command in a mail server, a method in the Connection 
class:

    def handle_get(self, message_id):
        path = os.path.join(mailbox_path, message_id)
        self.send_async(open(path, 'r'))

Now, let's say I want to do some kind of processing on the file as I send it 
(e.g., remove excessive curse words, or add new ones in if there aren't enough 
in any line):

    def handle_get(self, message_id):
        path = os.path.join(mailbox_path, message_id)
        def censored_file():
            with open(path, 'r') as file:
                for line in file:
                    yield self.censor(line)
        self.send_async(censored_file())

With my suggested idea, the last 5 lines could be replaced by this:

        self.send_async(self.censor(line) with open(path, 'r') as file for line 
in file)

Of course this async_chat-style model isn't the only way to write a server, but 
it is a common way to write a server, and I don't think it should be 
complicated.

----- Original Message ----
> From: Greg Ewing <greg.ewing at canterbury.ac.nz>
> To: python-ideas at python.org
> Sent: Thu, November 15, 2012 1:18:24 PM
> Subject: Re: [Python-ideas] With clauses for generator expressions
> 
> Andrew Barnert wrote:
> > 1. (foo(line) for line in baz(f) if 'bar' in line  with open('foo') as f)
> > 2. (foo(line) for line in baz(f) with open('foo')  as f if 'bar' in line)
> > 3. (foo(line) with open('foo') as f for line in  baz(f) if 'bar' in line)
> 
> Order 3 is the most consistent with existing  features. The
> only out-of-order thing about comprehensions currently is  that
> the result expression comes first instead of last. Everything
> else is  in the same order as the statement expansion.
> 
> --  Greg
> _______________________________________________
> Python-ideas mailing  list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>