[Python-ideas] adding an __exec__ method to context managers?

[Ryan Freckleton]

On Mon, Oct 12, 2009 at 7:46 PM, Sturla Molden <sturla at molden.no> wrote:
>
> I have been trying to implement an OpenMP-like threading API for Python. For
> those who don't know OpenMP, it is an alternative threading API for C, C++
> and Fortran, that unlike pthreads and Win32 threads has an intuitive syntax.
<snip>
>
> The virtue is that one can take sequential code, add in a few pragmas here
> and there, and end up having a multi-threaded program a human can
> understand. All the mess that makes multi-threaded programs error-prone and
> difficult to write is taken care of by the compiler.
>
> Ok, so what has this to do with Python?
>
> First Python already has the main machinery for OpenMP-like syntax using
> closures and decorators. For example, if we have a seqential function:
<snip>
> Now we could e.g. imagine using this syntax instead, using __exec__ to
> control execution in threads:
>
> def foobar():
>
>   with pymp.parallel_for( range(100) ) as mt:
>       for i in mt.iter:
>
>           dostuff1(i)
>
>           with mt.critical:
>               dostuff2(i)
>
>
> def foobar():
>       with pymp.parallel_sections as mt:
>           with mt.section:
>           task1()
>
>       with mt.section:
>           task2()
>
>       with mt.section:
>           task3()
>
>
<snip>

Have you looked at the API for python-safethread
[http://code.google.com/p/python-safethread/wiki/Branching]? I think
an API combining that and your semantics would be very cool.

If you use the same technique of passing functions into the context
manager, your examples become:

def foobar():
  with pymp.parallel_for(range(100)) as mt:
      for i in mt.iter:
          mt.add(lambda : dostuff1(i))
          mt.critical(lambda : dostuff2(i))

def foobar():
      with pymp.parallel_sections as mt:
          mt.section(task1)
          mt.section(task2)
          mt.section(task3)

Another option would be to execute the closure under the covers, e.g.

# The function dostuff_in_parallel is called when necessary by the pymp object.
@pymp.parallel_for(range(100))
def dostuff_in_parallel(mt):
    dostuff1(i)
    with mt.critical:
        dostuff2(i)

# The function sections is called when necessary by the pymp object,
e.g. by the parallel_sections decorator.
@pymp.parallel_sections
def sections(mt):
    mt.section(task1)
    mt.section(task2)
    mt.section(task3)

There are a couple PEPs about adding blocks to python that were
rejected in favor of the more constrained with statement, so you may
want to look at those as well.

Cheers,
=====
--Ryan E. Freckleton