[Python-ideas] Suggestion for a new thread API for Python (OpenMP inspired)
Sturla Molden
sturla at molden.no
Thu Jan 22 11:53:13 CET 2009
I have lately been using OpenMP to write parallel C and Fortran code. I
must admit I am impressed. OpenMP is a much better abstraction for
writing concurrent code than using Win32/posix threads directly (or
similarly threading.Thread i Java or Python). What matters most is that
code can be written as sequential, tested, and then parallelised using
compiler pragmas. This is much easier than writing code intended to be
parallel from the start.
Not only is the abstraction more easy to apply, it also leads to fewer
problems with deadlocks, race conditions, livelocks, etc.
I was thinking something similar could be created for Python, e.g. on
top of the existing thread or threading modules, and possibly
multiprocessing. I believe a context manager could be used for this
purpose. What I have in mind is an API that would look approximately
like this (OpenMP pragmas for C on top, proposed Python equivalent below):
#pragma omp parallel
with pymp.Pool() as pool:
#pragma omp for
for item in pool.parallel(<iterable>):
#pragma omp for shedule(guided)
for item in pool.parallel(<iterable>, shed='guided'):
#pragma omp parallel for
with pymp.Pool() as pool:
for item in pool.parallel(<iterable>):
#pragma omp barrier
pool.barrier()
#pragma omp section
pool.section(fun, *args, **kwargs)
#pragma omp parallel sections
with pymp.Pool() as pool:
pool.section(fun1, *args, **kwargs)
pool.section(fun2, *args, **kwargs)
#pragma omp master
if pool.master:
#pragma omp critical
#pragma omp atomic
with pool.lock:
#pragma omp single
with pool.single():
#pragma omp ordered
with pool.ordered():
This is all trivial to program, except for the context manager on top.
It has somehow to get access to the code block below, spawn multiple
threads, and execute that block in each of the threads. I am not sure
how to grab the next executable block as a Python object (so I could
pass it to eval or exec), so a little help would be appreciated :)
Regards,
Sturla Molden
More information about the Python-ideas
mailing list