[pypy-dev] Syntax for the 'transaction' module

[holger krekel]

Hi Armin,

On Tue, May 01, 2012 at 15:55 +0200, Armin Rigo wrote:
> Re-hi,
> 
> Now I did a full circle, and I'm wondering again if we couldn't go
> with the single primitive of an "atomic" object.  You use it as "with
> atomic:", in a way quite traditional for Transactional Memory.

i like this, more minimal, more obvious.

> Indeed, it seems after all possible to have the following model: a
> pypy-stm interpreter with the unmodified thread and threading modules,
> with just the addition of "atomic" (from some module, maybe "atomic",
> or "transaction" again, or "transact", or a better name).

Maybe "atomic" could become a __pypy__ builtin and there could be a "ame" or 
so package which atomic-using apps could depend on? In any case,
I really like the twist of "To remedy the GIL use AME" :)

> You can use
> it to ensure that a piece of code is run atomically; this would mean,
> in the sense of CPython's GIL, that the GIL is not released for the
> complete duration of the "with" statement.  (On top of CPython, this
> cannot be fully achieved without hacking at the interpreter, but to a
> limited extent we can emulate it by saying at least the following: two
> threads that both want to run "with atomic" are serialized.  Easy to
> do with just one lock.)
> 
> On top of that, it is possible to write more nicely usable constructs
> in pure Python.  For example, to write the existing
> transaction.add()/run(), we would write code that starts N threads,
> each polling a Queue and running the items in a "with atomic" block.
> Or we can write "schedule.Runner()" or similar.
> 
> This approach has several advantages: the fact that the thread pooling
> logic is written in pure Python and tweakable; but also the fact that
> it is compatible with normal threads.  This means all the doubts I had
> about blocking C calls are already resolved --- if you need really to
> do blocking C calls, do them in a separate thread.  For example, in
> twisted, the loop calling select() would be written in a thread --- or
> likely in the main thread --- while a pool of extra threads runs the
> actual logic.  (Not a normal thread pool, but one which uses "with
> atomic" to run each item.)  People are already used to this approach
> --- with the exception that this gives the additional *huge* benefit
> of illusive serial execution, so locks&friends are useless.
> 
> The new things here, when compared to CPython or PyPy-without-STM, are:
> 
> (1) the ability to use multiple cores, by running transactions that go
> from one release-the-GIL to the next one (which is already existing as
> work-in-progress on CPython for Hardware TM);
> 
> (2) the "with atomic" construct that I already proposed to CPython
> last year (and certainly is not new, but inspired directly from
> classical TM).
> 
> Does it make any sense to you?

To me so far yes.

I am wondering how this all applies to the execnet-execution model, btw.
(http://codespeak.net/execnet for those who wonder what i mean)
remote_exec()s on the same gateway run currently in different threads 
and thus only send/receive needs to use "with atomic", right?

best,
holger