syntax sugar for stm TransactionQueue
hi Armin, following the discussion we had today, that TransactionQueue could be easier to understand for people if you explain it as "a for loop in which you don't know the order of the iteration", I figured out that we might even introduce some syntactic sugar for it; not sure if it makes things simpler or more complicated, though :). Anyway, I'm thinking of something like this: def parallel(iterable): def decorator(f): tr = TransactionQueue() for item in iterable: tr.add(f, item) tr.run() return decorator to be used in this way: mylist = [1, 2, 3] @parallel(mylist) def for_(item): # do something with item pass ciao, Anto
Supplying argument list at decoration time is hardly ideal. Typical call site would pass argument sequence at invocation time. Thus you probably want to rewrite this decorator and allow application without argument list: @parallel def foo(item) return item ** 123 results = foo(range(9)) Perhaps OT: My immediate reaction when I saw TransactionQueue in the docs was: 1. does add() have set semantics, that is, can I add() same item several times? 2. is queue synchronous or asynchronous, that is, can I do the following: tq = TransactionQueue() tq.start() for item in some_expensive_stream(): tq.add(func, item) tq.wait() Or is item execution already implied during .add() before .run()? Perhaps it's just something that needs to be clarified in the docs; or it may influence your sugar. On 24 July 2015 at 00:40, Antonio Cuni <anto.cuni@gmail.com> wrote:
Hi,
On 24 July 2015 at 09:26, Dima Tisnek <dimaqq@gmail.com> wrote:
I'm open to suggestions. However, TransactionQueue is also about being Queue-like. More items can be added dynamically by running transactions, like this: def foo(n): ... tr.add(foo, 2 * n) tr.add(foo, 2 * n + 1) tr = TransactionQueue() tr.add(foo, 0) tr.run() In both cases, the add()ed functions are not called immediately, but only after the current "block of code" finishes. In one case it means after foo() returned. In the other case it is when tr.run() is invoked. Dima's suggestion in particular:
can be written without a magic decorator but with a simple map-like function: def foo(item): return item ** 123 results = transaction.map(foo, range(9)) This can be added easily (and probably should). However, if we go down this path, I'd like to stop before we copy half of the API of the multiprocessing module. I'd prefer to apply my own feelings about the recent CPython APIs (like async stuff). My feeling is that instead of adding tons and tons of new APIs, we should rather add only the core language feature in a given (CPython/PyPy) release, and let 3rd-party libraries be developed on top of it. Optionally, a few releases later, if there is one clear winner among these libraries, we can bring it into the core. A bientôt, Armin.
Supplying argument list at decoration time is hardly ideal. Typical call site would pass argument sequence at invocation time. Thus you probably want to rewrite this decorator and allow application without argument list: @parallel def foo(item) return item ** 123 results = foo(range(9)) Perhaps OT: My immediate reaction when I saw TransactionQueue in the docs was: 1. does add() have set semantics, that is, can I add() same item several times? 2. is queue synchronous or asynchronous, that is, can I do the following: tq = TransactionQueue() tq.start() for item in some_expensive_stream(): tq.add(func, item) tq.wait() Or is item execution already implied during .add() before .run()? Perhaps it's just something that needs to be clarified in the docs; or it may influence your sugar. On 24 July 2015 at 00:40, Antonio Cuni <anto.cuni@gmail.com> wrote:
Hi,
On 24 July 2015 at 09:26, Dima Tisnek <dimaqq@gmail.com> wrote:
I'm open to suggestions. However, TransactionQueue is also about being Queue-like. More items can be added dynamically by running transactions, like this: def foo(n): ... tr.add(foo, 2 * n) tr.add(foo, 2 * n + 1) tr = TransactionQueue() tr.add(foo, 0) tr.run() In both cases, the add()ed functions are not called immediately, but only after the current "block of code" finishes. In one case it means after foo() returned. In the other case it is when tr.run() is invoked. Dima's suggestion in particular:
can be written without a magic decorator but with a simple map-like function: def foo(item): return item ** 123 results = transaction.map(foo, range(9)) This can be added easily (and probably should). However, if we go down this path, I'd like to stop before we copy half of the API of the multiprocessing module. I'd prefer to apply my own feelings about the recent CPython APIs (like async stuff). My feeling is that instead of adding tons and tons of new APIs, we should rather add only the core language feature in a given (CPython/PyPy) release, and let 3rd-party libraries be developed on top of it. Optionally, a few releases later, if there is one clear winner among these libraries, we can bring it into the core. A bientôt, Armin.
participants (3)
-
Antonio Cuni -
Armin Rigo -
Dima Tisnek