[Python-ideas] fork

[Sven R. Kunze]

On 04.08.2015 05:21, Andrew Barnert wrote:
> On Aug 3, 2015, at 10:11, Sven R. Kunze <srkunze at mail.de 
> <mailto:srkunze at mail.de>> wrote:
>>
>> On 02.08.2015 02:02, Andrew Barnert wrote:
>>> Your idea of having a single global "pool manager" object, where you 
>>> could submit tasks and, depending on how they're marked, they get 
>>> handled differently might have merit. But that's something you could 
>>> build pretty easily on top of concurrent.futures (at least for 
>>> threads vs. processes; you can add in coroutines later, because 
>>> they're not quite as easy to integrate), upload to PyPI,
>>
>> You mean something like this?
>>
>> https://pypi.python.org/pypi/xfork
>
> Did you just write this today? Then yes, that proves my point about 
> how easy it is to write it. Now you just have to get people using it, 
> get some experience with it, etc. and you can come back with a 
> proposal to put something like this in the stdlib, add syntactic 
> support, etc. that it will be hard for anyone to disagree with. (Or to 
> discover that it has flaws that need to be fixed, or fundamental flaws 
> that can't be fixed, before making the proposal.)

I presented it today. The team members already showed interest. They 
also noted they like its simplicity. The missing syntax support seemed 
like minor issue compared to what complexity is hidden.

Others admitted they knew about the existence of concurrent.futures and 
such but never used it due to
  - its complexity
  - AND *drum roll* the '.result()' of the future objects
As it seems, it doesn't feel natural.

> One quick comment: from my experience (mostly with other languages 
> that are very different from Python, so I can't promise how well it 
> applies here...), implicit futures without implicit laziness or even 
> an explicit delay mechanism are not as useful as they look at first 
> glance. Code that forks off 8 Fibonacci calls, but waits for each 
> one's result before forking off the next one, might as well have just 
> stayed sequential. And if you're going to use the result by forking 
> off another job, then it's actually more convenient to use explicit 
> futures like the ones in the stdlib.
>
> One slightly bigger idea: If you really want to pursue your 
> implicit-as-possible design further, you might want to consider making 
> the decorators replace the function with an object whose __call__ 
> method just implicitly submits it to the pool.

I added two new decorators for this. But they don't work with the @ 
syntax. It seems like a well-known issue of Python:

_pickle.PicklingError: Can't pickle <function fib_fork at 
0x7f8eaeb09730>: it's not the same object as __main__.fib_fork

Would be great if somebody could fix that.

> Then you can use normal function-calling syntax and pretend everything 
> is magic. You can even add operator dunder methods to your future 
> class that do the same thing (so "result * 2" just builds a new future 
> out of "self.get() * 2", either submitted to the pool, probably 
> better, tacked on as an add_done_callback). I think there's a limit to 
> how far you can push this without some mechanism to mark when you need 
> to actual value (in ML-derived languages and C++, static types make 
> this easier: a cast, implicit or explicit, forces a wait; in Python, 
> that doesn't work), but it might be worth exploring that limit. Or it 
> might be better to just stop at the magic function calls and leave the 
> futures alone.

I actually like the idea of contagious futures and I might outline why 
this is not an issue with the current Python language.

Have a look at the following small interactive Python session:

 >>> 3+4
7
 >>> _
7
 >>> a=9
 >>> _
7
 >>> a+=10
 >>> _
7
 >>> a
19
 >>> _
19
 >>>


Question:
When has the add operation being executed?

Answer:
Unknown from the programmer's perspective.

Only requirement:
Exceptions are raised exactly where the operation is supposed to take 
place in the source code (even if the operation that raises the 
exception is performed later).


Best,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150804/9d0f4637/attachment.html>