<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>On Aug 4, 2015, at 11:09, Sven R. Kunze <<a href="mailto:srkunze@mail.de">srkunze@mail.de</a>> wrote:</div><div><br></div><blockquote type="cite"><div>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
On 04.08.2015 05:21, Andrew Barnert wrote:<br>
<blockquote cite="mid:6A8EA952-ED98-4C26-9A40-54BE54367849@yahoo.com" type="cite">
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<div>On Aug 3, 2015, at 10:11, Sven R. Kunze <<a moz-do-not-send="true" href="mailto:srkunze@mail.de">srkunze@mail.de</a>>
wrote:</div>
<blockquote type="cite">
<div> <br>
<div class="moz-cite-prefix">On 02.08.2015 02:02, Andrew
Barnert wrote:<br>
</div>
<blockquote cite="mid:D897A956-964A-4036-9423-922A5635C72B@yahoo.com" type="cite">
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
<div>Your idea of having a single global "pool manager"
object, where you could submit tasks and, depending on how
they're marked, they get handled differently might have
merit. But that's something you could build pretty easily
on top of concurrent.futures (at least for threads vs.
processes; you can add in coroutines later, because
they're not quite as easy to integrate), upload to PyPI,</div>
</blockquote>
<br>
You mean something like this?<br>
<br>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://pypi.python.org/pypi/xfork">https://pypi.python.org/pypi/xfork</a><br>
</div>
</blockquote>
<div><br>
</div>
Did you just write this today? Then yes, that proves my point
about how easy it is to write it. Now you just have to get people
using it, get some experience with it, etc. and you can come back
with a proposal to put something like this in the stdlib, add
syntactic support, etc. that it will be hard for anyone to
disagree with. (Or to discover that it has flaws that need to be
fixed, or fundamental flaws that can't be fixed, before making the
proposal.)</blockquote>
<br>
I presented it today. The team members already showed interest. They
also noted they like its simplicity. The missing syntax support
seemed like minor issue compared to what complexity is hidden.<br>
<br>
Others admitted they knew about the existence of concurrent.futures
and such but never used it due to<br>
- its complexity<br>
- AND *drum roll* the '.result()' of the future objects<br>
As it seems, it doesn't feel natural.<br></div></blockquote><div><br></div><div>I don't know how to put this nicely, but I think anyone who finds the complexity of concurrent.futures too daunting to even attempt to learn it should not be working on any code that uses less explicit concurrency. I have taught concurrent.futures to rank novices in a brief personal session or a single StackOverflow answer and they responded, "Wow, I didn't realize it could be this simple". Someone who can't grasp it is almost certain to be someone who introduces races all over your code and can't even understand the problem, much less debug it.</div><div><br></div><blockquote type="cite"><div>
<blockquote cite="mid:6A8EA952-ED98-4C26-9A40-54BE54367849@yahoo.com" type="cite">
<div>One quick comment: from my experience (mostly with other
languages that are very different from Python, so I can't
promise how well it applies here...), implicit futures without
implicit laziness or even an explicit delay mechanism are not as
useful as they look at first glance. Code that forks off 8
Fibonacci calls, but waits for each one's result before forking
off the next one, might as well have just stayed sequential. And
if you're going to use the result by forking off another job,
then it's actually more convenient to use explicit futures like
the ones in the stdlib.</div>
<div><br>
</div>
<div>One slightly bigger idea: If you really want to pursue your
implicit-as-possible design further, you might want to consider
making the decorators replace the function with an object whose
__call__ method just implicitly submits it to the pool.</div>
</blockquote>
<br>
I added two new decorators for this. But they don't work with the @
syntax. It seems like a well-known issue of Python:<br>
<br>
_pickle.PicklingError: Can't pickle <function fib_fork at
0x7f8eaeb09730>: it's not the same object as __main__.fib_fork<br>
<br>
Would be great if somebody could fix that.<br>
<br>
<blockquote cite="mid:6A8EA952-ED98-4C26-9A40-54BE54367849@yahoo.com" type="cite">
<div>Then you can use normal function-calling syntax and pretend
everything is magic. You can even add operator dunder methods to
your future class that do the same thing (so "result * 2" just
builds a new future out of "self.get() * 2", either submitted to
the pool, probably better, tacked on as an add_done_callback). I
think there's a limit to how far you can push this without some
mechanism to mark when you need to actual value (in ML-derived
languages and C++, static types make this easier: a cast,
implicit or explicit, forces a wait; in Python, that doesn't
work), but it might be worth exploring that limit. Or it might
be better to just stop at the magic function calls and leave the
futures alone.</div>
</blockquote>
<br>
I actually like the idea of contagious futures and I might outline
why this is not an issue with the current Python language.<br>
<br>
Have a look at the following small interactive Python session:<br>
<br>
>>> 3+4<br>
7<br>
>>> _<br>
7<br>
>>> a=9<br>
>>> _<br>
7<br>
>>> a+=10<br>
>>> _<br>
7<br>
>>> a<br>
19<br>
>>> _<br>
19<br>
>>> <br>
<br>
<br>
Question:<br>
When has the add operation being executed?<br>
<br>
Answer:<br>
Unknown from the programmer's perspective.<br></div></blockquote><div><br></div><div>Not true. The language clearly defines when each step happens. The a.__add__ method is called, then the result is assigned to a, then the statement finishes. (Then, in the next statement, nothing happens--except, because this is happening in the interactive interpreter, and it's an expression statement, after the statement finishes doing nothing, the value of the expression is assigned to _ and its repr is printed out.)</div><div><br></div><div>This ordering relationship may be very important if the variable a is shared by multiple threads, especially if more than one thread may modify it, especially if you're using non-atomic operations like += (where another thread can read, use, and assign the variable between the __add__ call and the assignment). If a references a mutable object with an __iadd__ method, the variable doesn't even need to be shared, only the value, for this to matter. The only way to safely ignore these problems is to never share any variables or any mutable values between threads. (This is why concurrency features are easier to design in pure functional languages.) Hiding this fact when you or the people you're hiding it from don't even understand the issue is exactly how you create races.</div><br><blockquote type="cite"><div>
Only requirement:<br>
Exceptions are raised exactly where the operation is supposed to
take place in the source code (even if the operation that raises the
exception is performed later).<br>
</div></blockquote></body></html>