<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>On Jul 29, 2015, at 18:46, Sven R. Kunze <<a href="mailto:srkunze@mail.de">srkunze@mail.de</a>> wrote:</div><div><br></div><blockquote type="cite"><div>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
Hi everybody,<br>
<br>
well during the discussion of the concurrency capabilities of
Python, I found this article reading worthwhile:
<a class="moz-txt-link-freetext" href="http://chriskiehl.com/article/parallelism-in-one-line/">http://chriskiehl.com/article/parallelism-in-one-line/</a> His statement
"Mmm.. Smell those Java roots." basically sums the whole topic up
for me.<br>
<br>
<br>
That is sequential code (almost plain English):<br>
<br>
<font face="monospace">for image in images: <br>
create_thumbnail(image)</font><br>
<br>
<br>
In order to have a start with parallelism and concurrency, we need
to do the following:<br>
<br>
<font face="monospace">pool = Pool()<br>
pool.map(create_thumbnail, images)<br>
pool.close() <br>
pool.join()</font><br></div></blockquote><div><br></div><div>No you don't, because this isn't Java:</div><div><br></div><div> with Pool() as pool:</div><div> pool.map(create_thumbnail, images)</div><div><br></div><div>Also note that if create_thumbnail returns a value, to assemble all the values into a list is just as simple:</div><div><br></div><div><div><span style="background-color: rgba(255, 255, 255, 0);"> with Pool() as pool:</span></div><div><span style="background-color: rgba(255, 255, 255, 0);"> thumbnails = pool.map(create_thumbnail, images)</span></div></div><div><br></div><div>And if you want to iterate over the thumbnails as created and don't care about the order, you can just replace the map with imap_unordered. (Or, of course, you can use an executor and just iterate as_completed on the list of futures.)</div><br><blockquote type="cite"><div>
Not bad (considering the other approaches), but why couldn't it not
look just like the sequential one, maybe like this:<br>
<br>
<font face="monospace">for image in images: <br>
fork create_thumbnail(image)</font><br></div></blockquote><div><br></div><div>To me, this strongly implies that you're actually forking a new child process (or at least a new thread) for every thumbnail. Which is probably a really bad idea if you have, say, 1000 of them. It definitely doesn't say "find/create some implicit pool somewhere, wrap this in a task, and submit it to the pool".</div><div><br></div><div>And I'm not sure I'd want it to. What if I want to use a process pool instead of a thread pool, or to use a pool of 12 threads instead of NCPU because I know I'm mostly waiting on a particular HTTP server farm and 12x concurrency is ideal?</div><div><br></div><div>Also, how would you extend this to return results? A statement can't have a result. And, even if this were an expression, it would look pretty ugly to do this:</div><div><br></div><div> thumbnails = []</div><div><div><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"> for image in images: <br> thumbnails.append(fork create_thumbnail(image))</span></font></div></div><div><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"><br></span></font></div><div><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">Or:</span></font></div><div><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"><br></span></font></div><div><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"> thumbnails = [fork create_thumbnail(image) for image in images]</span></font></div><div><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"><br></span></font></div><div><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">And, even if you liked the look of that, what exactly could thumbnails be? Obviously not a list of thumbnails. At best, a list of futures that you'd then still have to loop over with as_completed or similar.</span></font></div><div><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"><br></span></font></div><div><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">Of course you could design a new language with implicit futures built into the core (or, even better, a two-level variable store with dataflow variables and implicit blocking) to solve this, but it would be very different from Python semantics.</span></font></div><div><br></div><blockquote type="cite">
What I like about the Pool concept is that it frees me of thinking
about the interprocess/-thread communication and processes/threads
management (not sure how this works with coroutines, but the experts
of you do know).<br>
<br>
What I would like to be freed of as well is: pool management. It
actually reminds me of languages without garbage-collection.<br></blockquote><br><div>That's a good parallel--but that's exactly what's so nice about "with Pool() as pool:". When you need a pool to be deterministically managed, this is the nicest syntax in any language to do it (except maybe C++ with its RAII, which lets you hide deterministic destruction inside wrapper objects). It's hard to see how it could be any more minimal. After all, if you don't wait on the pool to finish, and you don't collect a bunch of futures to wait on, how do you know when all the thumbnails are created?</div></body></html>