<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto">@Guido As an aside, my understanding was libraries that fall back to c (Numpy as an example) release the GIL for load heavy operations. But I believe the explanation would hold in the general case if you replace thread with process using a ProcessPoolExecutor, that it would be good to be able to submit a callback function back to the executor.<br><br><div id="AppleMailSignature">Sent from my iPhone</div><div><br>On Jan 26, 2018, at 12:10 PM, Daniel Collins <<a href="mailto:dancollins34@gmail.com">dancollins34@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div><meta http-equiv="content-type" content="text/html; charset=utf-8">@Guido: I agree, that’s a much cleaner solution to pass the executor. However, I think the last line should be future.add_done_callback(callback)<div>return newf <br><div><br></div><div>not executor.submit. </div><div><br></div><div>I’ll rewrite it like this and resubmit tonight for discussion.<br><div><br><div id="AppleMailSignature">Sent from my iPhone</div><div><br>On Jan 26, 2018, at 11:59 AM, Guido van Rossum <<a href="mailto:guido@python.org">guido@python.org</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr"><div><div>@Bar: I don't know about exposing _chain_future(). Certainly it's
overkill for what the OP wants -- their PR only cares about chaining
concurrent.future.Future.<br><br></div>@Daniel: I present the following simpler
solution -- it requires you to explicitly pass the executor, but since
'fn' is being submitted to an executor, EIBTI.<br><br>def then(executor, future, fn):<br> newf = concurrent.futures.Future()<br> def callback(fut):<br> f = executor.submit(fn, fut)<br> try:<br> newf.set_result(f.result())<br> except CancelledError:<br> newf.cancel()<br> except Exception as err:<br> newf.set_exception(err)<br> return executor.submit(callback)<br><br></div>I can't quite follow your reasoning about worker threads (and did you realize that because of the GIL, Python doesn't actually use multiple cores?). But I suppose it doesn't matter whether I understand that -- your point is that you want the 'fn' function submitted to the executor, not run as a "done callback". And that's reasonable. But modifying so much code just so the Future can know which to executor it belongs so you can make then() a method seems overkill.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 26, 2018 at 8:54 AM, Daniel Collins <span dir="ltr"><<a href="mailto:dancollins34@gmail.com" target="_blank">dancollins34@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">So, just going point by point:<div><br></div><div>Yes, absolutely put this off for 3.8. I didn’t know the freeze was so close or I would have put the 3.8 tag on originally.</div><div><br></div><div>Yes, absolutely it is only meant for concurrent.futures futures, it only changes async where async uses concurrent.futures futures.</div><div><br></div><div>Here’s a more fleshed out description of the use case:</div><div><br></div><div>Assume you have two functions. Function a(x: str)->AResult fetches an AResult object from a web resource, function b(y: AResult) performs some computationally heavy work on AResult.</div><div><br></div><div>Assume you’re calling a 10 times with a threadpoolexecutor with 2 worker theads. If you were to schedule a as future using submit, and b as a callback, the executions would look like this:</div><div><br></div><div>ExecutorThread: b*10</div><div>Worker1: a*5</div><div>Worker2: a*5</div><div><br></div><div>This only gets worse as more work (b) is scheduled as a callback for the result from a.</div><div><br></div><div>Now you could resolve this by, instead of submitting b as a callback, submitting the following lambda:</div><div><br></div><div>lambda x: executor.submit(b, x) </div><div><br></div><div>But then you wouldn’t have easy access to this new future. You would have to build a lot of boilerplate code to collect that future into some external collection, and this would only get worse the deeper the nesting goes.</div><div><br></div><div>With this syntax on the other hand, if you run a 10 times using submit, but then run a_fut.then(b) for each future, execution instead looks like this:</div><div><br></div><div><div><span style="background-color:rgba(255,255,255,0)">ExecutorThread: </span></div><div><span style="background-color:rgba(255,255,255,0)">Worker1: a*5 b*5</span></div><div><span style="background-color:rgba(255,255,255,0)">Worker2: a*5 b*5</span></div><div><span style="background-color:rgba(255,255,255,0)"><br></span></div><div><span style="background-color:rgba(255,255,255,0)">You can also do additional depth easily. Suppose you want to run 3 c operations (processes the output of b) for each b operation. Then you could call this like</span></div><div><span style="background-color:rgba(255,255,255,0)"><br></span></div><div><span style="background-color:rgba(255,255,255,0)">b_fut = a_fut.then(b)</span></div><div><span style="background-color:rgba(255,255,255,0)"><br></span></div><div>for i in range(3):</div><div> b_fut.then(c)</div><div><br></div><div>And the execution would look like this:</div><div><br></div><div><div><span style="background-color:rgba(255,255,255,0)">ExecutorThread:</span></div><div><span style="background-color:rgba(255,255,255,0)">Worker1: a*5 b*5 c*15</span></div><div><span style="background-color:rgba(255,255,255,0)">Worker2: a*5 b*5 c*15</span></div></div><div><span style="background-color:rgba(255,255,255,0)"><br></span></div><div><span style="background-color:rgba(255,255,255,0)">Which would be very difficult to do otherwise, and distributes the load across the workers, while having direct access to the outputs of the calls to c.</span></div><div><span style="background-color:rgba(255,255,255,0)"><br></span></div><div><span style="background-color:rgba(255,255,255,0)">-dancollins34</span></div><br><div id="m_8599502052583474898AppleMailSignature">Sent from my iPhone</div><div><div class="h5"><div><br>On Jan 26, 2018, at 1:07 AM, Guido van Rossum <<a href="mailto:guido@python.org" target="_blank">guido@python.org</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr"><div>I really don't want to distract Yury with this. Let's consider this (or something that addresses the same need) for 3.8.<br><br>To be clear this is meant as a feature for concurrent.futures.Future, not for asyncio.Future. (It's a bit confusing since you also change asyncio.)<br><br>Also to be honest I don't understand the use case *or* the semantics very well. You have some explaining to do...<br><br></div>(Also, full links: <a href="https://bugs.python.org/issue32672" target="_blank">https://bugs.python.org/<wbr>issue32672</a>; <a href="https://github.com/python/cpython/pull/5335" target="_blank">https://github.com/python/<wbr>cpython/pull/5335</a>)<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 25, 2018 at 8:38 PM, Daniel Collins <span dir="ltr"><<a href="mailto:dancollins34@gmail.com" target="_blank">dancollins34@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space">Hello all,<div><br></div><div>So, first time posting here. I’ve been bothered for a while about the lack of the ability to chain futures in python, such that the next future will execute upon the first’s completion. So I submitted a pr to do this. This would add the .then(self, fn) method to concurrent.futures.Future. Thoughts?</div><div><br></div><div>-dancollins34</div><div><br></div><div>Github PR #5335</div><div><a href="http://bugs.python.org" target="_blank">bugs.python.org</a> issue #32672</div></div><br>______________________________<wbr>_________________<br>
Python-ideas mailing list<br>
<a href="mailto:Python-ideas@python.org" target="_blank">Python-ideas@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-ideas" rel="noreferrer" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/python-ideas</a><br>
Code of Conduct: <a href="http://python.org/psf/codeofconduct/" rel="noreferrer" target="_blank">http://python.org/psf/codeofco<wbr>nduct/</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div class="m_8599502052583474898gmail_signature" data-smartmail="gmail_signature">--Guido van Rossum (<a href="http://python.org/~guido" target="_blank">python.org/~guido</a>)</div>
</div>
</div></blockquote></div></div></div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">--Guido van Rossum (<a href="http://python.org/~guido" target="_blank">python.org/~guido</a>)</div>
</div>
</div></blockquote></div></div></div></div></blockquote></body></html>