<div dir="ltr">Hi Antoine - see inline below for my response...thanks for your time!<br><br><div class="gmail_quote"><div dir="ltr">On Fri, Sep 28, 2018 at 6:45 PM Antoine Pitrou <<a href="mailto:solipsis@pitrou.net">solipsis@pitrou.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hi,<br>
<br>
On Fri, 28 Sep 2018 17:07:33 -0400<br>
Sean Harrington <<a href="mailto:seanharr11@gmail.com" target="_blank">seanharr11@gmail.com</a>> wrote:<br>
> <br>
> In *short*, the implementation of the feature works as follows:<br>
> <br>
> 1. Exposes a kwarg on Pool.__init__ called `expect_initret`, that<br>
> defaults to False. When set to True:<br>
> 1. Capture the return value of the initializer kwarg of Pool<br>
> 2. Pass this value to the function being applied, as a kwarg.<br>
> <br>
> Again, in *short,* the motivation of the feature is to provide an explicit<br>
> "flow of data" from parent process to worker process, and to avoid being<br>
> *forced* to using the *global* keyword in initializer, or being *forced* to<br>
> create global variables in the parent process.<br>
<br>
Thanks for taking the time to explain your use case and write a<br>
proposal.<br>
<br>
My reactions to this are:<br>
<br>
1. The proposed API is ugly. This basically allows you to pass an<br>
argument which changes with which arguments another function is later<br>
called...</blockquote></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div class="gmail_quote"><div>> Yes I agree that this is a not-perfect contract, but isn't this also a concern with the current implementation? And isn't this pattern arguably more explicit than "The function-being-applied relying on the initializer to create a global variable from within it's lexical scope"?</div></div></blockquote><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> </blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2. A global variable seems like the adequate way to represent a<br>
process-global object (which is exactly your use case)</blockquote></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div class="gmail_quote"><div>> There is nothing wrong with using a global variable, especially in nearly every toy example found on the internet of using multiprocessing.Pool (i.e. optimizing a simple script). But what happens when you have lots of nested function calls in your applied function? My simple argument is that the developer should not be constrained to make the objects passed globally available in the process, as this MAY break encapsulation for large projects.</div></div></blockquote><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> </blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
3. If you don't like globals, you could probably do something like<br>
lazily-initialize the resource when a function needing it is executed;<br>
this also avoids creating the resource if the child doesn't use it at<br>
all. Would that work for you?<br></blockquote></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div class="gmail_quote"><div>> I have nothing against globals, my gripe is with being enforced to use them for every Pool use case. Further, if initializing the resource is expensive, we only want to do this ONE time per worker process. So no, this will not ~always~ work.</div></div></blockquote><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
As a more general remark, I understand the desire to make the Pool<br>
object more flexible, but we can also not pile up features until it<br>
satisfies all use cases.<br></blockquote></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div class="gmail_quote"><div>> I understand that this is a legitimate concern, but this is about API approachability. Python end-users of Pool are forced to declare a global from a lexical scope. Most Python end-users probably don't even know this is possible. Sure, this is adding a feature for a use case that I outlined, but really this is one of the two major use cases of "initializer" and "initargs" (<a href="https://thelaziestprogrammer.com/python/multiprocessing-pool-expect-initret-proposal">see my blog post for the 2 generalized use cases</a>), not some obscure use case. This is making that <b>very common</b> use case more approachable.</div></div></blockquote><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
As another general remark, concurrent.futures is IMHO the preferred API<br>
for the future, and where feature work should probably concentrate.<br></blockquote></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div class="gmail_quote"><div>> This is good to hear and know. And will keep this mind moving forward! </div></div></blockquote><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Regards<br>
<br>
Antoine.<br>
<br>
<br>
_______________________________________________<br>
Python-Dev mailing list<br>
<a href="mailto:Python-Dev@python.org" target="_blank">Python-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-dev" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="https://mail.python.org/mailman/options/python-dev/seanharr11%40gmail.com" rel="noreferrer" target="_blank">https://mail.python.org/mailman/options/python-dev/seanharr11%40gmail.com</a><br>
</blockquote></div></div>