[execnet-dev] Sub-processes PyPy

Ronny Pfannschmidt opensource at ronnypfannschmidt.de
Tue Nov 17 04:34:30 EST 2015 

I believe there is a bug in the sanity check when inner functions are passed inside a remote function
We should fix that

Am 17. November 2015 10:12:04 MEZ, schrieb holger krekel <holger at merlinux.eu>:
>On Mon, Nov 16, 2015 at 16:56 -0500, Luis José Novoa wrote:
>> Thanks again for the kind reply. After going through the
>multichannels
>> examples I have this toy example:
>> 
>> def summi(p):
>>     return p**2
>> 
>> 
>> def multiplier(channel, factor1, factor2, factor3):
>>     while not channel.isclosed():
>>         param = channel.receive()
>>         r = summi(p=param)
>>         channel.send(factor1 * factor2 + factor3 + r)
>> 
>> if __name__ == "__main__":
>> 
>>     factor11 = 20
>>     factor12 = 10
>>     factor13 = 200
>> 
>>     factor21 = 5
>>     factor22 = 5
>>     factor23 = 200
>> 
>> 
>>     gw = execnet.makegateway()
>>     channel1 = gw.remote_exec(multiplier, factor1=factor11,
>> factor2=factor12, factor3=factor13)
>>     channel1.send(1)
>>     gw2 = execnet.makegateway()
>>     channel2 = gw2.remote_exec(multiplier, factor1=factor21,
>> factor2=factor22, factor3=factor23)
>>     channel2.send(1)
>>     mch = execnet.MultiChannel([channel1, channel2])
>>     queue = mch.make_receive_queue()
>>     results = []
>>     for i in range(2):
>>         results.append(queue.get())
>>     print results
>> 
>> 
>> Now, when running it I get the following error:
>> 
>> ValueError: ("the use of non-builtin globals isn't supported",
>['summi']).
>> 
>> I tried the following:
>> 
>> def multiplier(channel, factor1, factor2, factor3):
>>      def summi(p):
>>          return p**2
>>     while not channel.isclosed():
>>         param = channel.receive()
>>         r = summi(p=param)
>>         channel.send(factor1 * factor2 + factor3 + r)
>> 
>> 
>> and with this change I got:
>> 
>> ValueError: ("the use of non-builtin globals isn't supported", ['p',
>'p'])
>
>You need to send a module rather than just a function if your
>function depends on another function in that module.
>execnet only sends the source of the function, otherwise.
>See here also:
>
>http://codespeak.net/execnet/example/test_info.html#remote-exec-a-module-avoiding-inlined-source-part-ii
>
>cheers,
>holger
>
>
>> 
>> 
>> I am just trying to understand the nature of the errors reported, and
>> possibly find a way around it. The function calls that I want to
>> parallelize invoke other functions from within, just as in the toy
>example.
>> 
>> 
>> 
>> Any clarification is much appreciated.
>> 
>> Thank you in advance.
>> 
>> Luis.
>> 
>> 
>> 
>> 
>> 
>> On Thu, Nov 12, 2015 at 6:50 AM, holger krekel <holger at merlinux.eu>
>wrote:
>> 
>> > On Tue, Nov 10, 2015 at 12:46 -0500, Luis José Novoa wrote:
>> > > Thanks for your reply. I was trying to replicate one of the
>execnet
>> > > examples available online and had problems using PyPy due to
>pydev (using
>> > > aptana under windows). But now it works. Thanks.
>> > >
>> > > Now, this is the kind of procedure that I am trying to replicate
>(which
>> > > currently uses the multiprocessing module and uses CPython to
>solve the
>> > > subproblems in parallel):
>> > >
>> > > out = mp.Queue()
>> > > subproblems = [mp.Process(target=MyFunction, args=(s, a, b, c, d,
>out))
>> > for
>> > > s in range(len(networks))]
>> > >
>> > > *# Run subproblems*
>> > > for sb in subproblems:
>> > >     sb.start()
>> > >
>> > > *# Get results from the out queue*
>> > > results = [out.get() for sb in subproblems]
>> >
>> > did you look into the multi-channel examples?
>> >
>> >     http://codespeak.net/execnet/example/test_multi.html
>> >
>> > you can start a pypy subprocess like so:
>> >
>> >     execnet.makegateway("popen//python=pypy")
>> >
>> > and that should work if used in the multichannel examples.
>> >
>> > holger
>> >
>> >
>> > >
>> > > So, what I would like to do, is to solve the subproblems with
>PyPy, as
>> > they
>> > > are time consuming. I would like to do this using execnet, but I
>still
>> > > haven't been able to get my head around it (Im clearly not even
>close to
>> > an
>> > > expert in these matters). I wonder if anyone can point me in the
>right
>> > > direction, maybe through an example, on how to do this.
>> > >
>> > > Thanks very much in advance. I apologize for any inconvenience
>this may
>> > > cause.
>> > >
>> > > On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski
><fijall at gmail.com>
>> > > wrote:
>> > >
>> > > > Hi Luis
>> > > >
>> > > > Things should work on pypy without any problem whatsoever, just
>like
>> > > > with normal python.
>> > > >
>> > > > Cheers,
>> > > > fijal
>> > > >
>> > > > On Thu, Nov 5, 2015 at 4:36 PM, Luis José Novoa <
>> > luisjosenovoa at gmail.com>
>> > > > wrote:
>> > > > > Hi Everyone,
>> > > > >
>> > > > > As part of a large scale optimization application, I am using
>the
>> > python
>> > > > > multiprocessing module to execute a procedure several times
>in
>> > parallel.
>> > > > > These executions in turn, return results used by other
>procedure,
>> > and the
>> > > > > whole thing repeats iteratively. I would like to call PyPy
>only for
>> > the
>> > > > > subprocesses (to improve performance), as in the main
>procedure I use
>> > > > > libraries like gurobipy, which are not compatible with PyPy.
>I
>> > looked at
>> > > > the
>> > > > > documentation of execnet online, but I could not find
>examples for
>> > > > > communication between CPython and PyPy.
>> > > > >
>> > > > > Any suggestions for using CPython + Multiprocessing with PyPy
>would
>> > be
>> > > > > highly appreciated.
>> > > > >
>> > > > > Have a great day.
>> > > > >
>> > > > > --
>> > > > > Luis J. Novoa
>> > > > >
>> > > > > _______________________________________________
>> > > > > execnet-dev mailing list
>> > > > > execnet-dev at python.org
>> > > > > https://mail.python.org/mailman/listinfo/execnet-dev
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Luis J. Novoa
>> >
>> > > _______________________________________________
>> > > execnet-dev mailing list
>> > > execnet-dev at python.org
>> > > https://mail.python.org/mailman/listinfo/execnet-dev
>> >
>> >
>> > --
>> > about me:    http://holgerkrekel.net/about-me/
>> > contracting: http://merlinux.eu
>> >
>> 
>> 
>> 
>> -- 
>> Luis J. Novoa
>
>-- 
>about me:    http://holgerkrekel.net/about-me/
>contracting: http://merlinux.eu
>_______________________________________________
>execnet-dev mailing list
>execnet-dev at python.org
>https://mail.python.org/mailman/listinfo/execnet-dev

-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20151117/44a95aec/attachment.html>

More information about the execnet-dev mailing list