[execnet-dev] Sub-processes PyPy
Ronny Pfannschmidt
opensource at ronnypfannschmidt.de
Tue Nov 17 07:47:30 EST 2015
I'm sorry for not clarifying, I was referring to his second example, which does use a inner function,
Am 17. November 2015 13:22:56 MEZ, schrieb holger krekel <holger at merlinux.eu>:
>On Tue, Nov 17, 2015 at 10:34 +0100, Ronny Pfannschmidt wrote:
>> I believe there is a bug in the sanity check when inner functions are
>passed inside a remote function
>> We should fix that
>
>but the below example is not an inner function.
>
>holger
>
>> Am 17. November 2015 10:12:04 MEZ, schrieb holger krekel
><holger at merlinux.eu>:
>> >On Mon, Nov 16, 2015 at 16:56 -0500, Luis José Novoa wrote:
>> >> Thanks again for the kind reply. After going through the
>> >multichannels
>> >> examples I have this toy example:
>> >>
>> >> def summi(p):
>> >> return p**2
>> >>
>> >>
>> >> def multiplier(channel, factor1, factor2, factor3):
>> >> while not channel.isclosed():
>> >> param = channel.receive()
>> >> r = summi(p=param)
>> >> channel.send(factor1 * factor2 + factor3 + r)
>> >>
>> >> if __name__ == "__main__":
>> >>
>> >> factor11 = 20
>> >> factor12 = 10
>> >> factor13 = 200
>> >>
>> >> factor21 = 5
>> >> factor22 = 5
>> >> factor23 = 200
>> >>
>> >>
>> >> gw = execnet.makegateway()
>> >> channel1 = gw.remote_exec(multiplier, factor1=factor11,
>> >> factor2=factor12, factor3=factor13)
>> >> channel1.send(1)
>> >> gw2 = execnet.makegateway()
>> >> channel2 = gw2.remote_exec(multiplier, factor1=factor21,
>> >> factor2=factor22, factor3=factor23)
>> >> channel2.send(1)
>> >> mch = execnet.MultiChannel([channel1, channel2])
>> >> queue = mch.make_receive_queue()
>> >> results = []
>> >> for i in range(2):
>> >> results.append(queue.get())
>> >> print results
>> >>
>> >>
>> >> Now, when running it I get the following error:
>> >>
>> >> ValueError: ("the use of non-builtin globals isn't supported",
>> >['summi']).
>> >>
>> >> I tried the following:
>> >>
>> >> def multiplier(channel, factor1, factor2, factor3):
>> >> def summi(p):
>> >> return p**2
>> >> while not channel.isclosed():
>> >> param = channel.receive()
>> >> r = summi(p=param)
>> >> channel.send(factor1 * factor2 + factor3 + r)
>> >>
>> >>
>> >> and with this change I got:
>> >>
>> >> ValueError: ("the use of non-builtin globals isn't supported",
>['p',
>> >'p'])
>> >
>> >You need to send a module rather than just a function if your
>> >function depends on another function in that module.
>> >execnet only sends the source of the function, otherwise.
>> >See here also:
>> >
>>
>>http://codespeak.net/execnet/example/test_info.html#remote-exec-a-module-avoiding-inlined-source-part-ii
>> >
>> >cheers,
>> >holger
>> >
>> >
>> >>
>> >>
>> >> I am just trying to understand the nature of the errors reported,
>and
>> >> possibly find a way around it. The function calls that I want to
>> >> parallelize invoke other functions from within, just as in the toy
>> >example.
>> >>
>> >>
>> >>
>> >> Any clarification is much appreciated.
>> >>
>> >> Thank you in advance.
>> >>
>> >> Luis.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Thu, Nov 12, 2015 at 6:50 AM, holger krekel
><holger at merlinux.eu>
>> >wrote:
>> >>
>> >> > On Tue, Nov 10, 2015 at 12:46 -0500, Luis José Novoa wrote:
>> >> > > Thanks for your reply. I was trying to replicate one of the
>> >execnet
>> >> > > examples available online and had problems using PyPy due to
>> >pydev (using
>> >> > > aptana under windows). But now it works. Thanks.
>> >> > >
>> >> > > Now, this is the kind of procedure that I am trying to
>replicate
>> >(which
>> >> > > currently uses the multiprocessing module and uses CPython to
>> >solve the
>> >> > > subproblems in parallel):
>> >> > >
>> >> > > out = mp.Queue()
>> >> > > subproblems = [mp.Process(target=MyFunction, args=(s, a, b, c,
>d,
>> >out))
>> >> > for
>> >> > > s in range(len(networks))]
>> >> > >
>> >> > > *# Run subproblems*
>> >> > > for sb in subproblems:
>> >> > > sb.start()
>> >> > >
>> >> > > *# Get results from the out queue*
>> >> > > results = [out.get() for sb in subproblems]
>> >> >
>> >> > did you look into the multi-channel examples?
>> >> >
>> >> > http://codespeak.net/execnet/example/test_multi.html
>> >> >
>> >> > you can start a pypy subprocess like so:
>> >> >
>> >> > execnet.makegateway("popen//python=pypy")
>> >> >
>> >> > and that should work if used in the multichannel examples.
>> >> >
>> >> > holger
>> >> >
>> >> >
>> >> > >
>> >> > > So, what I would like to do, is to solve the subproblems with
>> >PyPy, as
>> >> > they
>> >> > > are time consuming. I would like to do this using execnet, but
>I
>> >still
>> >> > > haven't been able to get my head around it (Im clearly not
>even
>> >close to
>> >> > an
>> >> > > expert in these matters). I wonder if anyone can point me in
>the
>> >right
>> >> > > direction, maybe through an example, on how to do this.
>> >> > >
>> >> > > Thanks very much in advance. I apologize for any inconvenience
>> >this may
>> >> > > cause.
>> >> > >
>> >> > > On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski
>> ><fijall at gmail.com>
>> >> > > wrote:
>> >> > >
>> >> > > > Hi Luis
>> >> > > >
>> >> > > > Things should work on pypy without any problem whatsoever,
>just
>> >like
>> >> > > > with normal python.
>> >> > > >
>> >> > > > Cheers,
>> >> > > > fijal
>> >> > > >
>> >> > > > On Thu, Nov 5, 2015 at 4:36 PM, Luis José Novoa <
>> >> > luisjosenovoa at gmail.com>
>> >> > > > wrote:
>> >> > > > > Hi Everyone,
>> >> > > > >
>> >> > > > > As part of a large scale optimization application, I am
>using
>> >the
>> >> > python
>> >> > > > > multiprocessing module to execute a procedure several
>times
>> >in
>> >> > parallel.
>> >> > > > > These executions in turn, return results used by other
>> >procedure,
>> >> > and the
>> >> > > > > whole thing repeats iteratively. I would like to call PyPy
>> >only for
>> >> > the
>> >> > > > > subprocesses (to improve performance), as in the main
>> >procedure I use
>> >> > > > > libraries like gurobipy, which are not compatible with
>PyPy.
>> >I
>> >> > looked at
>> >> > > > the
>> >> > > > > documentation of execnet online, but I could not find
>> >examples for
>> >> > > > > communication between CPython and PyPy.
>> >> > > > >
>> >> > > > > Any suggestions for using CPython + Multiprocessing with
>PyPy
>> >would
>> >> > be
>> >> > > > > highly appreciated.
>> >> > > > >
>> >> > > > > Have a great day.
>> >> > > > >
>> >> > > > > --
>> >> > > > > Luis J. Novoa
>> >> > > > >
>> >> > > > > _______________________________________________
>> >> > > > > execnet-dev mailing list
>> >> > > > > execnet-dev at python.org
>> >> > > > > https://mail.python.org/mailman/listinfo/execnet-dev
>> >> > > > >
>> >> > > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > > Luis J. Novoa
>> >> >
>> >> > > _______________________________________________
>> >> > > execnet-dev mailing list
>> >> > > execnet-dev at python.org
>> >> > > https://mail.python.org/mailman/listinfo/execnet-dev
>> >> >
>> >> >
>> >> > --
>> >> > about me: http://holgerkrekel.net/about-me/
>> >> > contracting: http://merlinux.eu
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Luis J. Novoa
>> >
>> >--
>> >about me: http://holgerkrekel.net/about-me/
>> >contracting: http://merlinux.eu
>> >_______________________________________________
>> >execnet-dev mailing list
>> >execnet-dev at python.org
>> >https://mail.python.org/mailman/listinfo/execnet-dev
>>
>> --
>> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail
>gesendet.
MFG Ronny
More information about the execnet-dev
mailing list