[IPython-dev] running tests using ipython parallel

[Ondrej Certik]

On Tue, Mar 17, 2009 at 8:07 AM, Brian Granger <ellisonbg.net at gmail.com> wrote:
>> I am playing with ipython parallel and it behaves much more robustly
>> than the multiprocessing module from python2.6. In the multiprocessing
>> module if I get an exception, the process basically gets stuck, ctrl-c
>> doesn't help and I need to kill it using the "kill" command.
>
> Yep, we have worked *very* hard to make sure that exceptions at least
> get propagated back to the client.  If they don't, it is a bug and
> please let us know.
>
>> In ipython it works pretty well. I have couple questions though:
>>
>> * which approach do you think would be the best to implement parallel
>> testing? Basically you have a test suite (nosetest and py.test
>> compatible) and currently in the sequential mode I just call
>> "execfile", get all functions from the file and execute them. In
>> parallel I do something along these lines:
>>
>>        from IPython.kernel import client
>>        mec = client.MultiEngineClient()
>>        mec.reset()
>>        print "running %d jobs" % self._jobs
>>        ids = mec.get_ids()
>>        mec.execute("filename = '%s'" % filename)
>>        mec.execute("gl = {'__file__': filename}")
>>        mec.execute("execfile(filename, gl)")
>>        i = 0
>>        for f in funcs:
>>            if i >= len(ids): i = 0
>>            #mec.push({"f": f})
>>            mec.push(dict(f=f))
>>            #print mec.execute("gl['%s']" % f.__name__, targets=[ids[i]])
>>            print mec.execute("f", targets=[ids[i]])
>>            i += 1
>>
>> funcs list contains all the test functions that I can then execute
>> using f(). Unfortunately this approach gives me an error at the line
>> "mec.push(dict(f=f))":
>
> Is f here an actual function?  If so, you need to use the method
> "push_function" instead of push.  The reason for this is that
> functions can't be pickled out of the box.  The push_function method
> has extra logic that makes it work.

Oops, you are right. Now I got a bit further. Simple functions now
work, but once some test function is decorated, it fails:


Traceback (most recent call last):
 File "bin/test", line 36, in <module>
   "jobs": options.jobs})
 File "/home/ondrej/repos/sympy/sympy/utilities/runtests.py", line 71, in test
   return t.test()
 File "/home/ondrej/repos/sympy/sympy/utilities/runtests.py", line 135, in test
   self.test_file(f)
 File "/home/ondrej/repos/sympy/sympy/utilities/runtests.py", line
214, in test_file
   mec.push_function(dict(f=f))
 File "/home/ondrej/lib/lib/python/IPython/kernel/multiengineclient.py",
line 591, in push_function
   return self._blockFromThread(self.smultiengine.push_function,
namespace, targets=targets, block=block)
 File "/home/ondrej/lib/lib/python/IPython/kernel/multiengineclient.py",
line 441, in _blockFromThread
   result = blockingCallFromThread(function, *args, **kwargs)
 File "/home/ondrej/lib/lib/python/IPython/kernel/twistedutil.py",
line 69, in blockingCallFromThread
   return twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
 File "/usr/lib/python2.6/dist-packages/twisted/internet/threads.py",
line 114, in blockingCallFromThread
   result.raiseException()
 File "/usr/lib/python2.6/dist-packages/twisted/python/failure.py",
line 326, in raiseException
   raise self.type, self.value, self.tb
ValueError: Sorry, cannot pickle code objects with closures


So it seems to me, that the only way to go forward is not to push any
code, because this will not work in general. So I think what I need is
to implement a simple function in sympy tests, that will parse the
test file and return me a list of tests (like a string or something)
and this will happen *on the engine*. And I will just tell it -- hey,
execute the test 37 and tell me the result.

>
> Other tips:
>
> * If the function you are calling don't take very long, the latency in
> your current approach will really get you.  A much better way would be
> to define a function that could test everything in a packages
> hierarchy below a certain point.  That way you could have an engine
> test an entire subpackage.  Then the latency will matter less.

At this moment I don't care about latency, but as it turns out, I
don't have any other option, see above.

>
> * The loop you are writing is basically just doing what the map method does:
>
> mec.map(lambda x: x**2, range(10))
>
> It works just like python's map, but is parallel.  The only difference
> is that map takes either a function or a string that can is exec'd.

Yes, in fact I wrote it using python map and then tried to use
multiprocessing map and it failed if the function uses a decorator. So
I think pushing any nontrivial code is a bad idea.

>
> * If you want to get rid of the code in strings "feature", just define
> functions, push the function using push_function and then call the
> function using execute.  We probably should also implement something
> like this:
>
> mec.call(f, args, **kwargs)
>
> If this would be useful to you, could you file a ticket for this?  In
> the ticket, could be mention that we should use a cache to make sure
> that functions are only pushed one time?

If I find it useful, I'll post the ticket. So far I am not sure due to
the problems above.

>
>
>
>
>>  File "/home/ondrej/repos/sympy/sympy/utilities/runtests.py", line
>> 214, in test_file
>>    mec.push(dict(f=f))
>>  File "/var/lib/python-support/python2.6/IPython/kernel/multiengineclient.py",
>> line 552, in push
>>    targets=targets, block=block)
>>  File "/var/lib/python-support/python2.6/IPython/kernel/multiengineclient.py",
>> line 441, in _blockFromThread
>>    result = blockingCallFromThread(function, *args, **kwargs)
>>  File "/var/lib/python-support/python2.6/IPython/kernel/twistedutil.py",
>> line 69, in blockingCallFromThread
>>    return twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
>>  File "/usr/lib/python2.6/dist-packages/twisted/internet/threads.py",
>> line 114, in blockingCallFromThread
>>    result.raiseException()
>>  File "/usr/lib/python2.6/dist-packages/twisted/python/failure.py",
>> line 326, in raiseException
>>    raise self.type, self.value, self.tb
>> TypeError: expected string or Unicode object, NoneType found
>>
>>
>> So I thought, ok, let's not push things in and execute everything at
>> the engines.
>>
>> * what things could be safely pushed to engines? I know it can push
>> some functions, but I didn't manage to get it actually working for the
>> actual test functions (it works nice for simple functions from the
>> tutorial). So the only option that seems to me that it should work is
>> that I first implement a function that executes for "n"th test case
>> from the test suite and this function will not be transfered to
>> engines. The only thing that will be pushed is one string ("function
>> name") and then couple integers to specify all the parameters.
>
> Other than functions, anything that can be pickled can be pushed.  The
> only big limitation is that classes need to be importable to be
> pushed.  Thus, classes that are defined interactively can't be pushed.
>
> Functions should be pushed using push_function.
>
> But, from the speed perspective, you want to push as little as
> possible, so just pushing strings is not a bad idea.  Why don't you
> get something working first though and then we can figure out the
> performance issues.

Exactly, that's my approach. I need to go the "do everything on
engines" route anyway though.

>
> One final note:  we are aware of some performance issues in the
> current parallel ipython.  These issues mainly affect latency (small
> amounts of work done in parallel) and pushing very large objects.  We
> are working on these things.

Sure, that's not my problem at the moment. :)

Ondrej