[IPython-dev] Using IPython Cluster with SGE -- help needed

Andreas Hilboll lists at hilboll.de
Sun Aug 4 10:29:48 EDT 2013

Am 04.08.2013 16:20, schrieb Matthieu Brucher:
> Hi,
> I guess we may want to start with the ipython documentation on this
> topic: http://ipython.org/ipython-doc/stable/parallel/parallel_process.html
> Cheers,
> 2013/8/4 Andreas Hilboll <lists at hilboll.de>:
>> Hi,
>> I would like to use IPython for calculations on our cluster. It's a
>> total of 11 compute + 1 management nodes (all running Linux), and we're
>> using SGE's qsub to submit jobs. The $HOME directory is shared via NFS
>> between all the nodes.
>> Even after reading the documentation, I'm unsure about how to get things
>> running. I assume that I'll have to execute ``ìpcluster -n 16`` on all
>> compute nodes (they have 16 cores each). I'd have the ipython shell
>> (notebook won't work due to firewall restrictions I cannot change) on
>> the management node. But how does the management node know about the
>> kernels which are running on the compute nodes and waiting for a job?
>> And how can I tell the management node that it shall use qsub to submit
>> the jobs to the individual kernels?
>> As I think this is a common use case, I'd be willing to write up a nice
>> tutorial about the setup, but I fear I need some help from you guys to
>> get things running ...
>> Cheers,
>> -- Andreas.
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev

Thanks, Matthieu,

not sure how I missed that :/ Is there a way to "re-use" the ipengines
on the nodes, like using qsub to "inject" a job into a (idle) running
ipengine? That way the sometimes longish startup times of the Python
interpreter on shared network filesystems could be avoided. Something like

1. start idling ipengines on all cores
2. use qsub to inject jobs into these
3. "cleanup" the ipengine after the job is done, making it ready for
another job.


-- Andreas.

More information about the IPython-dev mailing list