[IPython-dev] Using IPython Cluster with SGE -- help needed

[Matthieu Brucher]

Once the job is submitted, the client kernels are running. You can't use
qsub anymore, it's ipython's job to feed work job in the kernels. I think
it already behaves that way.
Le 4 août 2013 16:29, "Andreas Hilboll" <lists at hilboll.de> a écrit :

> Am 04.08.2013 16:20, schrieb Matthieu Brucher:
> > Hi,
> >
> > I guess we may want to start with the ipython documentation on this
> > topic:
> http://ipython.org/ipython-doc/stable/parallel/parallel_process.html
> >
> > Cheers,
> >
> > 2013/8/4 Andreas Hilboll <lists at hilboll.de>:
> >> Hi,
> >>
> >> I would like to use IPython for calculations on our cluster. It's a
> >> total of 11 compute + 1 management nodes (all running Linux), and we're
> >> using SGE's qsub to submit jobs. The $HOME directory is shared via NFS
> >> between all the nodes.
> >>
> >> Even after reading the documentation, I'm unsure about how to get things
> >> running. I assume that I'll have to execute ``ìpcluster -n 16`` on all
> >> compute nodes (they have 16 cores each). I'd have the ipython shell
> >> (notebook won't work due to firewall restrictions I cannot change) on
> >> the management node. But how does the management node know about the
> >> kernels which are running on the compute nodes and waiting for a job?
> >> And how can I tell the management node that it shall use qsub to submit
> >> the jobs to the individual kernels?
> >>
> >> As I think this is a common use case, I'd be willing to write up a nice
> >> tutorial about the setup, but I fear I need some help from you guys to
> >> get things running ...
> >>
> >> Cheers,
> >>
> >> -- Andreas.
> >> _______________________________________________
> >> IPython-dev mailing list
> >> IPython-dev at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
> Thanks, Matthieu,
>
> not sure how I missed that :/ Is there a way to "re-use" the ipengines
> on the nodes, like using qsub to "inject" a job into a (idle) running
> ipengine? That way the sometimes longish startup times of the Python
> interpreter on shared network filesystems could be avoided. Something like
>
> 1. start idling ipengines on all cores
> 2. use qsub to inject jobs into these
> 3. "cleanup" the ipengine after the job is done, making it ready for
> another job.
>
> Cheers,
>
> -- Andreas.
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20130805/499a55cc/attachment.html>