[IPython-dev] ipython cluster with mixture of windows and linux engines

John Gill jgill at tokiomillennium.com
Wed Apr 23 13:45:11 EDT 2014


First of all, thanks to everyone for the awesomeness that is ipython, cannot thank you enough.

Now, I am using the ipython cluster tools to help parallelise my data analysis.

I am using the load balanced view to submit jobs to the controller.   Jobs have dependencies and having the controller deal with all that for me is a joy.

Now one problem I have is that some of these jobs use code that only runs on windows.   Many of the jobs, though, could run on linux.   I can envisage in the future having some jobs that would only run on linux.

I am thinking it would be neat to have a mixed architecture cluster and have tasks specify constraints (such as I must run on linux or I must run on windows).    More generally, I can imagine a situation with a non-homogeneous cluster where some tasks might require larger memory for example and the same mechanism could be used to ensure tasks that could only run on some engines are run on the right engine.

Has anyone already solved this problem?

If not any pointers where I should look to implement a solution.

John

This communication and any attachments contain information which is confidential and may also be legally privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) please note that any form of disclosure, distribution, copying, printing or use of this communication or the information in it or in any attachments is strictly prohibited and may be unlawful. If you have received this communication in error, please return it with the title "received in error" to postmaster at tokiomillennium.com and then permanently delete the email and any attachments from your system.

E-mail communications cannot be guaranteed to be secure or error free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses. It is the recipient's responsibility to ensure that e-mail transmissions and any attachments are virus free. We do not accept liability for any damages or other consequences caused by information that is intercepted, corrupted, amended, lost, destroyed, arrives late or incomplete or contains viruses.
******************************************

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140423/8329229d/attachment.html>


More information about the IPython-dev mailing list