Hi all, Disclaimer: This isn't strictly about Scipy, but I think the people here are probably the most likely to have something interesting to say about this, as it's about scientific computing with Python. Myself and a student in our lab have written a package that we're using internally to distribute optimisation problems over several CPUs (or GPUs using PyCUDA) over the small cluster of computers in our lab using multiprocessing. It has two parts: some code for distributing work over several computers and over several CPUs/GPUs on those computers. There's no load balancing, we assume that each piece of work is roughly the same, however it does handle using shared memory arrays with numpy automatically (recently discussed on this list and the numpy one). It can use named pipes or IP for communications (we had to use named pipes because of the firewall setup in our lab). The other part of the code is a framework for optimisation routines that can be used with this. So far we've implemented a particle swarm optimisation algorithm which is ideal for this sort of distributed processing because it's very local and so splitting amongst several machines is quite nice. We're also thinking about implementing a genetic algorithm in the same framework. Anyway, getting to the point. We've written this largely for our own use, but it struck us that this might be of interest more widely. The code is fairly simple (a few hundred lines), but there are maybe some non-obvious things in there (at least for some people). We're thinking about releasing it as an independent package, but there's some work involved in that, so we were just looking to see if there was enough interest to make it worth the effort really. Also, it's very likely that our code is less than perfect, so if anyone would be interested in working on it that could be good too. Thanks for any comments! Dan
Dan, Thanks for the information. I am interested. Thanks, Deb
Dan Goodman <dg.gmane@thesamovar.net> 17/03/2010 12:31 pm >>> Hi all,
Disclaimer: This isn't strictly about Scipy, but I think the people here are probably the most likely to have something interesting to say about this, as it's about scientific computing with Python. Myself and a student in our lab have written a package that we're using internally to distribute optimisation problems over several CPUs (or GPUs using PyCUDA) over the small cluster of computers in our lab using multiprocessing. It has two parts: some code for distributing work over several computers and over several CPUs/GPUs on those computers. There's no load balancing, we assume that each piece of work is roughly the same, however it does handle using shared memory arrays with numpy automatically (recently discussed on this list and the numpy one). It can use named pipes or IP for communications (we had to use named pipes because of the firewall setup in our lab). The other part of the code is a framework for optimisation routines that can be used with this. So far we've implemented a particle swarm optimisation algorithm which is ideal for this sort of distributed processing because it's very local and so splitting amongst several machines is quite nice. We're also thinking about implementing a genetic algorithm in the same framework. Anyway, getting to the point. We've written this largely for our own use, but it struck us that this might be of interest more widely. The code is fairly simple (a few hundred lines), but there are maybe some non-obvious things in there (at least for some people). We're thinking about releasing it as an independent package, but there's some work involved in that, so we were just looking to see if there was enough interest to make it worth the effort really. Also, it's very likely that our code is less than perfect, so if anyone would be interested in working on it that could be good too. Thanks for any comments! Dan _______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user ************************************************************************************* This email message, including any attached files, is confidential and intended solely for the use of the individual or entity to whom it is addressed. The Department of Services, Technology and Administration prohibits the right to publish, copy, distribute or disclose any information contained in this email, or its attachments, by any party other than the intended recipient. If you have received this email in error please notify the sender and delete it from your system. No employee or agent is authorised to conclude any binding agreement on behalf of the Department of Services, Technology and Administration by email. The views or opinions presented in this email are solely those of the author and do not necessarily represent those of the Department, except where the sender expressly, and with authority, states them to be the views of the Department of Services, Technology and Administration. The Department of Services, Technology and Administration accepts no liability for any loss or damage arising from the use of this email and recommends that the recipient check this email and any attached files for the presence of viruses. *************************************************************************************
Hi Dan, I'm not sure if I can use your algorithm, but I'm definitely curious / interested. (We've got a similar (I guess?) setup where I work.) -Paul On 16. mars 2010, at 18.31, Dan Goodman wrote:
Hi all,
Disclaimer: This isn't strictly about Scipy, but I think the people here are probably the most likely to have something interesting to say about this, as it's about scientific computing with Python.
Myself and a student in our lab have written a package that we're using internally to distribute optimisation problems over several CPUs (or GPUs using PyCUDA) over the small cluster of computers in our lab using multiprocessing. It has two parts: some code for distributing work over several computers and over several CPUs/GPUs on those computers. There's no load balancing, we assume that each piece of work is roughly the same, however it does handle using shared memory arrays with numpy automatically (recently discussed on this list and the numpy one). It can use named pipes or IP for communications (we had to use named pipes because of the firewall setup in our lab).
The other part of the code is a framework for optimisation routines that can be used with this. So far we've implemented a particle swarm optimisation algorithm which is ideal for this sort of distributed processing because it's very local and so splitting amongst several machines is quite nice. We're also thinking about implementing a genetic algorithm in the same framework.
Anyway, getting to the point. We've written this largely for our own use, but it struck us that this might be of interest more widely. The code is fairly simple (a few hundred lines), but there are maybe some non-obvious things in there (at least for some people). We're thinking about releasing it as an independent package, but there's some work involved in that, so we were just looking to see if there was enough interest to make it worth the effort really. Also, it's very likely that our code is less than perfect, so if anyone would be interested in working on it that could be good too.
Thanks for any comments!
Dan
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
Hi all and thanks for comments. We've now released this package, available on Google code and soon on PyPI. For various reasons we ended up calling it playdoh (hope we don't get in trouble for that). http://code.google.com/p/playdoh/ Please give it a go and let us know what you think. If you're interested in contributing, that would be great too! ;-) Dan On 17/03/2010 02:31, Dan Goodman wrote:
Hi all,
Disclaimer: This isn't strictly about Scipy, but I think the people here are probably the most likely to have something interesting to say about this, as it's about scientific computing with Python.
Myself and a student in our lab have written a package that we're using internally to distribute optimisation problems over several CPUs (or GPUs using PyCUDA) over the small cluster of computers in our lab using multiprocessing. It has two parts: some code for distributing work over several computers and over several CPUs/GPUs on those computers. There's no load balancing, we assume that each piece of work is roughly the same, however it does handle using shared memory arrays with numpy automatically (recently discussed on this list and the numpy one). It can use named pipes or IP for communications (we had to use named pipes because of the firewall setup in our lab).
The other part of the code is a framework for optimisation routines that can be used with this. So far we've implemented a particle swarm optimisation algorithm which is ideal for this sort of distributed processing because it's very local and so splitting amongst several machines is quite nice. We're also thinking about implementing a genetic algorithm in the same framework.
Anyway, getting to the point. We've written this largely for our own use, but it struck us that this might be of interest more widely. The code is fairly simple (a few hundred lines), but there are maybe some non-obvious things in there (at least for some people). We're thinking about releasing it as an independent package, but there's some work involved in that, so we were just looking to see if there was enough interest to make it worth the effort really. Also, it's very likely that our code is less than perfect, so if anyone would be interested in working on it that could be good too.
Thanks for any comments!
Dan
participants (3)
-
Dan Goodman
-
Debabrata Midya
-
Paul Anton Letnes