Parallel OO programming in Python.
Hello. I am interested in parallel computing in Python. Except other modules I would like to use new modules for vector and matrix operations and scientific computing SciPy and NumPy. I have already installed LAPACK and BLAS libraries. It works well. For object oriented parallel programming in Python on a single machine I can use techniques such as Bulk Synchronous Parallelism (BSP) or Message Passing Interface (MPI). There are mentioned some modules and packages for Python on http://www.scipy.org/Topical_Software Article: Parallel and distributed programming. After reading prerequisites and limitations I thing that the following ones could be good for me: PyMPI Pypar MPI for Python But I can't distinguish which one brings fewer limitations, is efficient in application and will develop in future. My favorite is BSP, but I can't find a package for present SciPy and NumPy. Could anybody give me a recommendation, which way I should go. Thanks. Fie Pye --------------------------------------- Máme další spot P.Čtvrtníčka. Podívejte se www.neuservis.cz
For object oriented parallel programming in Python on a single machine I can use techniques such as Bulk Synchronous Parallelism (BSP) or Message Passing Interface (MPI). There are mentioned some modules and packages for Python on http://www.scipy.org/Topical_Software Article: Parallel and distributed programming. After reading prerequisites and limitations I thing that the following ones could be good for me: PyMPI Pypar MPI for Python
I think mpi4py is the best option. It is a very nice implementation and the developer is working really hard to make it a great package. Also, I don't think pympi and pypar are being developed actively anymore. Another thing you might check out is IPython's parallel computing facilities that let you use mpi4py (and other mpi bindings) interactively: http://ipython.scipy.org/moin/Parallel_Computing I have never used BPS, but am familiar with the model. Can probably get it working interactively with IPython as well. Cheers, Brian
But I can't distinguish which one brings fewer limitations, is efficient in application and will develop in future. My favorite is BSP, but I can't find a package for present SciPy and NumPy.
Could anybody give me a recommendation, which way I should go.
Thanks. Fie Pye
--------------------------------------- Máme další spot P.Čtvrtníčka. Podívejte se www.neuservis.cz
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
Thank you for this post and many thanks to Fernando Perez, for the awesome tool that Ipython1 is becomming!!! I am beginning to use it today!! Flávio On 10/29/06, Brian Granger <ellisonbg.net@gmail.com> wrote:
For object oriented parallel programming in Python on a single machine I can use techniques such as Bulk Synchronous Parallelism (BSP) or Message Passing Interface (MPI). There are mentioned some modules and packages for Python on http://www.scipy.org/Topical_Software Article: Parallel and distributed programming. After reading prerequisites and limitations I thing that the following ones could be good for me: PyMPI Pypar MPI for Python
I think mpi4py is the best option. It is a very nice implementation and the developer is working really hard to make it a great package. Also, I don't think pympi and pypar are being developed actively anymore.
Another thing you might check out is IPython's parallel computing facilities that let you use mpi4py (and other mpi bindings) interactively:
http://ipython.scipy.org/moin/Parallel_Computing
I have never used BPS, but am familiar with the model. Can probably get it working interactively with IPython as well.
Cheers,
Brian
But I can't distinguish which one brings fewer limitations, is efficient in application and will develop in future. My favorite is BSP, but I can't find a package for present SciPy and NumPy.
Could anybody give me a recommendation, which way I should go.
Thanks. Fie Pye
--------------------------------------- Máme další spot P.Čtvrtníčka. Podívejte se www.neuservis.cz
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
-- Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark
Hi, I also looked at the entry on Ipython1 and it looks really promising. Did I understand correctly that Ipython1 is able to run parallel python code, regardless of which "MPI for python" package one eventually uses? Giovanni Flavio Coelho wrote:
Thank you for this post and many thanks to Fernando Perez, for the awesome tool that Ipython1 is becomming!!! I am beginning to use it today!!
Flávio
On 10/29/06, * Brian Granger* <ellisonbg.net@gmail.com <mailto:ellisonbg.net@gmail.com>> wrote:
> For object oriented parallel programming in Python on a single machine I can use techniques such as Bulk Synchronous Parallelism (BSP) or Message Passing Interface (MPI). There are mentioned some modules and packages for Python on > http://www.scipy.org/Topical_Software > Article: Parallel and distributed programming. > After reading prerequisites and limitations I thing that the following ones could be good for me: > PyMPI > Pypar > MPI for Python
I think mpi4py is the best option. It is a very nice implementation and the developer is working really hard to make it a great package. Also, I don't think pympi and pypar are being developed actively anymore.
Another thing you might check out is IPython's parallel computing facilities that let you use mpi4py (and other mpi bindings) interactively:
http://ipython.scipy.org/moin/Parallel_Computing
I have never used BPS, but am familiar with the model. Can probably get it working interactively with IPython as well.
Cheers,
Brian
> But I can't distinguish which one brings fewer limitations, is efficient in application and will develop in future. > My favorite is BSP, but I can't find a package for present SciPy and NumPy. > > Could anybody give me a recommendation, which way I should go. > > Thanks. > Fie Pye > > > > > --------------------------------------- > Máme další spot P.Čtvrtníčka. Podívejte se www.neuservis.cz <http://www.neuservis.cz> > > > _______________________________________________ > SciPy-user mailing list > SciPy-user@scipy.org <mailto:SciPy-user@scipy.org> > http://projects.scipy.org/mailman/listinfo/scipy-user <http://projects.scipy.org/mailman/listinfo/scipy-user> > _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org <mailto:SciPy-user@scipy.org> http://projects.scipy.org/mailman/listinfo/scipy-user
-- Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark ------------------------------------------------------------------------
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
I also looked at the entry on Ipython1 and it looks really promising. Did I understand correctly that Ipython1 is able to run parallel python code, regardless of which "MPI for python" package one eventually uses?
In principle yes. The IPython engines have a configuration option that you use to import the mpi bindings of your choice. But, to work properly, the import statement must cause MPI_Init() to be called. If the bindings you are using don't do that, it should b easy to write a wrapper module that imports and also calls MPI_Init(). Also, some of this depends on what MPI implementation you are using. The more moderns ones (like openmpi) are really nice and pretty forgiving. The older ones, though, sometimes require that Python itself (not your script) call MPI_Init(). This is also very easy to work around. The bottom line is that we can help you get it to work with whatever mpi binding and implementation you want. Withh that said, I highly do highly recommend mpi4py and openmpi - we use these regularly with ipython and it all works well. Brian
Giovanni
Flavio Coelho wrote:
Thank you for this post and many thanks to Fernando Perez, for the awesome tool that Ipython1 is becomming!!! I am beginning to use it today!!
Flávio
On 10/29/06, * Brian Granger* <ellisonbg.net@gmail.com <mailto:ellisonbg.net@gmail.com>> wrote:
> For object oriented parallel programming in Python on a single machine I can use techniques such as Bulk Synchronous Parallelism (BSP) or Message Passing Interface (MPI). There are mentioned some modules and packages for Python on > http://www.scipy.org/Topical_Software > Article: Parallel and distributed programming. > After reading prerequisites and limitations I thing that the following ones could be good for me: > PyMPI > Pypar > MPI for Python
I think mpi4py is the best option. It is a very nice implementation and the developer is working really hard to make it a great package. Also, I don't think pympi and pypar are being developed actively anymore.
Another thing you might check out is IPython's parallel computing facilities that let you use mpi4py (and other mpi bindings) interactively:
http://ipython.scipy.org/moin/Parallel_Computing
I have never used BPS, but am familiar with the model. Can probably get it working interactively with IPython as well.
Cheers,
Brian
> But I can't distinguish which one brings fewer limitations, is efficient in application and will develop in future. > My favorite is BSP, but I can't find a package for present SciPy and NumPy. > > Could anybody give me a recommendation, which way I should go. > > Thanks. > Fie Pye > > > > > --------------------------------------- > Máme další spot P.Čtvrtníčka. Podívejte se www.neuservis.cz <http://www.neuservis.cz> > > > _______________________________________________ > SciPy-user mailing list > SciPy-user@scipy.org <mailto:SciPy-user@scipy.org> > http://projects.scipy.org/mailman/listinfo/scipy-user <http://projects.scipy.org/mailman/listinfo/scipy-user> > _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org <mailto:SciPy-user@scipy.org> http://projects.scipy.org/mailman/listinfo/scipy-user
-- Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark ------------------------------------------------------------------------
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
Hi Brian, Can Ipython1 take advantage of multiple core cpus or SMP machines in the same way it does for a networked cluster? It would be nice to have that since quad-core cpu are arriving on the market and SMP machines are pretty common already. Thanks! Flávio On 10/30/06, Brian Granger <ellisonbg.net@gmail.com> wrote:
I also looked at the entry on Ipython1 and it looks really promising. Did I understand correctly that Ipython1 is able to run parallel python code, regardless of which "MPI for python" package one eventually uses?
In principle yes. The IPython engines have a configuration option that you use to import the mpi bindings of your choice. But, to work properly, the import statement must cause MPI_Init() to be called. If the bindings you are using don't do that, it should b easy to write a wrapper module that imports and also calls MPI_Init(). Also, some of this depends on what MPI implementation you are using. The more moderns ones (like openmpi) are really nice and pretty forgiving. The older ones, though, sometimes require that Python itself (not your script) call MPI_Init(). This is also very easy to work around. The bottom line is that we can help you get it to work with whatever mpi binding and implementation you want.
Withh that said, I highly do highly recommend mpi4py and openmpi - we use these regularly with ipython and it all works well.
Brian
Giovanni
Thank you for this post and many thanks to Fernando Perez, for the awesome tool that Ipython1 is becomming!!! I am beginning to use it today!!
Flávio
On 10/29/06, * Brian Granger* <ellisonbg.net@gmail.com <mailto:ellisonbg.net@gmail.com>> wrote:
> For object oriented parallel programming in Python on a single machine I can use techniques such as Bulk Synchronous Parallelism (BSP) or Message Passing Interface (MPI). There are mentioned some modules and packages for Python on > http://www.scipy.org/Topical_Software > Article: Parallel and distributed programming. > After reading prerequisites and limitations I thing that the following ones could be good for me: > PyMPI > Pypar > MPI for Python
I think mpi4py is the best option. It is a very nice implementation and the developer is working really hard to make it a great
Flavio Coelho wrote: package.
Also, I don't think pympi and pypar are being developed actively anymore.
Another thing you might check out is IPython's parallel computing facilities that let you use mpi4py (and other mpi bindings) interactively:
http://ipython.scipy.org/moin/Parallel_Computing
I have never used BPS, but am familiar with the model. Can
probably
get it working interactively with IPython as well.
Cheers,
Brian
> But I can't distinguish which one brings fewer limitations, is efficient in application and will develop in future. > My favorite is BSP, but I can't find a package for present SciPy and NumPy. > > Could anybody give me a recommendation, which way I should go. > > Thanks. > Fie Pye > > > > > --------------------------------------- > Máme další spot P.Čtvrtníčka. Podívejte se www.neuservis.cz <http://www.neuservis.cz> > > > _______________________________________________ > SciPy-user mailing list > SciPy-user@scipy.org <mailto:SciPy-user@scipy.org> > http://projects.scipy.org/mailman/listinfo/scipy-user <http://projects.scipy.org/mailman/listinfo/scipy-user> > _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org <mailto:SciPy-user@scipy.org> http://projects.scipy.org/mailman/listinfo/scipy-user
-- Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
-- Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark
Can Ipython1 take advantage of multiple core cpus or SMP machines in the same way it does for a networked cluster? It would be nice to have that since quad-core cpu are arriving on the market and SMP machines are pretty common already.
This was one of the major usage cases we have had in mind. So yes. Some details though: In IPython1 computations are done by a separate process that we are calling the IPython engine. Multiple IPython engines can be run on multiple systems (distributed memory), a single (multi CPU and/or multi-core) system or a combination of the two. I have run two IPython engines on my dual core MacBook and seen 2x speedup with simple python scripts. I wasn't using MPI, but with MPi you should see a similar behavior as long as your algorithm scales linearly to 2 instances processes. I can't wait to use IPython1 with 4/8/16 cores on my workstation :) Please let us know if you have questions or problems. Brian On 10/30/06, Flavio Coelho <fccoelho@gmail.com> wrote:
Hi Brian,
Can Ipython1 take advantage of multiple core cpus or SMP machines in the same way it does for a networked cluster? It would be nice to have that since quad-core cpu are arriving on the market and SMP machines are pretty common already.
Thanks!
Flávio
I also looked at the entry on Ipython1 and it looks really promising. Did I understand correctly that Ipython1 is able to run parallel python code, regardless of which "MPI for python" package one eventually uses?
In principle yes. The IPython engines have a configuration option that you use to import the mpi bindings of your choice. But, to work properly, the import statement must cause MPI_Init() to be called. If the bindings you are using don't do that, it should b easy to write a wrapper module that imports and also calls MPI_Init(). Also, some of this depends on what MPI implementation you are using. The more moderns ones (like openmpi) are really nice and pretty forgiving. The older ones, though, sometimes require that Python itself (not your script) call MPI_Init(). This is also very easy to work around. The bottom line is that we can help you get it to work with whatever mpi binding and implementation you want.
Withh that said, I highly do highly recommend mpi4py and openmpi - we use these regularly with ipython and it all works well.
Brian
Giovanni
Flavio Coelho wrote:
Thank you for this post and many thanks to Fernando Perez, for the awesome tool that Ipython1 is becomming!!! I am beginning to use it today!!
Flávio
On 10/29/06, * Brian Granger* < ellisonbg.net@gmail.com <mailto:ellisonbg.net@gmail.com>> wrote:
> For object oriented parallel programming in Python on a single machine I can use techniques such as Bulk Synchronous Parallelism (BSP) or Message Passing Interface (MPI). There are mentioned some modules and packages for Python on > http://www.scipy.org/Topical_Software > Article: Parallel and distributed programming. > After reading prerequisites and limitations I thing that the following ones could be good for me: > PyMPI > Pypar > MPI for Python
I think mpi4py is the best option. It is a very nice implementation and the developer is working really hard to make it a great
Also, I don't think pympi and pypar are being developed actively anymore.
Another thing you might check out is IPython's parallel computing facilities that let you use mpi4py (and other mpi bindings) interactively:
http://ipython.scipy.org/moin/Parallel_Computing
I have never used BPS, but am familiar with the model. Can
On 10/30/06, Brian Granger <ellisonbg.net@gmail.com> wrote: package. probably
get it working interactively with IPython as well.
Cheers,
Brian
> But I can't distinguish which one brings fewer limitations, is efficient in application and will develop in future. > My favorite is BSP, but I can't find a package for present SciPy and NumPy. > > Could anybody give me a recommendation, which way I should go. > > Thanks. > Fie Pye > > > > > --------------------------------------- > Máme další spot P.Čtvrtníčka. Podívejte se www.neuservis.cz <http://www.neuservis.cz> > > > _______________________________________________ > SciPy-user mailing list > SciPy-user@scipy.org <mailto:SciPy-user@scipy.org> >
http://projects.scipy.org/mailman/listinfo/scipy-user
<http://projects.scipy.org/mailman/listinfo/scipy-user >
> _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org <mailto: SciPy-user@scipy.org>
http://projects.scipy.org/mailman/listinfo/scipy-user
-- Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
--
Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
Thanks Brian, I am playing with it right now! I am planning to run it soon on multiple machines with multiple CPUs. Will I have to install Ipython1 on every node or Will the controller host spawn the processes across the net? In the README you seem to be starting the engines locally on each host.... but before I get Ahead of myself , I am getting the following error when I try to start an engine on my localhost: fccoelho@sombra ~/Downloads/Twisted $ ipengine & [2] 32018 fccoelho@sombra ~/Downloads/Twisted $ 2006/10/30 16:44 -0200 [-] Log opened. 2006/10/30 16:44 -0200 [-] Starting factory < ipython1.kernel.enginevanilla.VanillaEngineClientFactoryFromEngineServiceobject at 0xb6706eac> 2006/10/30 16:44 -0200 [VanillaEngineServerProtocol,0,127.0.0.1] Unhandled Error Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/twisted/python/log.py", line 48, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/log.py", line 33, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/usr/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line 139, in _doReadOrWrite why = getattr(selectable, method)() File "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py", line 362, in doRead return self.protocol.dataReceived(data) File "/usr/lib/python2.4/site-packages/twisted/protocols/basic.py", line 99, in dataReceived self.doData() File "/usr/lib/python2.4/site-packages/twisted/protocols/basic.py", line 62, in doData self.stringReceived(self.__buffer) File "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line 557, in stringReceived self.nextHandler(msg) File "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line 573, in dispatch f(args) File "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line 706, in handle_REGISTER qe = engineservice.QueuedEngine(self, keepUpToDate=True) File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 342, in __init__ self.registerMethods() File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 345, in registerMethods zi.alsoProvides(self, *self.engine.__provides__) exceptions.AttributeError: 'module' object has no attribute 'alsoProvides' 2006/10/30 16:44 -0200 [VanillaEngineServerProtocol,0,127.0.0.1] unregistered engine None 2006/10/30 16:44 -0200 [VanillaEngineClientProtocol,client] Stopping factory <ipython1.kernel.enginevanilla.VanillaEngineClientFactoryFromEngineServiceobject at 0xb6706eac> 2006/10/30 16:44 -0200 [VanillaEngineServerProtocol,0,127.0.0.1] Unhandled Error Traceback (most recent call last): File "/usr/bin/ipcontroller", line 51, in main reactor.run() File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 218, in run self.mainLoop() File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 229, in mainLoop self.doIteration(t) File "/usr/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line 133, in doSelect _logrun(selectable, _drdw, selectable, method, dict) --- <exception caught here> --- File "/usr/lib/python2.4/site-packages/twisted/python/log.py", line 48, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/log.py", line 33, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/usr/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line 149, in _doReadOrWrite self._disconnectSelectable(selectable, why, method=="doRead") File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 256, in _disconnectSelectable selectable.connectionLost(failure.Failure(why)) File "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py", line 416, in connectionLost protocol.connectionLost(reason) File "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line 538, in connectionLost self.factory.unregisterEngine(self.id) File "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line 1023, in unregisterEngine return self.service.unregisterEngine(id) File "/usr/lib/python2.4/site-packages/ipython1/kernel/controllerservice.py", line 300, in unregisterEngine del self.engines[id] exceptions.KeyError: On 10/30/06, Brian Granger <ellisonbg.net@gmail.com> wrote:
Can Ipython1 take advantage of multiple core cpus or SMP machines in the same way it does for a networked cluster? It would be nice to have that since quad-core cpu are arriving on the market and SMP machines are pretty common already.
This was one of the major usage cases we have had in mind. So yes. Some details though:
In IPython1 computations are done by a separate process that we are calling the IPython engine. Multiple IPython engines can be run on multiple systems (distributed memory), a single (multi CPU and/or multi-core) system or a combination of the two. I have run two IPython engines on my dual core MacBook and seen 2x speedup with simple python scripts. I wasn't using MPI, but with MPi you should see a similar behavior as long as your algorithm scales linearly to 2 instances processes.
I can't wait to use IPython1 with 4/8/16 cores on my workstation :)
Please let us know if you have questions or problems.
Brian
Hi Brian,
Can Ipython1 take advantage of multiple core cpus or SMP machines in the same way it does for a networked cluster? It would be nice to have that since quad-core cpu are arriving on the market and SMP machines are
common already.
Thanks!
Flávio
On 10/30/06, Brian Granger <ellisonbg.net@gmail.com> wrote:
I also looked at the entry on Ipython1 and it looks really
Did I understand correctly that Ipython1 is able to run parallel
On 10/30/06, Flavio Coelho <fccoelho@gmail.com> wrote: pretty promising. python
code, regardless of which "MPI for python" package one eventually uses?
In principle yes. The IPython engines have a configuration option that you use to import the mpi bindings of your choice. But, to work properly, the import statement must cause MPI_Init() to be called. If the bindings you are using don't do that, it should b easy to write a wrapper module that imports and also calls MPI_Init(). Also, some of this depends on what MPI implementation you are using. The more moderns ones (like openmpi) are really nice and pretty forgiving. The older ones, though, sometimes require that Python itself (not your script) call MPI_Init(). This is also very easy to work around. The bottom line is that we can help you get it to work with whatever mpi binding and implementation you want.
Withh that said, I highly do highly recommend mpi4py and openmpi - we use these regularly with ipython and it all works well.
Brian
Giovanni
Thank you for this post and many thanks to Fernando Perez, for the awesome tool that Ipython1 is becomming!!! I am beginning to use it today!!
Flávio
On 10/29/06, * Brian Granger* < ellisonbg.net@gmail.com <mailto:ellisonbg.net@gmail.com>> wrote:
> For object oriented parallel programming in Python on a single machine I can use techniques such as Bulk Synchronous Parallelism (BSP) or Message Passing Interface (MPI). There are mentioned some modules and packages for Python on > http://www.scipy.org/Topical_Software > Article: Parallel and distributed programming. > After reading prerequisites and limitations I thing that the following ones could be good for me: > PyMPI > Pypar > MPI for Python
I think mpi4py is the best option. It is a very nice implementation and the developer is working really hard to make it a great
Flavio Coelho wrote: package.
Also, I don't think pympi and pypar are being developed
actively
anymore.
Another thing you might check out is IPython's parallel
computing
facilities that let you use mpi4py (and other mpi bindings) interactively:
http://ipython.scipy.org/moin/Parallel_Computing
I have never used BPS, but am familiar with the model. Can
probably
get it working interactively with IPython as well.
Cheers,
Brian
> But I can't distinguish which one brings fewer limitations,
is
efficient in application and will develop in future. > My favorite is BSP, but I can't find a package for present
SciPy
and NumPy. > > Could anybody give me a recommendation, which way I should
go.
> > Thanks. > Fie Pye > > > > > --------------------------------------- > Máme další spot P.Čtvrtníčka. Podívejte se www.neuservis.cz <http://www.neuservis.cz> > > > _______________________________________________ > SciPy-user mailing list > SciPy-user@scipy.org <mailto:SciPy-user@scipy.org> >
http://projects.scipy.org/mailman/listinfo/scipy-user
<http://projects.scipy.org/mailman/listinfo/scipy-user >
> _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org <mailto: SciPy-user@scipy.org>
http://projects.scipy.org/mailman/listinfo/scipy-user
-- Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
--
Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
-- Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark
It looks like you are using an older version of zope.interface. I would grab this version and try it. http://www.zope.org/Products/ZopeInterface/3.1.0c1/ZopeInterface-3.1.0c1.tgz also, I would use a recent (I alway use the svn trunk) version of twisted. Let me know if that helps.
I am playing with it right now! I am planning to run it soon on multiple machines with multiple CPUs. Will I have to install Ipython1 on every node or Will the controller host spawn the processes across the net? In the README you seem to be starting the engines locally on each host....
Yes, you will need to install everything on each host and start the engines on each machine. If you have mpi installed, you can start the engines using mpirun and this will handle it. Let us know how it goes. Brian
but before I get Ahead of myself , I am getting the following error when I try to start an engine on my localhost: fccoelho@sombra ~/Downloads/Twisted $ ipengine & [2] 32018 fccoelho@sombra ~/Downloads/Twisted $ 2006/10/30 16:44 -0200 [-] Log opened. 2006/10/30 16:44 -0200 [-] Starting factory <ipython1.kernel.enginevanilla.VanillaEngineClientFactoryFromEngineService object at 0xb6706eac> 2006/10/30 16:44 -0200 [VanillaEngineServerProtocol,0, 127.0.0.1] Unhandled Error Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/twisted/python/log.py", line 48, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/log.py", line 33, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/usr/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line 139, in _doReadOrWrite why = getattr(selectable, method)() File "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py", line 362, in doRead return self.protocol.dataReceived(data) File "/usr/lib/python2.4/site-packages/twisted/protocols/basic.py", line 99, in dataReceived self.doData() File "/usr/lib/python2.4/site-packages/twisted/protocols/basic.py", line 62, in doData self.stringReceived(self.__buffer) File "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line 557, in stringReceived self.nextHandler(msg) File "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line 573, in dispatch f(args) File "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line 706, in handle_REGISTER qe = engineservice.QueuedEngine(self, keepUpToDate=True) File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 342, in __init__ self.registerMethods () File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 345, in registerMethods zi.alsoProvides(self, *self.engine.__provides__) exceptions.AttributeError : 'module' object has no attribute 'alsoProvides'
2006/10/30 16:44 -0200 [VanillaEngineServerProtocol,0,127.0.0.1] unregistered engine None 2006/10/30 16:44 -0200 [VanillaEngineClientProtocol,client] Stopping factory < ipython1.kernel.enginevanilla.VanillaEngineClientFactoryFromEngineService object at 0xb6706eac> 2006/10/30 16:44 -0200 [VanillaEngineServerProtocol,0,127.0.0.1] Unhandled Error Traceback (most recent call last): File "/usr/bin/ipcontroller", line 51, in main reactor.run() File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 218, in run self.mainLoop() File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 229, in mainLoop self.doIteration(t) File "/usr/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line 133, in doSelect _logrun(selectable, _drdw, selectable, method, dict) --- <exception caught here> --- File "/usr/lib/python2.4/site-packages/twisted/python/log.py", line 48, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/log.py", line 33, in callWithContext return context.call ({ILogContext: newCtx}, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/usr/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line 149, in _doReadOrWrite self._disconnectSelectable(selectable, why, method=="doRead") File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 256, in _disconnectSelectable selectable.connectionLost(failure.Failure(why)) File "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py", line 416, in connectionLost protocol.connectionLost(reason) File "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line 538, in connectionLost self.factory.unregisterEngine(self.id) File "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line 1023, in unregisterEngine return self.service.unregisterEngine(id) File "/usr/lib/python2.4/site-packages/ipython1/kernel/controllerservice.py", line 300, in unregisterEngine del self.engines[id] exceptions.KeyError :
On 10/30/06, Brian Granger <ellisonbg.net@gmail.com> wrote:
Can Ipython1 take advantage of multiple core cpus or SMP machines in the same way it does for a networked cluster? It would be nice to have that since quad-core cpu are arriving on the market and SMP machines are
common already.
This was one of the major usage cases we have had in mind. So yes. Some details though:
In IPython1 computations are done by a separate process that we are calling the IPython engine. Multiple IPython engines can be run on multiple systems (distributed memory), a single (multi CPU and/or multi-core) system or a combination of the two. I have run two IPython engines on my dual core MacBook and seen 2x speedup with simple python scripts. I wasn't using MPI, but with MPi you should see a similar behavior as long as your algorithm scales linearly to 2 instances processes.
I can't wait to use IPython1 with 4/8/16 cores on my workstation :)
Please let us know if you have questions or problems.
Brian
On 10/30/06, Flavio Coelho <fccoelho@gmail.com> wrote:
Hi Brian,
Can Ipython1 take advantage of multiple core cpus or SMP machines in the same way it does for a networked cluster? It would be nice to have that since quad-core cpu are arriving on the market and SMP machines are
common already.
Thanks!
Flávio
On 10/30/06, Brian Granger <ellisonbg.net@gmail.com> wrote:
I also looked at the entry on Ipython1 and it looks really
Did I understand correctly that Ipython1 is able to run parallel
pretty pretty promising. python
code, regardless of which "MPI for python" package one eventually uses?
In principle yes. The IPython engines have a configuration option that you use to import the mpi bindings of your choice. But, to work properly, the import statement must cause MPI_Init() to be called. If the bindings you are using don't do that, it should b easy to write a wrapper module that imports and also calls MPI_Init(). Also, some of this depends on what MPI implementation you are using. The more moderns ones (like openmpi) are really nice and pretty forgiving. The older ones, though, sometimes require that Python itself (not your script) call MPI_Init(). This is also very easy to work around. The bottom line is that we can help you get it to work with whatever mpi binding and implementation you want.
Withh that said, I highly do highly recommend mpi4py and openmpi - we use these regularly with ipython and it all works well.
Brian
Giovanni
Thank you for this post and many thanks to Fernando Perez, for the awesome tool that Ipython1 is becomming!!! I am beginning to use it today!!
Flávio
On 10/29/06, * Brian Granger* < ellisonbg.net@gmail.com <mailto:ellisonbg.net@gmail.com>> wrote:
> For object oriented parallel programming in Python on a single machine I can use techniques such as Bulk Synchronous Parallelism (BSP) or Message Passing Interface (MPI). There are mentioned some modules and packages for Python on > http://www.scipy.org/Topical_Software > Article: Parallel and distributed programming. > After reading prerequisites and limitations I thing that the following ones could be good for me: > PyMPI > Pypar > MPI for Python
I think mpi4py is the best option. It is a very nice implementation and the developer is working really hard to make it a great
Flavio Coelho wrote: package.
Also, I don't think pympi and pypar are being developed
actively
anymore.
Another thing you might check out is IPython's parallel
computing
facilities that let you use mpi4py (and other mpi bindings) interactively:
http://ipython.scipy.org/moin/Parallel_Computing
I have never used BPS, but am familiar with the model. Can
probably
get it working interactively with IPython as well.
Cheers,
Brian
> But I can't distinguish which one brings fewer limitations,
is
efficient in application and will develop in future. > My favorite is BSP, but I can't find a package for present
SciPy
and NumPy. > > Could anybody give me a recommendation, which way I should
go.
> > Thanks. > Fie Pye > > > > > --------------------------------------- > Máme další spot P.Čtvrtníčka. Podívejte se www.neuservis.cz < http://www.neuservis.cz> > > >
> SciPy-user mailing list > SciPy-user@scipy.org <mailto:SciPy-user@scipy.org> >
>
SciPy-user mailing list SciPy-user@scipy.org <mailto: SciPy-user@scipy.org>
http://projects.scipy.org/mailman/listinfo/scipy-user
-- Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org
http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org
http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
--
Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
-- Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
Hi Brian, Sorry for taking so long to answer, I was very busy. I have tried again after updating ZopeInterface to the version you suggest, and Twisted and Nevow to SVN trunk. The error message I am getting is pretty much the same. I also tried running the tests. Here is the output of "trial ipython1": # trial ipython1 Running 29 tests. /usr/lib/python2.4/site-packages/twisted/trial/runner.py:243: exceptions.DeprecationWarning: log.startKeepingErrors is deprecated since Twisted 2.5 ipython1.test.test_controllerpb BasicControllerPBTest testDeferreds ... [ERROR] testExecute ... [ERROR] testInterfaces ... [ERROR] testPullNamespace ... [ERROR] testPushPull ... [ERROR] testPushPullSerialized ... [ERROR] testResult ... [ERROR] testScatterGather ... [ERROR] ipython1.test.test_controllerservice BasicControllerServiceTest testDeferreds ... [ERROR] testExecute ... [ERROR] testInterfaces ... [ERROR] testPullNamespace ... [ERROR] testPushPull ... [ERROR] testPushPullSerialized ... [ERROR] testResult ... [ERROR] testScatterGather ... [ERROR] [ERROR] ipython1.test.test_engineservice BasicEngineServiceTest testCompletedEmptyEngine ... [ERROR] testDeferreds ... [OK] testExecute ... [OK] testGetResult ... [OK] testInterfaces ... [ERROR] testPullNamespace ... [OK] testPushPull ... [OK] testPushPullSerialized ... [OK] testStatus ... [OK] [ERROR] ipython1.test.test_shell BasicShellTest testCommand ... [OK] testExecute ... [OK] testPutGet ... [OK] testUpdate ... [OK] =============================================================================== [ERROR]: ipython1.test.test_controllerpb.BasicControllerPBTest.testDeferreds Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_controllerpb.py", line 36, in setUp self.s = reactor.listenTCP(10111, self.sf) File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 386, in listenTCP p.startListening() File "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py", line 733, in startListening raise CannotListenError, (self.interface, self.port, le) twisted.internet.error.CannotListenError: Couldn't listen on any:10111: (98, 'Address already in use'). =============================================================================== [ERROR]: ipython1.test.test_controllerpb.BasicControllerPBTest.testExecute Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_controllerpb.py", line 36, in setUp self.s = reactor.listenTCP(10111, self.sf) File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 386, in listenTCP p.startListening() File "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py", line 733, in startListening raise CannotListenError, (self.interface, self.port, le) twisted.internet.error.CannotListenError: Couldn't listen on any:10111: (98, 'Address already in use'). =============================================================================== [ERROR]: ipython1.test.test_controllerpb.BasicControllerPBTest.testInterfaces Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_controllerpb.py", line 36, in setUp self.s = reactor.listenTCP(10111, self.sf) File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 386, in listenTCP p.startListening() File "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py", line 733, in startListening raise CannotListenError, (self.interface, self.port, le) twisted.internet.error.CannotListenError: Couldn't listen on any:10111: (98, 'Address already in use'). =============================================================================== [ERROR]: ipython1.test.test_controllerpb.BasicControllerPBTest.testPullNamespace Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_controllerpb.py", line 36, in setUp self.s = reactor.listenTCP(10111, self.sf) File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 386, in listenTCP p.startListening() File "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py", line 733, in startListening raise CannotListenError, (self.interface, self.port, le) twisted.internet.error.CannotListenError: Couldn't listen on any:10111: (98, 'Address already in use'). =============================================================================== [ERROR]: ipython1.test.test_controllerpb.BasicControllerPBTest.testPushPull Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_controllerpb.py", line 36, in setUp self.s = reactor.listenTCP(10111, self.sf) File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 386, in listenTCP p.startListening() File "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py", line 733, in startListening raise CannotListenError, (self.interface, self.port, le) twisted.internet.error.CannotListenError: Couldn't listen on any:10111: (98, 'Address already in use'). =============================================================================== [ERROR]: ipython1.test.test_controllerpb.BasicControllerPBTest.testPushPullSerialized Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_controllerpb.py", line 36, in setUp self.s = reactor.listenTCP(10111, self.sf) File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 386, in listenTCP p.startListening() File "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py", line 733, in startListening raise CannotListenError, (self.interface, self.port, le) twisted.internet.error.CannotListenError: Couldn't listen on any:10111: (98, 'Address already in use'). =============================================================================== [ERROR]: ipython1.test.test_controllerpb.BasicControllerPBTest.testResult Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_controllerpb.py", line 36, in setUp self.s = reactor.listenTCP(10111, self.sf) File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 386, in listenTCP p.startListening() File "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py", line 733, in startListening raise CannotListenError, (self.interface, self.port, le) twisted.internet.error.CannotListenError: Couldn't listen on any:10111: (98, 'Address already in use'). =============================================================================== [ERROR]: ipython1.test.test_controllerpb.BasicControllerPBTest.testScatterGather Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_controllerpb.py", line 36, in setUp self.s = reactor.listenTCP(10111, self.sf) File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 386, in listenTCP p.startListening() File "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py", line 733, in startListening raise CannotListenError, (self.interface, self.port, le) twisted.internet.error.CannotListenError: Couldn't listen on any:10111: (98, 'Address already in use'). =============================================================================== [ERROR]: ipython1.test.test_controllerservice.BasicControllerServiceTest.testDeferreds Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_controllerservice.py", line 31, in setUp self.addEngine(1) File "/usr/lib/python2.4/site-packages/ipython1/test/multienginetest.py", line 20, in addEngine e = es.completeEngine(es.EngineService(InteractiveShell)) File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 157, in completeEngine zi.alsoProvides(engine, IEngineComplete) exceptions.AttributeError: 'module' object has no attribute 'alsoProvides' =============================================================================== [ERROR]: ipython1.test.test_controllerservice.BasicControllerServiceTest.testExecute Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_controllerservice.py", line 31, in setUp self.addEngine(1) File "/usr/lib/python2.4/site-packages/ipython1/test/multienginetest.py", line 20, in addEngine e = es.completeEngine(es.EngineService(InteractiveShell)) File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 157, in completeEngine zi.alsoProvides(engine, IEngineComplete) exceptions.AttributeError: 'module' object has no attribute 'alsoProvides' =============================================================================== [ERROR]: ipython1.test.test_controllerservice.BasicControllerServiceTest.testInterfaces Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_controllerservice.py", line 31, in setUp self.addEngine(1) File "/usr/lib/python2.4/site-packages/ipython1/test/multienginetest.py", line 20, in addEngine e = es.completeEngine(es.EngineService(InteractiveShell)) File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 157, in completeEngine zi.alsoProvides(engine, IEngineComplete) exceptions.AttributeError: 'module' object has no attribute 'alsoProvides' =============================================================================== [ERROR]: ipython1.test.test_controllerservice.BasicControllerServiceTest.testPullNamespace Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_controllerservice.py", line 31, in setUp self.addEngine(1) File "/usr/lib/python2.4/site-packages/ipython1/test/multienginetest.py", line 20, in addEngine e = es.completeEngine(es.EngineService(InteractiveShell)) File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 157, in completeEngine zi.alsoProvides(engine, IEngineComplete) exceptions.AttributeError: 'module' object has no attribute 'alsoProvides' =============================================================================== [ERROR]: ipython1.test.test_controllerservice.BasicControllerServiceTest.testPushPull Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_controllerservice.py", line 31, in setUp self.addEngine(1) File "/usr/lib/python2.4/site-packages/ipython1/test/multienginetest.py", line 20, in addEngine e = es.completeEngine(es.EngineService(InteractiveShell)) File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 157, in completeEngine zi.alsoProvides(engine, IEngineComplete) exceptions.AttributeError: 'module' object has no attribute 'alsoProvides' =============================================================================== [ERROR]: ipython1.test.test_controllerservice.BasicControllerServiceTest.testPushPullSerialized Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_controllerservice.py", line 31, in setUp self.addEngine(1) File "/usr/lib/python2.4/site-packages/ipython1/test/multienginetest.py", line 20, in addEngine e = es.completeEngine(es.EngineService(InteractiveShell)) File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 157, in completeEngine zi.alsoProvides(engine, IEngineComplete) exceptions.AttributeError: 'module' object has no attribute 'alsoProvides' =============================================================================== [ERROR]: ipython1.test.test_controllerservice.BasicControllerServiceTest.testResult Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_controllerservice.py", line 31, in setUp self.addEngine(1) File "/usr/lib/python2.4/site-packages/ipython1/test/multienginetest.py", line 20, in addEngine e = es.completeEngine(es.EngineService(InteractiveShell)) File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 157, in completeEngine zi.alsoProvides(engine, IEngineComplete) exceptions.AttributeError: 'module' object has no attribute 'alsoProvides' =============================================================================== [ERROR]: ipython1.test.test_controllerservice.BasicControllerServiceTest.testScatterGather Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_controllerservice.py", line 31, in setUp self.addEngine(1) File "/usr/lib/python2.4/site-packages/ipython1/test/multienginetest.py", line 20, in addEngine e = es.completeEngine(es.EngineService(InteractiveShell)) File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 157, in completeEngine zi.alsoProvides(engine, IEngineComplete) exceptions.AttributeError: 'module' object has no attribute 'alsoProvides' =============================================================================== [ERROR]: ipython1.test.test_enginepb Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/twisted/trial/runner.py", line 486, in loadPackage module = modinfo.load() File "/usr/lib/python2.4/site-packages/twisted/python/modules.py", line 337, in load return self.pathEntry.pythonPath.moduleLoader(self.name) File "/usr/lib/python2.4/site-packages/twisted/python/modules.py", line 498, in moduleLoader return self._moduleLoader(modname) File "/usr/lib/python2.4/site-packages/twisted/python/reflect.py", line 357, in namedAny topLevelPackage = __import__(trialname) File "/usr/lib/python2.4/site-packages/ipython1/test/test_enginepb.py", line 33, in ? from ipython1.test import completeenginetest as cet File "/usr/lib/python2.4/site-packages/ipython1/test/completeenginetest.py", line 22, in ? class CompleteEngineTestCase(DeferredTestCase): File "/usr/lib/python2.4/site-packages/ipython1/test/completeenginetest.py", line 27, in CompleteEngineTestCase engine = es.completeEngine(Empty()) File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 157, in completeEngine zi.alsoProvides(engine, IEngineComplete) exceptions.AttributeError: 'module' object has no attribute 'alsoProvides' =============================================================================== [ERROR]: ipython1.test.test_engineservice.BasicEngineServiceTest.testCompletedEmptyEngine Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_engineservice.py", line 73, in testCompletedEmptyEngine c = es.completeEngine(Empty()) File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 157, in completeEngine zi.alsoProvides(engine, IEngineComplete) exceptions.AttributeError : 'module' object has no attribute 'alsoProvides' =============================================================================== [ERROR]: ipython1.test.test_engineservice.BasicEngineServiceTest.testInterfaces Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/ipython1/test/test_engineservice.py", line 43, in testInterfaces q = es.QueuedEngine(self.engine) File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 342, in __init__ self.registerMethods() File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 345, in registerMethods zi.alsoProvides(self, *self.engine.__provides__) exceptions.AttributeError : 'module' object has no attribute 'alsoProvides' =============================================================================== [ERROR]: ipython1.test.test_enginevanilla Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/twisted/trial/runner.py", line 486, in loadPackage module = modinfo.load() File "/usr/lib/python2.4/site-packages/twisted/python/modules.py", line 337, in load return self.pathEntry.pythonPath.moduleLoader(self.name) File "/usr/lib/python2.4/site-packages/twisted/python/modules.py", line 498, in moduleLoader return self._moduleLoader(modname) File "/usr/lib/python2.4/site-packages/twisted/python/reflect.py", line 357, in namedAny topLevelPackage = __import__(trialname) File "/usr/lib/python2.4/site-packages/ipython1/test/test_enginevanilla.py", line 36, in ? from ipython1.test import completeenginetest as cet File "/usr/lib/python2.4/site-packages/ipython1/test/completeenginetest.py", line 22, in ? class CompleteEngineTestCase(DeferredTestCase): File "/usr/lib/python2.4/site-packages/ipython1/test/completeenginetest.py", line 27, in CompleteEngineTestCase engine = es.completeEngine(Empty()) File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 157, in completeEngine zi.alsoProvides(engine, IEngineComplete) exceptions.AttributeError: 'module' object has no attribute 'alsoProvides' ------------------------------------------------------------------------------- Ran 29 tests in 1.931s FAILED (errors=20, successes=11) On 10/30/06, Brian Granger <ellisonbg.net@gmail.com> wrote:
It looks like you are using an older version of zope.interface. I would grab this version and try it.
http://www.zope.org/Products/ZopeInterface/3.1.0c1/ZopeInterface-3.1.0c1.tgz
also, I would use a recent (I alway use the svn trunk) version of twisted. Let me know if that helps.
I am playing with it right now! I am planning to run it soon on multiple machines with multiple CPUs. Will I have to install Ipython1 on every node or Will the controller host spawn the processes across the net? In the README you seem to be starting the engines locally on each host....
Yes, you will need to install everything on each host and start the engines on each machine. If you have mpi installed, you can start the engines using mpirun and this will handle it. Let us know how it goes.
Brian
Hi Flavio, On 11/6/06, Flavio Coelho <fccoelho@gmail.com> wrote:
Hi Brian,
Sorry for taking so long to answer, I was very busy. I have tried again after updating ZopeInterface to the version you suggest, and Twisted and Nevow to SVN trunk. The error message I am getting is pretty much the same. I also tried running the tests. Here is the output of "trial ipython1":
Many thanks for the report. We'll look into it (well, probably Brian will, mostly :) but I'd like to ask you that we take this particular discussion over to the ipython list, since it's getting to be a little off-topic for scipy-user. Even if there is significant overlap between the two groups of users, this is now technically an ipython discussion and is thust best held over at the ipython lists. In case you are not on the lists, the subscription info is here: http://ipython.scipy.org/moin/MailingLists Regards, f
Ok Fernando, I have subscribed to the Ipython-users list. Lets take it from there Flávio On 11/6/06, Fernando Perez <fperez.net@gmail.com> wrote:
Hi Flavio,
On 11/6/06, Flavio Coelho <fccoelho@gmail.com> wrote:
Hi Brian,
Sorry for taking so long to answer, I was very busy. I have tried again after updating ZopeInterface to the version you suggest, and Twisted and Nevow to SVN trunk. The error message I am getting is pretty much the same. I also tried running the tests. Here is the output of "trial ipython1":
Many thanks for the report. We'll look into it (well, probably Brian will, mostly :) but I'd like to ask you that we take this particular discussion over to the ipython list, since it's getting to be a little off-topic for scipy-user. Even if there is significant overlap between the two groups of users, this is now technically an ipython discussion and is thust best held over at the ipython lists.
In case you are not on the lists, the subscription info is here:
http://ipython.scipy.org/moin/MailingLists
Regards,
f _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
-- Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark
I have reponded on the ipython-users list. But just for the record, this still looks like you don't have a new enough version of ZopeInterface. Brian On 11/6/06, Flavio Coelho <fccoelho@gmail.com> wrote:
Ok Fernando,
I have subscribed to the Ipython-users list. Lets take it from there
Flávio
On 11/6/06, Fernando Perez < fperez.net@gmail.com> wrote:
Hi Flavio,
On 11/6/06, Flavio Coelho < fccoelho@gmail.com> wrote:
Hi Brian,
Sorry for taking so long to answer, I was very busy. I have tried again after updating ZopeInterface to the version you suggest, and Twisted and Nevow to SVN trunk. The error message I am getting is pretty much the same. I also tried running the tests. Here is the output of "trial ipython1":
Many thanks for the report. We'll look into it (well, probably Brian will, mostly :) but I'd like to ask you that we take this particular discussion over to the ipython list, since it's getting to be a little off-topic for scipy-user. Even if there is significant overlap between the two groups of users, this is now technically an ipython discussion and is thust best held over at the ipython lists.
In case you are not on the lists, the subscription info is here:
http://ipython.scipy.org/moin/MailingLists
Regards,
f _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
--
Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
On 10/30/06, Flavio Coelho <fccoelho@gmail.com> wrote:
Thank you for this post and many thanks to Fernando Perez, for the awesome tool that Ipython1 is becomming!!! I am beginning to use it today!!
As much as I like to see my name attached to such statements, it's important to acknowledge that virtually all the credit for this effort should go to Brian Granger and not to me :) Without his interest, boundless energy, and his involvement of Benjamin Ragan-Kelley (who also deserves much credit for doing lots of the nasty Twisted coding over this summer), we wouldn't be anywhere near this. But thanks anyway ;) Cheers, f
Ooops, Sorry for that mistake! So I redirect my thanks to Brian !! Thanks Brian! Flávio On 10/30/06, Fernando Perez <fperez.net@gmail.com> wrote:
On 10/30/06, Flavio Coelho <fccoelho@gmail.com> wrote:
Thank you for this post and many thanks to Fernando Perez, for the awesome tool that Ipython1 is becomming!!! I am beginning to use it today!!
As much as I like to see my name attached to such statements, it's important to acknowledge that virtually all the credit for this effort should go to Brian Granger and not to me :) Without his interest, boundless energy, and his involvement of Benjamin Ragan-Kelley (who also deserves much credit for doing lots of the nasty Twisted coding over this summer), we wouldn't be anywhere near this.
But thanks anyway ;)
Cheers,
f _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
-- Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark
participants (5)
-
Brian Granger -
Fernando Perez -
fiepye＠atlas.cz -
Flavio Coelho -
Giovanni Samaey