Thanks Brian, I am playing with it right now! I am planning to run it soon on multiple machines with multiple CPUs. Will I have to install Ipython1 on every node or Will the controller host spawn the processes across the net? In the README you seem to be starting the engines locally on each host.... but before I get Ahead of myself , I am getting the following error when I try to start an engine on my localhost: fccoelho@sombra ~/Downloads/Twisted $ ipengine & [2] 32018 fccoelho@sombra ~/Downloads/Twisted $ 2006/10/30 16:44 -0200 [-] Log opened. 2006/10/30 16:44 -0200 [-] Starting factory < ipython1.kernel.enginevanilla.VanillaEngineClientFactoryFromEngineServiceobject at 0xb6706eac> 2006/10/30 16:44 -0200 [VanillaEngineServerProtocol,0,127.0.0.1] Unhandled Error Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/twisted/python/log.py", line 48, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/log.py", line 33, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/usr/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line 139, in _doReadOrWrite why = getattr(selectable, method)() File "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py", line 362, in doRead return self.protocol.dataReceived(data) File "/usr/lib/python2.4/site-packages/twisted/protocols/basic.py", line 99, in dataReceived self.doData() File "/usr/lib/python2.4/site-packages/twisted/protocols/basic.py", line 62, in doData self.stringReceived(self.__buffer) File "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line 557, in stringReceived self.nextHandler(msg) File "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line 573, in dispatch f(args) File "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line 706, in handle_REGISTER qe = engineservice.QueuedEngine(self, keepUpToDate=True) File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 342, in __init__ self.registerMethods() File "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line 345, in registerMethods zi.alsoProvides(self, *self.engine.__provides__) exceptions.AttributeError: 'module' object has no attribute 'alsoProvides' 2006/10/30 16:44 -0200 [VanillaEngineServerProtocol,0,127.0.0.1] unregistered engine None 2006/10/30 16:44 -0200 [VanillaEngineClientProtocol,client] Stopping factory <ipython1.kernel.enginevanilla.VanillaEngineClientFactoryFromEngineServiceobject at 0xb6706eac> 2006/10/30 16:44 -0200 [VanillaEngineServerProtocol,0,127.0.0.1] Unhandled Error Traceback (most recent call last): File "/usr/bin/ipcontroller", line 51, in main reactor.run() File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 218, in run self.mainLoop() File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 229, in mainLoop self.doIteration(t) File "/usr/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line 133, in doSelect _logrun(selectable, _drdw, selectable, method, dict) --- <exception caught here> --- File "/usr/lib/python2.4/site-packages/twisted/python/log.py", line 48, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/log.py", line 33, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/usr/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line 149, in _doReadOrWrite self._disconnectSelectable(selectable, why, method=="doRead") File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 256, in _disconnectSelectable selectable.connectionLost(failure.Failure(why)) File "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py", line 416, in connectionLost protocol.connectionLost(reason) File "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line 538, in connectionLost self.factory.unregisterEngine(self.id) File "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line 1023, in unregisterEngine return self.service.unregisterEngine(id) File "/usr/lib/python2.4/site-packages/ipython1/kernel/controllerservice.py", line 300, in unregisterEngine del self.engines[id] exceptions.KeyError: On 10/30/06, Brian Granger <ellisonbg.net@gmail.com> wrote:
Can Ipython1 take advantage of multiple core cpus or SMP machines in the same way it does for a networked cluster? It would be nice to have that since quad-core cpu are arriving on the market and SMP machines are pretty common already.
This was one of the major usage cases we have had in mind. So yes. Some details though:
In IPython1 computations are done by a separate process that we are calling the IPython engine. Multiple IPython engines can be run on multiple systems (distributed memory), a single (multi CPU and/or multi-core) system or a combination of the two. I have run two IPython engines on my dual core MacBook and seen 2x speedup with simple python scripts. I wasn't using MPI, but with MPi you should see a similar behavior as long as your algorithm scales linearly to 2 instances processes.
I can't wait to use IPython1 with 4/8/16 cores on my workstation :)
Please let us know if you have questions or problems.
Brian
Hi Brian,
Can Ipython1 take advantage of multiple core cpus or SMP machines in the same way it does for a networked cluster? It would be nice to have that since quad-core cpu are arriving on the market and SMP machines are
common already.
Thanks!
Flávio
On 10/30/06, Brian Granger <ellisonbg.net@gmail.com> wrote:
I also looked at the entry on Ipython1 and it looks really
Did I understand correctly that Ipython1 is able to run parallel
On 10/30/06, Flavio Coelho <fccoelho@gmail.com> wrote: pretty promising. python
code, regardless of which "MPI for python" package one eventually uses?
In principle yes. The IPython engines have a configuration option that you use to import the mpi bindings of your choice. But, to work properly, the import statement must cause MPI_Init() to be called. If the bindings you are using don't do that, it should b easy to write a wrapper module that imports and also calls MPI_Init(). Also, some of this depends on what MPI implementation you are using. The more moderns ones (like openmpi) are really nice and pretty forgiving. The older ones, though, sometimes require that Python itself (not your script) call MPI_Init(). This is also very easy to work around. The bottom line is that we can help you get it to work with whatever mpi binding and implementation you want.
Withh that said, I highly do highly recommend mpi4py and openmpi - we use these regularly with ipython and it all works well.
Brian
Giovanni
Thank you for this post and many thanks to Fernando Perez, for the awesome tool that Ipython1 is becomming!!! I am beginning to use it today!!
Flávio
On 10/29/06, * Brian Granger* < ellisonbg.net@gmail.com <mailto:ellisonbg.net@gmail.com>> wrote:
> For object oriented parallel programming in Python on a single machine I can use techniques such as Bulk Synchronous Parallelism (BSP) or Message Passing Interface (MPI). There are mentioned some modules and packages for Python on > http://www.scipy.org/Topical_Software > Article: Parallel and distributed programming. > After reading prerequisites and limitations I thing that the following ones could be good for me: > PyMPI > Pypar > MPI for Python
I think mpi4py is the best option. It is a very nice implementation and the developer is working really hard to make it a great
Flavio Coelho wrote: package.
Also, I don't think pympi and pypar are being developed
actively
anymore.
Another thing you might check out is IPython's parallel
computing
facilities that let you use mpi4py (and other mpi bindings) interactively:
http://ipython.scipy.org/moin/Parallel_Computing
I have never used BPS, but am familiar with the model. Can
probably
get it working interactively with IPython as well.
Cheers,
Brian
> But I can't distinguish which one brings fewer limitations,
is
efficient in application and will develop in future. > My favorite is BSP, but I can't find a package for present
SciPy
and NumPy. > > Could anybody give me a recommendation, which way I should
go.
> > Thanks. > Fie Pye > > > > > --------------------------------------- > Máme další spot P.Čtvrtníčka. Podívejte se www.neuservis.cz <http://www.neuservis.cz> > > > _______________________________________________ > SciPy-user mailing list > SciPy-user@scipy.org <mailto:SciPy-user@scipy.org> >
http://projects.scipy.org/mailman/listinfo/scipy-user
<http://projects.scipy.org/mailman/listinfo/scipy-user >
> _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org <mailto: SciPy-user@scipy.org>
http://projects.scipy.org/mailman/listinfo/scipy-user
-- Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
--
Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
-- Flávio Codeço Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark