From walter at livinglogic.de Wed Nov 1 13:52:26 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 01 Nov 2006 19:52:26 +0100 Subject: [IPython-dev] IPython.Extensions on sys.path Message-ID: <4548ECEA.3020605@livinglogic.de> Hello list! Currently IPython/__init__.py adds IPython.Extensions to sys.path. IMHO this is bad, because it can lead to hard to detect bugs: If a module is imported from both locations you get two modules that look and smell the same, yet are totally unrelated (chaing one doesn't change the other). Furthermore modules in IPython.Extensions can shadow modules that are installed globally which leads to different behaviour depending on whether code runs in pure Python or in IPython. Is there a reason why IPython.Extensions is added? Could we drop this in the next IPython release? Servus, Walter From vivainio at gmail.com Wed Nov 1 16:30:20 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Wed, 1 Nov 2006 23:30:20 +0200 Subject: [IPython-dev] IPython.Extensions on sys.path In-Reply-To: <4548ECEA.3020605@livinglogic.de> References: <4548ECEA.3020605@livinglogic.de> Message-ID: <46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com> On 11/1/06, Walter D?rwald wrote: > Is there a reason why IPython.Extensions is added? Could we drop this in > the next IPython release? There is a reason - it makes the modules in the directory easy to import (import bah instead of from IPython.Extensions import bah), and we can ship modules there that could also be available as third party modules (e.g. 'path'). That way, the same code can be used in and out of ipython (instead of having to type "from IPython.Extensions.path import path' or even 'from IPython.path import path', which would be invalid if IPython was not installed). A case in point - the "pickleshare" module just imports top level "path", and as such doesn't require IPython to be usable. The module is available outside IPython, and I didn't feel like "forking" a version for IPython (or trying to import path both from top level and IPython). One alternative would perhaps be to declare all the stuff in Extensions as py_modules in setup.py. That way, at least with setuptools, they would be visible outside the IPython package as top level modules (yet still reside in the IPython egg directory). Is there really a problem, apart from the "path" module? If path is the only problem, we could add 'path' to 'py_modules' and remove it from Extensions. But the rest should IMHO stay as it is. Many modules in Extensions are unusable outside IPython anyway. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From walter at livinglogic.de Thu Nov 2 03:00:29 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Thu, 02 Nov 2006 09:00:29 +0100 Subject: [IPython-dev] IPython.Extensions on sys.path In-Reply-To: <46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com> References: <4548ECEA.3020605@livinglogic.de> <46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com> Message-ID: <4549A59D.8060307@livinglogic.de> Ville M. Vainio wrote: > On 11/1/06, Walter D?rwald wrote: > >> Is there a reason why IPython.Extensions is added? Could we drop this in >> the next IPython release? > > There is a reason - it makes the modules in the directory easy to > import (import bah instead of from IPython.Extensions import bah), and > we can ship modules there that could also be available as third party > modules (e.g. 'path'). That way, the same code can be used in and out > of ipython (instead of having to type "from IPython.Extensions.path > import path' or even 'from IPython.path import path', which would be > invalid if IPython was not installed). IMHO the proper way to treat such dependencies would be to document the dependency and let the user install the required module. With setuptools this can be done automatically: setup( ..., install_requires=[ "path.py >= 2.1", ], ) On Windows we already have a dependency on other packages (actually two: pyreadline and ctypes) anyway. > A case in point - the "pickleshare" module just imports top level > "path", and as such doesn't require IPython to be usable. The module > is available outside IPython, and I didn't feel like "forking" a > version for IPython (or trying to import path both from top level and > IPython). Would this mean that all modules that are just copies of third-party module and can work independently from IPython should be imported via "import foo" and modules that depend on IPython should be imported via "from IPython.Extensions import foo"? > One alternative would perhaps be to declare all the stuff in > Extensions as py_modules in setup.py. That way, at least with > setuptools, they would be visible outside the IPython package as top > level modules (yet still reside in the IPython egg directory). I'm not sure. At least there would no longer be identical copies of the same module. But installing IPython would then "overwrite" the original version of the module, i.e. when importing it you'll get the IPython version instead of the original one. > Is there really a problem, apart from the "path" module? If path is > the only problem, we could add 'path' to 'py_modules' and remove it > from Extensions. But the rest should IMHO stay as it is. Many modules > in Extensions are unusable outside IPython anyway. I'd like to use Philip Eby's simplegeneric module (http://cheeseshop.python.org/pypi/simplegeneric) for ipipe. xrepr(), xiter() and xattrs() could benefit from being generic functions. I already experimented in this direction some time ago, but now that a simple standalone generic function module is available it no longer makes sense to reinvent the wheel. Philip's OK with us bundling the module, as long as it doesn't conflict with any installed version of simplegeneric, which it would (but only when running inside IPython). Servus, Walter From vivainio at gmail.com Thu Nov 2 03:30:46 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Thu, 2 Nov 2006 10:30:46 +0200 Subject: [IPython-dev] IPython.Extensions on sys.path In-Reply-To: <4549A59D.8060307@livinglogic.de> References: <4548ECEA.3020605@livinglogic.de> <46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com> <4549A59D.8060307@livinglogic.de> Message-ID: <46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com> On 11/2/06, Walter D?rwald wrote: > IMHO the proper way to treat such dependencies would be to document the > dependency and let the user install the required module. With setuptools > this can be done automatically: > setup( > ..., > install_requires=[ > "path.py >= 2.1", > ], > ) We are still in "no setuptools required" zone (and will for a while, probably - it didn't end up in python 2.5). And path.py isn't installable via easy_install at this moment. I'd like to see all these handy little modules bundled with IPython instead. > On Windows we already have a dependency on other packages (actually two: > pyreadline and ctypes) anyway. ctypes is bundled with python 2.5, and pyreadline is easy_install-able. > Would this mean that all modules that are just copies of third-party > module and can work independently from IPython should be imported via > "import foo" and modules that depend on IPython should be imported via > "from IPython.Extensions import foo"? Yes. Though I'm not so sure about "from IPython.Extensions", it's up to the author of the code to determine which feels more comfortable. Technically, "Extensions" should all be IPython dependent (IPython extension plugins) but ATM it also contains stuff that is importable as top level modules when running inside IPython. > I'm not sure. At least there would no longer be identical copies of the > same module. But installing IPython would then "overwrite" the original > version of the module, i.e. when importing it you'll get the IPython > version instead of the original one. Yeah, I don't like that either. > I'd like to use Philip Eby's simplegeneric module > (http://cheeseshop.python.org/pypi/simplegeneric) for ipipe. xrepr(), > xiter() and xattrs() could benefit from being generic functions. I > already experimented in this direction some time ago, but now that a > simple standalone generic function module is available it no longer > makes sense to reinvent the wheel. > > Philip's OK with us bundling the module, as long as it doesn't conflict > with any installed version of simplegeneric, which it would (but only > when running inside IPython). Ok. Put it into IPython folder (not IPython/Extensions) and import it explicitly from there ("from IPython import simplegeneric"). I'm still -1 on removing Extensions from sys.path, though. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From walter at livinglogic.de Thu Nov 2 05:55:39 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Thu, 02 Nov 2006 11:55:39 +0100 Subject: [IPython-dev] IPython.Extensions on sys.path In-Reply-To: <46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com> References: <4548ECEA.3020605@livinglogic.de> <46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com> <4549A59D.8060307@livinglogic.de> <46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com> Message-ID: <4549CEAB.5050907@livinglogic.de> Ville M. Vainio wrote: > On 11/2/06, Walter D?rwald wrote: > >> IMHO the proper way to treat such dependencies would be to document the >> dependency and let the user install the required module. With setuptools >> this can be done automatically: >> setup( >> ..., >> install_requires=[ >> "path.py >= 2.1", >> ], >> ) > > We are still in "no setuptools required" zone (and will for a while, > probably - it didn't end up in python 2.5). And path.py isn't > installable via easy_install at this moment. Well, easy_install http://www.jorendorff.com/articles/python/path/path-2.1.zip works, but I agree that it would be better if path 2.1 was available from the cheeseshop. > I'd like to see all these > handy little modules bundled with IPython instead. OK. >> On Windows we already have a dependency on other packages (actually two: >> pyreadline and ctypes) anyway. > > ctypes is bundled with python 2.5, and pyreadline is easy_install-able. Maybe we should convince Jason Orendorff to upload a new version to the cheeseshop? >> Would this mean that all modules that are just copies of third-party >> module and can work independently from IPython should be imported via >> "import foo" and modules that depend on IPython should be imported via >> "from IPython.Extensions import foo"? > > Yes. Though I'm not so sure about "from IPython.Extensions", it's up > to the author of the code to determine which feels more comfortable. > Technically, "Extensions" should all be IPython dependent (IPython > extension plugins) but ATM it also contains stuff that is importable > as top level modules when running inside IPython. > >> I'm not sure. At least there would no longer be identical copies of the >> same module. But installing IPython would then "overwrite" the original >> version of the module, i.e. when importing it you'll get the IPython >> version instead of the original one. > > Yeah, I don't like that either. > >> I'd like to use Philip Eby's simplegeneric module >> (http://cheeseshop.python.org/pypi/simplegeneric) for ipipe. xrepr(), >> xiter() and xattrs() could benefit from being generic functions. I >> already experimented in this direction some time ago, but now that a >> simple standalone generic function module is available it no longer >> makes sense to reinvent the wheel. >> >> Philip's OK with us bundling the module, as long as it doesn't conflict >> with any installed version of simplegeneric, which it would (but only >> when running inside IPython). > > Ok. Put it into IPython folder (not IPython/Extensions) and import it > explicitly from there ("from IPython import simplegeneric"). OK, will do. > I'm still -1 on removing Extensions from sys.path, though. As long as we have a conflict free way of bundling third-party modules, I can live with that. Servus, Walter From vivainio at gmail.com Thu Nov 2 06:13:51 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Thu, 2 Nov 2006 13:13:51 +0200 Subject: [IPython-dev] IPython.Extensions on sys.path In-Reply-To: <4549CEAB.5050907@livinglogic.de> References: <4548ECEA.3020605@livinglogic.de> <46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com> <4549A59D.8060307@livinglogic.de> <46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com> <4549CEAB.5050907@livinglogic.de> Message-ID: <46cb515a0611020313u699fe901m6a13132a667b694c@mail.gmail.com> On 11/2/06, Walter D?rwald wrote: > Maybe we should convince Jason Orendorff to upload a new version to the > cheeseshop? That would be cool; though I don't think we are quite ready yet to assume everyone is able/willing to easy_install stuff. > As long as we have a conflict free way of bundling third-party modules, > I can live with that. At the moment "path.py" is the only one that could cause conflicts, but path.py has been relavitely stable forever, so it doesn't matter much what version of path.py gets used. For more volatile (and yet widely used) third party modules, it may indeed make more sense to use them from inside IPython package. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From fperez.net at gmail.com Fri Nov 3 03:14:06 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 3 Nov 2006 01:14:06 -0700 Subject: [IPython-dev] IPython.Extensions on sys.path In-Reply-To: <46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com> References: <4548ECEA.3020605@livinglogic.de> <46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com> <4549A59D.8060307@livinglogic.de> <46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com> Message-ID: On 11/2/06, Ville M. Vainio wrote: > On 11/2/06, Walter D?rwald wrote: > Yes. Though I'm not so sure about "from IPython.Extensions", it's up > to the author of the code to determine which feels more comfortable. > Technically, "Extensions" should all be IPython dependent (IPython > extension plugins) but ATM it also contains stuff that is importable > as top level modules when running inside IPython. > > > I'm not sure. At least there would no longer be identical copies of the > > same module. But installing IPython would then "overwrite" the original > > version of the module, i.e. when importing it you'll get the IPython > > version instead of the original one. > > Yeah, I don't like that either. > > > I'd like to use Philip Eby's simplegeneric module > > (http://cheeseshop.python.org/pypi/simplegeneric) for ipipe. xrepr(), > > xiter() and xattrs() could benefit from being generic functions. I > > already experimented in this direction some time ago, but now that a > > simple standalone generic function module is available it no longer > > makes sense to reinvent the wheel. > > > > Philip's OK with us bundling the module, as long as it doesn't conflict > > with any installed version of simplegeneric, which it would (but only > > when running inside IPython). > > Ok. Put it into IPython folder (not IPython/Extensions) and import it > explicitly from there ("from IPython import simplegeneric"). I'd suggest making a subdir in IPython for third-party modules, so that we can import them from there. It doesn't particularly hurt us to have to type an extra word in the import statements, and it is more self-documenting that these are simply pulled in for convenience. Something like IPython/contrib or 'external' can work. from IPython.external import simplegeneric lets us clearly know in the future that this comes from the outside. We can simply document to users which external modules we ship as a convenience, so they can use them from within ipython. > I'm still -1 on removing Extensions from sys.path, though. This one isn't terrible IMO because we're putting it, along with ~/.ipython, at the very end of sys.path, so the danger of it shadowing a user's expected directory is minimal. It will shadow an ImportError, but it won't cover another directory. What does bother me is that Python (I didn't know this) prepends any script's execution directory to sys.path. Try this: longs[~/test]> pwd /home/fperez/test longs[~/test]> cat spath.py import pprint,sys pprint.pprint(sys.path) longs[~/test]> python spath.py ['/home/fperez/test', '/home/fperez/test', '/home/fperez/tmp/local/lib/python2.4/site-packages', '/home/fperez/usr/lib/python', '/home/fperez/usr/lib/python2.4/site-packages', '/home/fperez/usr/local/lib/python', '/home/fperez/usr/local/lib/python2.4/site-packages', '/usr/local/lib/python2.4/site-packages', [etc...] I just added this code to the main ipython starter script: # Start by cleaning up sys.path: Python automatically inserts the script's # base directory into sys.path, at the front. This can lead to unpleasant # surprises. import sys sys.path.pop(0) Now, the front of sys.path is identical for a plain python and an ipython shell. The only difference is that under ipython, the very last two entries are: '/home/fperez/usr/lib/python2.4/site-packages/IPython/Extensions', '/home/fperez/.ipython'] which don't exist for plain Python. I don't see these as a big problem, though. If others disagree, let us know. Cheers, f From walter at livinglogic.de Fri Nov 3 05:42:51 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Fri, 03 Nov 2006 11:42:51 +0100 Subject: [IPython-dev] IPython.Extensions on sys.path In-Reply-To: References: <4548ECEA.3020605@livinglogic.de> <46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com> <4549A59D.8060307@livinglogic.de> <46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com> Message-ID: <454B1D2B.509@livinglogic.de> Fernando Perez wrote: > On 11/2/06, Ville M. Vainio wrote: >> On 11/2/06, Walter D?rwald wrote: > [...] >> > I'd like to use Philip Eby's simplegeneric module >> > (http://cheeseshop.python.org/pypi/simplegeneric) for ipipe. xrepr(), >> > xiter() and xattrs() could benefit from being generic functions. I >> > already experimented in this direction some time ago, but now that a >> > simple standalone generic function module is available it no longer >> > makes sense to reinvent the wheel. >> > >> > Philip's OK with us bundling the module, as long as it doesn't conflict >> > with any installed version of simplegeneric, which it would (but only >> > when running inside IPython). >> >> Ok. Put it into IPython folder (not IPython/Extensions) and import it >> explicitly from there ("from IPython import simplegeneric"). > > I'd suggest making a subdir in IPython for third-party modules, so > that we can import them from there. It doesn't particularly hurt us > to have to type an extra word in the import statements, and it is more > self-documenting that these are simply pulled in for convenience. > Something like IPython/contrib or 'external' can work. > > from IPython.external import simplegeneric > > lets us clearly know in the future that this comes from the outside. > We can simply document to users which external modules we ship as a > convenience, so they can use them from within ipython. OK, then I'll add it to a new external subpackage. > [...] Servus, Walter From rocky at panix.com Fri Nov 3 06:41:38 2006 From: rocky at panix.com (R. Bernstein) Date: Fri, 3 Nov 2006 06:41:38 -0500 Subject: [IPython-dev] ipython/pydb integeration - much progress, more still? Message-ID: <17739.10994.675434.379180@panix3.panix.com> Thanks for putting in the remaining patches for pydb integration! One small debug "print" slipped into magic_pdef around line 672 of Magic.py. (It was in my patch). Some other remaining things. I believe addressing the readline history problem obviates the need for *all* of the Python version-checking in Debugger.py. In particular, the code that starts this: # Ugly hack: for Python 2.3-2.4, we can't call the parent constructor, # because it binds readline and breaks tab-completion. This means we # have to COPY the constructor here. SVN checkin comments for this portion of the code are this: fperez - Cleanup [1786], which went in with unfinished stuff by accident fperez - Apply Ville's patch, closes #87 As mentioned previously, readline gets imported on Python 2.5's pdb unconditionally just as it does in 2.4, so I'm at a loss as to what was changed in Python 2.5's pdb that "fixed" things. And again, given the more general readline history fix, is this *still* needed? I tried a little test on Python 2.4 and don't see a problem. But then it's not clear to me from the above comments what the problem was or how one would try to recreate the problem observed. If this version testing is removed then pydb's post-mortem will work on all versions of Python. Next, I wonder about the need for having to issue "import ipy_pydb". Why not just uconditionally surround import via a "try/except" block? If it fails, no problem, there won't be a %pydb magic. Finally there's the problem of getting ipython's Pdb class used when calling pydb.runl rather than pydb's Pdb class, so that the ipython-specific changes get applied when calling via %pydb. Again, thanks for moving forward on this. From walter at livinglogic.de Fri Nov 3 09:07:18 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Fri, 03 Nov 2006 15:07:18 +0100 Subject: [IPython-dev] IPython.Extensions on sys.path In-Reply-To: References: <4548ECEA.3020605@livinglogic.de> <46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com> <4549A59D.8060307@livinglogic.de> <46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com> Message-ID: <454B4D16.90503@livinglogic.de> Fernando Perez wrote: > [...] > I'd suggest making a subdir in IPython for third-party modules, so > that we can import them from there. It doesn't particularly hurt us > to have to type an extra word in the import statements, and it is more > self-documenting that these are simply pulled in for convenience. > Something like IPython/contrib or 'external' can work. > > from IPython.external import simplegeneric OK, done. Should I move path.py too? Servus, Walter From vivainio at gmail.com Fri Nov 3 09:27:37 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Fri, 3 Nov 2006 16:27:37 +0200 Subject: [IPython-dev] IPython.Extensions on sys.path In-Reply-To: <454B4D16.90503@livinglogic.de> References: <4548ECEA.3020605@livinglogic.de> <46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com> <4549A59D.8060307@livinglogic.de> <46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com> <454B4D16.90503@livinglogic.de> Message-ID: <46cb515a0611030627r34a84137v19185c1b5bcc0a95@mail.gmail.com> On 11/3/06, Walter D?rwald wrote: > > from IPython.external import simplegeneric > > OK, done. Should I move path.py too? If you add 'external' to sys.path, yes. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From walter at livinglogic.de Fri Nov 3 10:21:53 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Fri, 03 Nov 2006 16:21:53 +0100 Subject: [IPython-dev] IPython.Extensions on sys.path In-Reply-To: <46cb515a0611030627r34a84137v19185c1b5bcc0a95@mail.gmail.com> References: <4548ECEA.3020605@livinglogic.de> <46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com> <4549A59D.8060307@livinglogic.de> <46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com> <454B4D16.90503@livinglogic.de> <46cb515a0611030627r34a84137v19185c1b5bcc0a95@mail.gmail.com> Message-ID: <454B5E91.7000808@livinglogic.de> Ville M. Vainio wrote: > On 11/3/06, Walter D?rwald wrote: > >> > from IPython.external import simplegeneric >> >> OK, done. Should I move path.py too? > > If you add 'external' to sys.path, yes. But then we're back to where we started: A module in IPython.external shadows a global module. Servus, Walter From walter at livinglogic.de Fri Nov 3 11:23:52 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Fri, 03 Nov 2006 17:23:52 +0100 Subject: [IPython-dev] IPython.Extensions on sys.path In-Reply-To: References: <4548ECEA.3020605@livinglogic.de> <46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com> <4549A59D.8060307@livinglogic.de> <46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com> Message-ID: <454B6D18.9020500@livinglogic.de> Fernando Perez wrote: > [...] > > I just added this code to the main ipython starter script: > > # Start by cleaning up sys.path: Python automatically inserts the script's > # base directory into sys.path, at the front. This can lead to unpleasant > # surprises. > import sys > sys.path.pop(0) > > Now, the front of sys.path is identical for a plain python and an > ipython shell. The only difference is that under ipython, the very > last two entries are: > > '/home/fperez/usr/lib/python2.4/site-packages/IPython/Extensions', > '/home/fperez/.ipython'] > > which don't exist for plain Python. I don't see these as a big problem, > though. > > If others disagree, let us know. This breaks running the start script directly. With the current version I get a sys.path that looks like this: ['', '/usr/local/lib/python2.5/site-packages/mpcp-1.5-py2.5.egg', '/usr/local/lib/python2.5/site-packages/docutils-0.4-py2.5.egg', '/usr/local/lib/python2.5/site-packages/ipython-0.7.2-py2.5.egg', '/usr/local/lib/python2.5/site-packages/setuptools-0.6c3-py2.5.egg', '/usr/local/lib/python2.5/site-packages/cssutils-0.9a6-py2.5.egg', '/usr/local/lib/python2.5/site-packages/simplegeneric-0.6-py2.5.egg', '/usr/local/lib/python2.5/site-packages/path-2.1-py2.5.egg', '/var/home/walter/dist-py', '/usr/local/lib/svn-python', '/usr/local/lib/python25.zip', '/usr/local/lib/python2.5', '/usr/local/lib/python2.5/plat-linux2', '/usr/local/lib/python2.5/lib-tk', '/usr/local/lib/python2.5/lib-dynload', '/usr/local/lib/python2.5/site-packages', '/var/home/walter/pythonroot', '/usr/local/lib/python2.5/site-packages/ipython-0.7.2-py2.5.egg/IPython/Extensions', '/var/home/walter/.ipython'] Note that the IPython/Extensions directory from the globally installed IPython has been added. Commenting out the sys.path.pop(0) gives me: ['', '/var/home/walter/checkouts/IPython/current', '/usr/local/lib/python2.5/site-packages/mpcp-1.5-py2.5.egg', '/usr/local/lib/python2.5/site-packages/docutils-0.4-py2.5.egg', '/usr/local/lib/python2.5/site-packages/ipython-0.7.2-py2.5.egg', '/usr/local/lib/python2.5/site-packages/setuptools-0.6c3-py2.5.egg', '/usr/local/lib/python2.5/site-packages/cssutils-0.9a6-py2.5.egg', '/usr/local/lib/python2.5/site-packages/simplegeneric-0.6-py2.5.egg', '/usr/local/lib/python2.5/site-packages/path-2.1-py2.5.egg', '/var/home/walter/dist-py', '/usr/local/lib/svn-python', '/usr/local/lib/python25.zip', '/usr/local/lib/python2.5', '/usr/local/lib/python2.5/plat-linux2', '/usr/local/lib/python2.5/lib-tk', '/usr/local/lib/python2.5/lib-dynload', '/usr/local/lib/python2.5/site-packages', '/var/home/walter/pythonroot', '/var/home/walter/checkouts/IPython/current/IPython/Extensions', '/var/home/walter/.ipython'] Servus, Walter From john.d.kimball at navy.mil Sat Nov 4 18:09:38 2006 From: john.d.kimball at navy.mil (Kimball, John D CIV NSWCDL G33) Date: Sat, 4 Nov 2006 18:09:38 -0500 Subject: [IPython-dev] IPython1 unit tests fail Message-ID: I installed ipython1 and dependecies in accordance with the INSTALL file. The unit tests fail, and I'm posting the problem here per the directions in the INSTALL file. Thanks for your assistance. You all are doing awesome work! I'm using: enthon 1.0.0 (python 2.4.3) Windows 2000 SP4 ZopeInterface-3.1.0c1 The versions of Twisted and Nevow current as of 4PM 11-04-06. I get the following result: Running 29 tests. C:\Python24\lib\site-packages\twisted\trial\runner.py:243: exceptions.DeprecationWarning: log.startKeepingErrors is deprecated since Twisted 2.5 ipython1.test.test_controllerpb BasicControllerPBTest testDeferreds ... [ERROR] cleanup errors [ERROR] C:\Python24\lib\site-packages\twisted\trial\reporter.py:201: twisted.trial.reporter.BrokenTestCaseWarning: REACTOR UNCLEAN! traceback(s) follow: Traceback (most recent call last): File "C:\Python24\Lib\site-packages\twisted\trial\unittest.py", line 651, in _classCleanUp util._Janitor().postClassCleanup() File "C:\Python24\Lib\site-packages\twisted\trial\util.py", line 68, in postClassCleanup 'cleanPending', 'cleanThreads') File "C:\Python24\Lib\site-packages\twisted\trial\util.py", line 72, in _dispatch getattr(self, "do_%s" % attr)() File "C:\Python24\Lib\site-packages\twisted\trial\util.py", line 126, in do_cleanReactor raise DirtyReactorError(' '.join(s)) twisted.trial.util.DirtyReactorError: THIS WILL BECOME AN ERROR SOON! reactor left in unclean state, the following Selectables were left over: < of twisted.spread.pb.PBServerFactory on 10111> < to ('127.0.0.1', 10111) at 18b1ab0> testExecute ... [ERROR] testInterfaces ... [ERROR] testPullNamespace ... [ERROR] testPushPull ... [ERROR] testPushPullSerialized ... [ERROR] testResult ... [ERROR] testScatterGather ... [ERROR] ipython1.test.test_controllerservice BasicControllerServiceTest testDeferreds ... [ERROR] testExecute ... [ERROR] testInterfaces ... [ERROR] testPullNamespace ... [ERROR] testPushPull ... [ERROR] testPushPullSerialized ... [ERROR] testResult ... [ERROR] testScatterGather ... [ERROR] [ERROR] ipython1.test.test_engineservice BasicEngineServiceTest testCompletedEmptyEngine ... [ERROR] testDeferreds ... [OK] testExecute ... [OK] testGetResult ... [OK] testInterfaces ... [ERROR] testPullNamespace ... [OK] testPushPull ... [OK] testPushPullSerialized ... [OK] testStatus ... [OK] [ERROR] ipython1.test.test_shell BasicShellTest testCommand ... [OK] testExecute ... [OK] testPutGet ... [OK] testUpdate ... [OK] ======================================================================== ======= [ERROR]: ipython1.test.test_controllerpb.BasicControllerPBTest.testDeferreds Traceback (most recent call last): File "C:\Python24\Lib\site-packages\ipython1\test\test_controllerpb.py", line 41, in setUp self.addEngine(1) File "C:\Python24\Lib\site-packages\ipython1\test\multienginetest.py", line 20, in addEngine e = es.completeEngine(es.EngineService(InteractiveShell)) File "C:\Python24\Lib\site-packages\ipython1\kernel\engineservice.py", line 157, in completeEngine zi.alsoProvides(engine, IEngineComplete) exceptions.AttributeError: 'module' object has no attribute 'alsoProvides' ======================================================================== ======= [ERROR]: ipython1.test.test_controllerpb.BasicControllerPBTest.testExecute Traceback (most recent call last): File "C:\Python24\Lib\site-packages\ipython1\test\test_controllerpb.py", line 36, in setUp self.s = reactor.listenTCP(10111, self.sf) File "C:\Python24\Lib\site-packages\twisted\internet\posixbase.py", line 386, in listenTCP p.startListening() File "C:\Python24\Lib\site-packages\twisted\internet\tcp.py", line 733, in startListening raise CannotListenError, (self.interface, self.port, le) twisted.internet.error.CannotListenError: Couldn't listen on any:10111: (10048, 'Address already in use'). ======================================================================== ======= and so on... (same errors repeated) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellisonbg.net at gmail.com Sat Nov 4 19:48:42 2006 From: ellisonbg.net at gmail.com (Brian Granger) Date: Sat, 4 Nov 2006 17:48:42 -0700 Subject: [IPython-dev] IPython1 unit tests fail In-Reply-To: References: Message-ID: <6ce0ac130611041648i66e66711md5e641b38382398b@mail.gmail.com> zi.alsoProvides(engine, IEngineComplete) exceptions.AttributeError: 'module' object has no attribute 'alsoProvides' This error from zope.interface makes me think that you have an older version of zope.interface than ZopeInterface-3.1.0c1. The alsoProvides method was new to this release. Are you sure you really have this release of zope.interface? You are venturing into new territory - all of IPython1 has been developed on Mac OS X/Linux/AIX. It should work on Windows, but I am not sure anyone has really testing everything out. Have any other Windows users tried IPython1? The first step is to get the right version of zope.interface though and see if that helps. Let me know what you find and we can debug these problems. Brian On 11/4/06, Kimball, John D CIV NSWCDL G33 wrote: > > > > I installed ipython1 and dependecies in accordance with the INSTALL file. > The unit tests fail, and I'm posting the problem here per the directions in > the INSTALL file. > > Thanks for your assistance. You all are doing awesome work! > > I'm using: > > enthon 1.0.0 (python 2.4.3) > Windows 2000 SP4 > ZopeInterface-3.1.0c1 > The versions of Twisted and Nevow current as of 4PM 11-04-06. > > I get the following result: > > Running 29 tests. > > C:\Python24\lib\site-packages\twisted\trial\runner.py:243: > exceptions.DeprecationWarning: log.startKeepingErrors is deprecated since > Twisted 2.5 > ipython1.test.test_controllerpb > BasicControllerPBTest > testDeferreds ... > [ERROR] > cleanup errors > [ERROR] > > C:\Python24\lib\site-packages\twisted\trial\reporter.py:201: > twisted.trial.reporter.BrokenTestCaseWarning: REACTOR > UNCLEAN! traceback(s) follow: > Traceback (most recent call last): > File > "C:\Python24\Lib\site-packages\twisted\trial\unittest.py", > line 651, in _classCleanUp > util._Janitor().postClassCleanup() > File > "C:\Python24\Lib\site-packages\twisted\trial\util.py", line > 68, in postClassCleanup > 'cleanPending', 'cleanThreads') > File > "C:\Python24\Lib\site-packages\twisted\trial\util.py", line > 72, in _dispatch > getattr(self, "do_%s" % attr)() > File > "C:\Python24\Lib\site-packages\twisted\trial\util.py", line > 126, in do_cleanReactor > raise DirtyReactorError(' '.join(s)) > twisted.trial.util.DirtyReactorError: THIS WILL BECOME AN > ERROR SOON! reactor left in unclean state, the following Selectables were > left over: < of > twisted.spread.pb.PBServerFactory on 10111> < 'twisted.internet.tcp.Client'> to ('127.0.0.1', 10111) at 18b1ab0> #0 on 10111> > > testExecute ... > [ERROR] > testInterfaces ... > [ERROR] > testPullNamespace ... > [ERROR] > testPushPull ... > [ERROR] > testPushPullSerialized ... > [ERROR] > testResult ... > [ERROR] > testScatterGather ... > [ERROR] > ipython1.test.test_controllerservice > BasicControllerServiceTest > testDeferreds ... > [ERROR] > testExecute ... > [ERROR] > testInterfaces ... > [ERROR] > testPullNamespace ... > [ERROR] > testPushPull ... > [ERROR] > testPushPullSerialized ... > [ERROR] > testResult ... > [ERROR] > testScatterGather ... > [ERROR] > [ERROR] > ipython1.test.test_engineservice > BasicEngineServiceTest > testCompletedEmptyEngine ... > [ERROR] > testDeferreds ... > [OK] > testExecute ... > [OK] > testGetResult ... > [OK] > testInterfaces ... > [ERROR] > testPullNamespace ... > [OK] > testPushPull ... > [OK] > testPushPullSerialized ... > [OK] > testStatus ... > [OK] > > [ERROR] > ipython1.test.test_shell > BasicShellTest > testCommand ... > [OK] > testExecute ... > [OK] > testPutGet ... > [OK] > testUpdate ... > [OK] > > =============================================================================== > [ERROR]: > ipython1.test.test_controllerpb.BasicControllerPBTest.testDeferreds > > Traceback (most recent call last): > File > "C:\Python24\Lib\site-packages\ipython1\test\test_controllerpb.py", > line 41, in setUp > self.addEngine(1) > File > "C:\Python24\Lib\site-packages\ipython1\test\multienginetest.py", > line 20, in addEngine > e = es.completeEngine(es.EngineService(InteractiveShell)) > File > "C:\Python24\Lib\site-packages\ipython1\kernel\engineservice.py", > line 157, in completeEngine > zi.alsoProvides(engine, IEngineComplete) > exceptions.AttributeError: 'module' object has no attribute 'alsoProvides' > =============================================================================== > [ERROR]: > ipython1.test.test_controllerpb.BasicControllerPBTest.testExecute > > Traceback (most recent call last): > File > "C:\Python24\Lib\site-packages\ipython1\test\test_controllerpb.py", > line 36, in setUp > self.s = reactor.listenTCP(10111, self.sf) > File > "C:\Python24\Lib\site-packages\twisted\internet\posixbase.py", > line 386, in listenTCP > p.startListening() > File > "C:\Python24\Lib\site-packages\twisted\internet\tcp.py", > line 733, in startListening > raise CannotListenError, (self.interface, self.port, le) > twisted.internet.error.CannotListenError: Couldn't listen > on any:10111: (10048, 'Address already in use'). > =============================================================================== > and so on... (same errors repeated) > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/ipython-dev > > > From john.d.kimball at navy.mil Sat Nov 4 20:32:39 2006 From: john.d.kimball at navy.mil (Kimball, John D CIV NSWCDL G33) Date: Sat, 4 Nov 2006 20:32:39 -0500 Subject: [IPython-dev] IPython1 unit tests fail In-Reply-To: <6ce0ac130611041648i66e66711md5e641b38382398b@mail.gmail.com> Message-ID: Brian; THE TESTS NOW PASS! I deleted the old zope.interface package and reinstalled 3.1.0c1, and now all the tests pass. I notice that the svn path in the INSTALL file checks out the Sumo version of twisted, which claims to include zope interface (version unspecified) though I couldn't find it in the source code. Perhaps setuptools was confused by the existing version? I installed in the order in the INSTALL file, and when the problem occurred I attempted to reinstall zope.interface but apparently it wasn't installed correctly until I deleted and reinstalled. Thanks for the prompt response. John -----Original Message----- From: Brian Granger [mailto:ellisonbg.net at gmail.com] Sent: Saturday, November 04, 2006 7:49 PM To: Kimball, John D CIV NSWCDL G33 Cc: ipython-dev at scipy.org Subject: Re: [IPython-dev] IPython1 unit tests fail zi.alsoProvides(engine, IEngineComplete) exceptions.AttributeError: 'module' object has no attribute 'alsoProvides' This error from zope.interface makes me think that you have an older version of zope.interface than ZopeInterface-3.1.0c1. The alsoProvides method was new to this release. Are you sure you really have this release of zope.interface? You are venturing into new territory - all of IPython1 has been developed on Mac OS X/Linux/AIX. It should work on Windows, but I am not sure anyone has really testing everything out. Have any other Windows users tried IPython1? The first step is to get the right version of zope.interface though and see if that helps. Let me know what you find and we can debug these problems. Brian On 11/4/06, Kimball, John D CIV NSWCDL G33 wrote: > > > > I installed ipython1 and dependecies in accordance with the INSTALL > file. The unit tests fail, and I'm posting the problem here per the > directions in the INSTALL file. > > Thanks for your assistance. You all are doing awesome work! > > I'm using: > > enthon 1.0.0 (python 2.4.3) > Windows 2000 SP4 > ZopeInterface-3.1.0c1 > The versions of Twisted and Nevow current as of 4PM 11-04-06. > > I get the following result: > > Running 29 tests. > > C:\Python24\lib\site-packages\twisted\trial\runner.py:243: > exceptions.DeprecationWarning: log.startKeepingErrors is deprecated > since Twisted 2.5 ipython1.test.test_controllerpb > BasicControllerPBTest > testDeferreds ... > [ERROR] > cleanup errors > [ERROR] > > C:\Python24\lib\site-packages\twisted\trial\reporter.py:201: > twisted.trial.reporter.BrokenTestCaseWarning: REACTOR UNCLEAN! > traceback(s) follow: Traceback (most recent call last): > File > "C:\Python24\Lib\site-packages\twisted\trial\unittest.py", > line 651, in _classCleanUp > util._Janitor().postClassCleanup() > File > "C:\Python24\Lib\site-packages\twisted\trial\util.py", line > 68, in postClassCleanup > 'cleanPending', 'cleanThreads') > File > "C:\Python24\Lib\site-packages\twisted\trial\util.py", line > 72, in _dispatch > getattr(self, "do_%s" % attr)() > File > "C:\Python24\Lib\site-packages\twisted\trial\util.py", line > 126, in do_cleanReactor > raise DirtyReactorError(' '.join(s)) > twisted.trial.util.DirtyReactorError: THIS WILL BECOME AN > ERROR SOON! reactor left in unclean state, the following Selectables were > left over: < of > twisted.spread.pb.PBServerFactory on 10111> < 'twisted.internet.tcp.Client'> to ('127.0.0.1', 10111) at 18b1ab0> #0 on 10111> > > testExecute ... > [ERROR] > testInterfaces ... > [ERROR] > testPullNamespace ... > [ERROR] > testPushPull ... > [ERROR] > testPushPullSerialized ... > [ERROR] > testResult ... > [ERROR] > testScatterGather ... > [ERROR] > ipython1.test.test_controllerservice > BasicControllerServiceTest > testDeferreds ... > [ERROR] > testExecute ... > [ERROR] > testInterfaces ... > [ERROR] > testPullNamespace ... > [ERROR] > testPushPull ... > [ERROR] > testPushPullSerialized ... > [ERROR] > testResult ... > [ERROR] > testScatterGather ... > [ERROR] > [ERROR] > ipython1.test.test_engineservice > BasicEngineServiceTest > testCompletedEmptyEngine ... > [ERROR] > testDeferreds ... > [OK] > testExecute ... > [OK] > testGetResult ... > [OK] > testInterfaces ... > [ERROR] > testPullNamespace ... > [OK] > testPushPull ... > [OK] > testPushPullSerialized ... > [OK] > testStatus ... > [OK] > > [ERROR] > ipython1.test.test_shell > BasicShellTest > testCommand ... > [OK] > testExecute ... > [OK] > testPutGet ... > [OK] > testUpdate ... > [OK] > > ====================================================================== > ========= > [ERROR]: > ipython1.test.test_controllerpb.BasicControllerPBTest.testDeferreds > > Traceback (most recent call last): > File > "C:\Python24\Lib\site-packages\ipython1\test\test_controllerpb.py", > line 41, in setUp > self.addEngine(1) > File > "C:\Python24\Lib\site-packages\ipython1\test\multienginetest.py", > line 20, in addEngine > e = es.completeEngine(es.EngineService(InteractiveShell)) > File > "C:\Python24\Lib\site-packages\ipython1\kernel\engineservice.py", > line 157, in completeEngine > zi.alsoProvides(engine, IEngineComplete) > exceptions.AttributeError: 'module' object has no attribute > 'alsoProvides' > ======================================================================== ======= > [ERROR]: > ipython1.test.test_controllerpb.BasicControllerPBTest.testExecute > > Traceback (most recent call last): > File > "C:\Python24\Lib\site-packages\ipython1\test\test_controllerpb.py", > line 36, in setUp > self.s = reactor.listenTCP(10111, self.sf) > File "C:\Python24\Lib\site-packages\twisted\internet\posixbase.py", > line 386, in listenTCP > p.startListening() > File > "C:\Python24\Lib\site-packages\twisted\internet\tcp.py", > line 733, in startListening > raise CannotListenError, (self.interface, self.port, le) > twisted.internet.error.CannotListenError: Couldn't listen > on any:10111: (10048, 'Address already in use'). > ======================================================================== ======= > and so on... (same errors repeated) > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/ipython-dev > > > From alantrick at gmail.com Sat Nov 4 20:53:30 2006 From: alantrick at gmail.com (Alan Trick) Date: Sat, 04 Nov 2006 17:53:30 -0800 Subject: [IPython-dev] breaking readline Message-ID: <1162691610.19366.12.camel@localhost> Hi I'm trying to create a wrapper for sys.stdout. I have some code (link at bottom) and it works fine on the standards python console, but not so well in ipython. In ipython it breaks readline so you end up seeing a lot of weird charachters and certain keys stop working. It works fine in code.InteractiveConsole. Anyone know what ipython is doing to cause this. n.b. I know my code has issues, like using open('/dev/stdout') and a few other thing, but as it is it *should* work. It's mostly a proof off concept Link: http://terminate.svn.sourceforge.net/viewvc/terminate/trunk/terminate/wrapper.py?revision=68&view=markup From fperez.net at gmail.com Sun Nov 5 14:25:31 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 5 Nov 2006 12:25:31 -0700 Subject: [IPython-dev] IPython.Extensions on sys.path In-Reply-To: <454B6D18.9020500@livinglogic.de> References: <4548ECEA.3020605@livinglogic.de> <46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com> <4549A59D.8060307@livinglogic.de> <46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com> <454B6D18.9020500@livinglogic.de> Message-ID: On 11/3/06, Walter D?rwald wrote: > Fernando Perez wrote: > > > [...] > > > > I just added this code to the main ipython starter script: > > > > # Start by cleaning up sys.path: Python automatically inserts the script's > > # base directory into sys.path, at the front. This can lead to unpleasant > > # surprises. > > import sys > > sys.path.pop(0) > > > > Now, the front of sys.path is identical for a plain python and an > > ipython shell. The only difference is that under ipython, the very > > last two entries are: > > > > '/home/fperez/usr/lib/python2.4/site-packages/IPython/Extensions', > > '/home/fperez/.ipython'] > > > > which don't exist for plain Python. I don't see these as a big problem, > > though. > > > > If others disagree, let us know. > > This breaks running the start script directly. With the current version > I get a sys.path that looks like this: Mmh, I also re-checked, and found that plain python also adds that directory to sys.path. I must have been mistaken before. I'll revert this change now then, to avoid problems. We should have a very clear policy on how we manipulate sys.path. Currently (once I revert the sys.path.pop() change) we have: 1. At startup, ipython's sys.path matches a plain python shell, except for the addition at the end of: '/home/fperez/usr/lib/python2.4/site-packages/IPython/Extensions', '/home/fperez/.ipython' Since these two go at the very end, they don't worry me too much: they are a convenience for interactive configuration and by being at the end, they can't shadow anything in the default sys.path. I could see removing the last one to be a bit more minimalistic, but I'm not sure it's worth it. 2. There's a more delicate issue: sys.path changes from %run. In order to mimic the running of scripts from a system prompt, %run used to do this: 2384 # find things also in current directory 2385 dname = os.path.dirname(fname) 2386 if not sys.path.count(dname): 2387 sys.path.append(dname) which I removed in r1874. I then realized that this is needed so that scripts can find local things, but because ipython persists the state of sys.path across runs (something that simply does not occur in a plain python run from the command line), these changes 'contaminate' sys.path for the rest of the session. I'm going to add this back, but in a modified form that will try to intelligently revert out any changes made to sys.path. I will leave any changes actually made by the user to persist, in case people have scripts they use to manipulate sys.path for further interactive use. Any feedback you may have on this is welcome. Cheers, f From python at koepsell.de Tue Nov 7 14:08:18 2006 From: python at koepsell.de (Kilian Koepsell) Date: Tue, 7 Nov 2006 11:08:18 -0800 Subject: [IPython-dev] ipython unit test failed Message-ID: <65235DB8-FA1A-4E1E-A4AA-E3A8D690CDC9@koepsell.de> hi, i just compiled and installed ipython1 on mac-intel. the 'trial ipython1' command gives the following two errors: > ====================================================================== > ========= > [ERROR]: ipython1.test.test_enginepb > > File "/sw/lib/python2.5/site-packages/twisted/python/reflect.py", > line 352, in namedAny > topLevelPackage = __import__(trialname) > : Non-ASCII character '\xe2' in > file /data/ipython1/ipython1/test/test_enginepb.py on line 7, but > no encoding declared; see http://www.python.org/peps/pep-0263.html > for details (test_enginepb.py, line 6) > ====================================================================== > ========= > [ERROR]: ipython1.test.test_enginevanilla > > File "/sw/lib/python2.5/site-packages/twisted/python/reflect.py", > line 352, in namedAny > topLevelPackage = __import__(trialname) > : Non-ASCII character '\xe2' in > file /data/ipython1/ipython1/test/test_enginevanilla.py on line 6, > but no encoding declared; see http://www.python.org/peps/ > pep-0263.html for details (test_enginevanilla.py, line 5) > ---------------------------------------------------------------------- > --------- > Ran 29 tests in 0.481s > > FAILED (errors=2, successes=29) i am using the following versions: mac osx 10.4.8 python 2.5 ipython 0.7.2 ipython1 from svn zopeinterface 3.1.0c1 twisted 2.4.0 any idea what's going wrong? thanks, kilian From fperez.net at gmail.com Tue Nov 7 14:47:49 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 7 Nov 2006 12:47:49 -0700 Subject: [IPython-dev] ipython unit test failed In-Reply-To: <65235DB8-FA1A-4E1E-A4AA-E3A8D690CDC9@koepsell.de> References: <65235DB8-FA1A-4E1E-A4AA-E3A8D690CDC9@koepsell.de> Message-ID: On 11/7/06, Kilian Koepsell wrote: > i am using the following versions: > python 2.5 > any idea what's going wrong? Yup, unfortunately the line above :( Says Glyf (Twisted lead developer): http://glyf.livejournal.com/62308.html The take-home part is: There's no release of Twisted that works with Python 2.5 yet. Sorry, but that's outside of our scope. Obviously they are aware of the issue, so I hope it will get resolved soon. I'll add a note to this effect on the wiki so others don't fall in the same trap. Cheers, f From vivainio at gmail.com Wed Nov 8 06:02:30 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Wed, 8 Nov 2006 13:02:30 +0200 Subject: [IPython-dev] regarding usefullness of ipconfig() & all Message-ID: <46cb515a0611080302t51bec859l46a5f498ddaa1fdb@mail.gmail.com> Just noted the added ipconfig() function and related magic: - _ip.options already exists for this, and can be directly manipulated. - I don't think we should have ipANYTHING in user namespace. They are there for backwards compatibility, bet the documented approach should be _ip.ANYTHING. I.e. ipmagic -> _ip.magic, etc. _ip, of course, is the ipapi object. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From fullung at gmail.com Wed Nov 8 18:52:18 2006 From: fullung at gmail.com (Albert Strasheim) Date: Thu, 9 Nov 2006 01:52:18 +0200 Subject: [IPython-dev] Issue in error path in case of invalid command Message-ID: <01ce01c70390$ec9ffb70$0a83a8c0@ratbert> Hello all I'm back to playing with IPython1 on Windows again. As per the instructions in http://projects.scipy.org/ipython/ipython/browser/ipython/branches/chainsaw/README I started up my controller and my engine. In a script I did: import ipython1.kernel.api as kernel rc = kernel.RemoteController(('localhost', 10201)) print rc.getIDs() When I run this, I get the following error: 2006/11/09 01:45 +0200 [VanillaEngineServerProtocol,1,127.0.0.1] Unhandled Error Traceback (most recent call last): File "C:\Python24\Lib\site-packages\twisted\python\log.py", line 48, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "C:\Python24\Lib\site-packages\twisted\python\log.py", line 33, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "C:\Python24\Lib\site-packages\twisted\python\context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "C:\Python24\Lib\site-packages\twisted\python\context.py", line 37, in callWithContext return func(*args,**kw) --- --- File "C:\Python24\Lib\site-packages\twisted\internet\selectreactor.py", line 139, in _doReadOrWrite why = getattr(selectable, method)() File "C:\Python24\Lib\site-packages\twisted\internet\tcp.py", line 362, in doRead return self.protocol.dataReceived(data) File "C:\Python24\Lib\site-packages\twisted\protocols\basic.py", line 99, in dataReceived self.doData() File "C:\Python24\Lib\site-packages\twisted\protocols\basic.py", line 62, in doData self.stringReceived(self.__buffer) File "C:\Python24\lib\site-packages\ipython1\kernel\enginevanilla.py", line 558, in stringReceived self.nextHandler(msg) File "C:\Python24\lib\site-packages\ipython1\kernel\enginevanilla.py", line 576, in dispatch self.dieLoudly('Command could not be dispatched: ' + msg) File "C:\Python24\lib\site-packages\ipython1\kernel\enginevanilla.py", line 588, in dieLoudly log.msg('Protocol Error [%i]: ' % id + repr(a)) exceptions.TypeError: int argument required 2006/11/09 01:45 +0200 [VanillaEngineServerProtocol,1,127.0.0.1] unregistered engine None 2006/11/09 01:45 +0200 [VanillaEngineServerProtocol,1,127.0.0.1] Unhandled Error Traceback (most recent call last): File "C:\Python24\Scripts\ipcontroller.py", line 51, in main reactor.run() File "C:\Python24\Lib\site-packages\twisted\internet\posixbase.py", line 218, in run self.mainLoop() File "C:\Python24\Lib\site-packages\twisted\internet\posixbase.py", line 229, in mainLoop self.doIteration(t) File "C:\Python24\Lib\site-packages\twisted\internet\selectreactor.py", line 133, in doSelect _logrun(selectable, _drdw, selectable, method, dict) --- --- File "C:\Python24\Lib\site-packages\twisted\python\log.py", line 48, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "C:\Python24\Lib\site-packages\twisted\python\log.py", line 33, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "C:\Python24\Lib\site-packages\twisted\python\context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "C:\Python24\Lib\site-packages\twisted\python\context.py", line 37, in callWithContext return func(*args,**kw) File "C:\Python24\Lib\site-packages\twisted\internet\selectreactor.py", line 149, in _doReadOrWrite self._disconnectSelectable(selectable, why, method=="doRead") File "C:\Python24\Lib\site-packages\twisted\internet\posixbase.py", line 256, in _disconnectSelectable selectable.connectionLost(failure.Failure(why)) File "C:\Python24\Lib\site-packages\twisted\internet\tcp.py", line 416, in connectionLost protocol.connectionLost(reason) File "C:\Python24\lib\site-packages\ipython1\kernel\enginevanilla.py", line 539, in connectionLost self.factory.unregisterEngine(self.id) File "C:\Python24\lib\site-packages\ipython1\kernel\enginevanilla.py", line 1026, in unregisterEngine return self.service.unregisterEngine(id) File "C:\Python24\Lib\site-packages\ipython1\kernel\controllerservice.py", line 304, in unregisterEngine log.msg("engine %i was not registered" % id) exceptions.TypeError: int argument required In this case, id is None and repr(a) is 'Command could not be dispatched: GETIDS'. Cheers, Albert From fullung at gmail.com Wed Nov 8 18:58:57 2006 From: fullung at gmail.com (Albert Strasheim) Date: Thu, 9 Nov 2006 01:58:57 +0200 Subject: [IPython-dev] Issue in error path in case of invalid command References: <01ce01c70390$ec9ffb70$0a83a8c0@ratbert> Message-ID: <01e101c70391$daee9ac0$0a83a8c0@ratbert> Hello all > I'm back to playing with IPython1 on Windows again. As per the > instructions > in > > http://projects.scipy.org/ipython/ipython/browser/ipython/branches/chainsaw/README > > I started up my controller and my engine. In a script I did: > > import ipython1.kernel.api as kernel > rc = kernel.RemoteController(('localhost', 10201)) > print rc.getIDs() > > When I run this, I get the following error: I should probably mention that I'm using IPython1 + dependencies fresh (as of right now) from SVN. Cheers, Albert From fperez.net at gmail.com Wed Nov 8 19:14:19 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 8 Nov 2006 17:14:19 -0700 Subject: [IPython-dev] Issue in error path in case of invalid command In-Reply-To: <01ce01c70390$ec9ffb70$0a83a8c0@ratbert> References: <01ce01c70390$ec9ffb70$0a83a8c0@ratbert> Message-ID: On 11/8/06, Albert Strasheim wrote: > Hello all > > I'm back to playing with IPython1 on Windows again. As per the instructions > in > > http://projects.scipy.org/ipython/ipython/browser/ipython/branches/chainsaw/README > > I started up my controller and my engine. In a script I did: > > import ipython1.kernel.api as kernel > rc = kernel.RemoteController(('localhost', 10201)) > print rc.getIDs() Quick question (Brian may have more ideas later). 10201 is not the default port, did you start the controller on that port? I doubt any of us has access to a windows box with enough development tools to test this, unfortunately. Up until now I've been able to do lightweight testing of ipython trunk under windows because it's pure python, but now we use things that require compilers and scary things like that. My windows VMware instance is basically a tax-preparation tool that runs one or two programs, and I don't have a real development setup there. I hope this is a problem that we can track from within a *nix box... Cheers, f From fullung at gmail.com Wed Nov 8 19:57:40 2006 From: fullung at gmail.com (Albert Strasheim) Date: Thu, 9 Nov 2006 02:57:40 +0200 Subject: [IPython-dev] Issue in error path in case of invalid command References: <01ce01c70390$ec9ffb70$0a83a8c0@ratbert> <6ce0ac130611081615x690a4d99p3487296572884731@mail.gmail.com> Message-ID: <000401c7039a$0e83bcf0$0a83a8c0@ratbert> Thanks, my bad. Still seems like this points to issues in the lines log.msg('Protocol Error [%i]: ' % id + repr(a)) and log.msg("engine %i was not registered" % id) since by connecting to this wrong port, id was None in both cases, which caused the TypeError to be raised. Cheers, Albert ----- Original Message ----- From: "Brian Granger" To: "Albert Strasheim" Sent: Thursday, November 09, 2006 2:15 AM Subject: Re: [IPython-dev] Issue in error path in case of invalid command > You are connecting on the wrong port. 10201 is the port the engine > connect on. You want port 10105. If you are running the controler on > localhost you can simply do: > > rc = kernel.RemoteController(kernel.defaultController) > > and it will do the right thing. From fullung at gmail.com Wed Nov 8 20:07:35 2006 From: fullung at gmail.com (Albert Strasheim) Date: Thu, 9 Nov 2006 03:07:35 +0200 Subject: [IPython-dev] RemoteController.test issue Message-ID: <000e01c7039b$70d63850$0a83a8c0@ratbert> Hello all With the right port in place, I can get the IDs of my two engines. However, when I run the test method of the RemoteController, I get: The IPython Controller magics only work within IPython. Connecting to controller: ('127.0.0.1', 10105) [0, 1] ### here I run rc.test() Running tests on 2 engines Testing execute... ?[0;32m[127.0.0.1:0]?[0;34m In [140]:?[0m a ?[0;32m[127.0.0.1:0]?[0;31m Out[140]:?[0m 1 ?[0;32m[127.0.0.1:0]?[0;34m In [141]:?[0m print a ?[0;32m[127.0.0.1:0]?[0;31m Out[141]:?[0m 1 ?[0;32m[127.0.0.1:1]?[0;34m In [45]:?[0m print a ?[0;32m[127.0.0.1:1]?[0;31m Out[45]:?[0m 1 ?[0;32m[127.0.0.1:0]?[0;34m In [142]:?[0m b-3 ?[0;32m[127.0.0.1:0]?[0;31m Out[142]:?[0m -1 ?[0;32m[127.0.0.1:1]?[0;34m In [46]:?[0m b-3 ?[0;32m[127.0.0.1:1]?[0;31m Out[46]:?[0m -1 Need something to do! Need something to do! execute OK Testing push/pull Need something to do! Need something to do! push/pull FAIL: list index out of range Traceback (most recent call last): File "C:\home\albert\work2\pyspkrec\frags\frags9.py", line 4, in ? rc.test() File "C:\Python24\Lib\site-packages\ipython1\kernel\controllerclient.py", line 171, in test self[1:max(ids)][0]['qwert'] = 3 File "C:\Python24\Lib\site-packages\ipython1\kernel\controllerclient.py", line 336, in __getitem__ return EngineProxy(self.rc, self._originalIDs[id]) IndexError: list index out of range Does this test work for you guys? Cheers, Albert From fperez.net at gmail.com Wed Nov 8 20:42:48 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 8 Nov 2006 18:42:48 -0700 Subject: [IPython-dev] RemoteController.test issue In-Reply-To: <000e01c7039b$70d63850$0a83a8c0@ratbert> References: <000e01c7039b$70d63850$0a83a8c0@ratbert> Message-ID: On 11/8/06, Albert Strasheim wrote: > Hello all > > With the right port in place, I can get the IDs of my two engines. However, > when I run the test method of the RemoteController, I get: > Does this test work for you guys? I get the same exact failure as you do. We'll look into it. You should be able to run things from the tutorial, for example. I'm starting to use this stuff in production work as well, so keep the complaints coming :) Since I'm on the same boat, we'll be fixing them as they come. Cheers, f From fullung at gmail.com Wed Nov 8 22:25:57 2006 From: fullung at gmail.com (Albert Strasheim) Date: Thu, 9 Nov 2006 05:25:57 +0200 Subject: [IPython-dev] pushAll and executeAll speed issue Message-ID: <009d01c703ae$c5bc6700$0a83a8c0@ratbert> Hello all I'm doing some experiments with IPython1 on Windows. I'm using DeinoMPI's GUI on top of mpi4py to start two kernels on my Core 2 Duo system. They connect to the controller without problems. I make a RemoteController and off I go. This is too cool. (By the way: as I understand it, MPI is currently only being used to start the kernels. From what I can see, it is not yet being used inside things like pushAll, etc. Is this the next step to use the MPI functions if they are available? Anybody working on this? If not, maybe I can try my hand at it. Any hints or suggestions on where to start appreciated.) Anyway, back to my experiment. The script I'm using is attached. It starts off with some local benchmarks running on a single core. I get: cPickle dumps time 4.95649055955 cPickle loads time 2.39126201794 local map time 10.3841574837 Then I push to all and time it. Here I get: total pushAll time 70.6287321433 I was expecting something proportional to dumpsTime+N*loadsTime (if I understand correctly that it is serially pushing out the data to each of the N kernels) or maybe even less. I noticed that this section gets dramatically slower even with a relatively small increase in data size. While this is busy, I see Python using up all the CPU on one core as expected, but I also see that the Windows "System" process is very busy. Next up, the timing over the executeAll call that does the map: executeAll time for map 139.186242386 but the kernels print: engine map time 12.8248084603 engine map time 11.3900467797 which is only slightly more than the single core time. However, I have to wonder where the heck the rest of the time went?! :-) Do you guys see similar timings on Linux? Any thoughts on where my time is going? Could it be Twisted? I tried with the latest from SVN and the official 2.4.0 release for Windows. Cheers, Albert -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: frags9.py URL: From ellisonbg.net at gmail.com Wed Nov 8 22:50:18 2006 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 8 Nov 2006 20:50:18 -0700 Subject: [IPython-dev] Issue in error path in case of invalid command In-Reply-To: <000401c7039a$0e83bcf0$0a83a8c0@ratbert> References: <01ce01c70390$ec9ffb70$0a83a8c0@ratbert> <6ce0ac130611081615x690a4d99p3487296572884731@mail.gmail.com> <000401c7039a$0e83bcf0$0a83a8c0@ratbert> Message-ID: <6ce0ac130611081950r69d568d7t36302a848e5a53e7@mail.gmail.com> Even though you were connecting to the wrong port we need to add some error checking code so that the controller doesn't die and the error messages are more understandable. Brian On 11/8/06, Albert Strasheim wrote: > Thanks, my bad. > > Still seems like this points to issues in the lines > > log.msg('Protocol Error [%i]: ' % id + repr(a)) > > and > > log.msg("engine %i was not registered" % id) > > since by connecting to this wrong port, id was None in both cases, which > caused the TypeError to be raised. > > Cheers, > > Albert > > ----- Original Message ----- > From: "Brian Granger" > To: "Albert Strasheim" > Sent: Thursday, November 09, 2006 2:15 AM > Subject: Re: [IPython-dev] Issue in error path in case of invalid command > > > > You are connecting on the wrong port. 10201 is the port the engine > > connect on. You want port 10105. If you are running the controler on > > localhost you can simply do: > > > > rc = kernel.RemoteController(kernel.defaultController) > > > > and it will do the right thing. > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/ipython-dev > From ellisonbg.net at gmail.com Wed Nov 8 23:01:55 2006 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 8 Nov 2006 21:01:55 -0700 Subject: [IPython-dev] RemoteController.test issue In-Reply-To: References: <000e01c7039b$70d63850$0a83a8c0@ratbert> Message-ID: <6ce0ac130611082001r540c4400mc8dc2d76bf0fde41@mail.gmail.com> The issue is that the test() method requires at least 4 engines to run. I just committed a change to trunk that tests how many engines you have running and raises an exception if you don't have at least 4. It prints an error message indicating what to do. The tests should pass with 4 engines running. Cheers, Brian On 11/8/06, Fernando Perez wrote: > On 11/8/06, Albert Strasheim wrote: > > Hello all > > > > With the right port in place, I can get the IDs of my two engines. However, > > when I run the test method of the RemoteController, I get: > > > Does this test work for you guys? > > I get the same exact failure as you do. We'll look into it. > > You should be able to run things from the tutorial, for example. I'm > starting to use this stuff in production work as well, so keep the > complaints coming :) Since I'm on the same boat, we'll be fixing them > as they come. > > Cheers, > > f > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/ipython-dev > From fperez.net at gmail.com Wed Nov 8 23:29:09 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 8 Nov 2006 21:29:09 -0700 Subject: [IPython-dev] pushAll and executeAll speed issue In-Reply-To: <009d01c703ae$c5bc6700$0a83a8c0@ratbert> References: <009d01c703ae$c5bc6700$0a83a8c0@ratbert> Message-ID: Hey Albert, Brian will probably pitch in later with more comments, since he knows the underbelly of this code better than I do. But in the meantime... On 11/8/06, Albert Strasheim wrote: > Hello all > > I'm doing some experiments with IPython1 on Windows. I'm using DeinoMPI's > GUI on top of mpi4py to start two kernels on my Core 2 Duo system. They > connect to the controller without problems. I make a RemoteController and > off I go. This is too cool. > > (By the way: as I understand it, MPI is currently only being used to start > the kernels. From what I can see, it is not yet being used inside things > like pushAll, etc. Is this the next step to use the MPI functions if they > are available? Anybody working on this? If not, maybe I can try my hand at > it. Any hints or suggestions on where to start appreciated.) There is MPI code in there already, but currently it is not being used for many core things. We should intelligently take advantage of it when present, while gracefully falling back to pickle when MPI is not there. > Anyway, back to my experiment. The script I'm using is attached. > > It starts off with some local benchmarks running on a single core. I get: > > cPickle dumps time 4.95649055955 > cPickle loads time 2.39126201794 > local map time 10.3841574837 > > Then I push to all and time it. Here I get: > > total pushAll time 70.6287321433 > > I was expecting something proportional to dumpsTime+N*loadsTime (if I > understand correctly that it is serially pushing out the data to each of the > N kernels) or maybe even less. I noticed that this section gets dramatically > slower even with a relatively small increase in data size. While this is > busy, I see Python using up all the CPU on one core as expected, but I also > see that the Windows "System" process is very busy. > > Next up, the timing over the executeAll call that does the map: > > executeAll time for map 139.186242386 > > but the kernels print: > > engine map time 12.8248084603 > engine map time 11.3900467797 > > which is only slightly more than the single core time. However, I have to > wonder where the heck the rest of the time went?! :-) > > Do you guys see similar timings on Linux? Any thoughts on where my time is > going? Could it be Twisted? I tried with the latest from SVN and the > official 2.4.0 release for Windows. First, a numpy aside. Note that I had to change the original line x = N.arange... for x = range(), because otherwise I was getting zillions of Warning: overflow encountered in long_scalars It seems to come from this: numpy long scalars don't gracefully coerce to Python longs: In [7]: x[-1] Out[7]: 99999 In [8]: x[-1]*x[-1] Warning: overflow encountered in long_scalars Out[8]: 1409865409 while the answer should be: In [22]: y=99999 In [23]: y*y Out[23]: 9999800001L But that's for the numpy list :) Here's what is going on, most likely. The size of that enormous pickled list is very large: In [24]: x = range(10000000) In [25]: len(cPickle.dumps(x)) Out[25]: 98888896 That's a 100MB string, that needs to be shipped over the wire. At this point, we are probably being hurt by Twisted, which makes unnecessary copies of very large objects when transmitting them. I know Brian and Min spent some time looking at this, but I'm not sure how far they got; I think it wasn't too easy to fix directly, though it /will/ need to be done eventually. I see two possibilities: a) Finding how to make Twisted-based sending of pickled strings as efficient as possible, and with minimal copies. b) Taking better advantage of MPI if available, so that the string buffer is sent using MPI directly. I see fairly similar results to yours: In [10]: run frags9.py ('127.0.0.1', 10105) Connecting to controller: ('127.0.0.1', 10105) [0, 1] cPickle dumps time 5.15 cPickle loads time 5.92 local map time 7.06 [127.0.0.1:0] In [29]: import time [127.0.0.1:1] In [22]: import time pushing to all... total pushAll time 1.24 [127.0.0.1:0] In [30]: t = time.clock() [127.0.0.1:1] In [23]: t = time.clock() executing on all... [127.0.0.1:0] In [31]: z = map(lambda y:y*y, x) [127.0.0.1:1] In [24]: z = map(lambda y:y*y, x) executeAll time for map 0.0 [127.0.0.1:0] In [32]: t = time.clock() - t [127.0.0.1:1] In [25]: t = time.clock() - t total time for push and execute 1.24 [127.0.0.1:0] In [33]: print "engine map time", t [127.0.0.1:0] Out[33]: engine map time 7.16 [127.0.0.1:1] In [26]: print "engine map time", t [127.0.0.1:1] Out[26]: engine map time 7.19 (0, 33, 'print "engine map time", t', 'engine map time 7.16\n', '') (1, 26, 'print "engine map time", t', 'engine map time 7.19\n', '') I also (under linux) saw an enormous amount of CPU usage from the kernel; that's typical of large memory allocations being done. Further, by the end of the runs my 2GB box was almost out of RAM. For usage of read-only data in multicore machines, it would also be nice to expose a shared-memory interface, but I'm not really up to speed on the details of doing that, and whether Python exposes an API to allow it to be done portably. Basically, I think that we're starting to see where the performance bottlenecks are. This is pre-alpha, after all ;) We'll keep working on it... Cheers, f From ellisonbg.net at gmail.com Thu Nov 9 01:14:41 2006 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 8 Nov 2006 23:14:41 -0700 Subject: [IPython-dev] pushAll and executeAll speed issue In-Reply-To: <009d01c703ae$c5bc6700$0a83a8c0@ratbert> References: <009d01c703ae$c5bc6700$0a83a8c0@ratbert> Message-ID: <6ce0ac130611082214q6dfba628od9a3b2d2d3347ea5@mail.gmail.com> > I'm doing some experiments with IPython1 on Windows. I'm using DeinoMPI's > GUI on top of mpi4py to start two kernels on my Core 2 Duo system. They > connect to the controller without problems. I make a RemoteController and > off I go. This is too cool. Yes, cool, but it is also dangerous.....read on... > (By the way: as I understand it, MPI is currently only being used to start > the kernels. From what I can see, it is not yet being used inside things > like pushAll, etc. Is this the next step to use the MPI functions if they > are available? Anybody working on this? If not, maybe I can try my hand at > it. Any hints or suggestions on where to start appreciated.) There are two ways one might way to use MPI: 1. The send data between engines. This is supported already through the usage of various python mpi bindings like mpi4py. Our docs have some details on how to get started with this, but it can be subtle depending on your mpi implementation etc. Please feel free to bug us about this. 2. To send data between the client (your python script) and the engines. At first this may seem attractive, but it is not really what you want to do. The reason is that then you IPython/Python script also has to be part of the MPi universe. That make it impossible to do many important things: - Run your script on your laptop, but run the engines/controller on a remote cluster. - Disconnect/reconnect to/from the controller. - multiple users connecting to the controller Because of these reason, we feel that it is a "feature" that MPi is not used to send data between the client (your python script or an interactive session) and the engines. With that said, you shouldn't be sending lots of data between the client and engines anyway - see below for more discussion of this... > Anyway, back to my experiment. The script I'm using is attached. > > It starts off with some local benchmarks running on a single core. I get: > > cPickle dumps time 4.95649055955 > cPickle loads time 2.39126201794 > local map time 10.3841574837 > > Then I push to all and time it. Here I get: > > total pushAll time 70.6287321433 > > I was expecting something proportional to dumpsTime+N*loadsTime (if I > understand correctly that it is serially pushing out the data to each of the > N kernels) or maybe even less. I noticed that this section gets dramatically > slower even with a relatively small increase in data size. While this is > busy, I see Python using up all the CPU on one core as expected, but I also > see that the Windows "System" process is very busy. > > Next up, the timing over the executeAll call that does the map: > > executeAll time for map 139.186242386 > > but the kernels print: > > engine map time 12.8248084603 > engine map time 11.3900467797 > > which is only slightly more than the single core time. However, I have to > wonder where the heck the rest of the time went?! :-) > > Do you guys see similar timings on Linux? Any thoughts on where my time is > going? Could it be Twisted? I tried with the latest from SVN and the > official 2.4.0 release for Windows. The current incarnation of your script has a few problems: 1. It creates multiple copies of a large object on 4 processes on your system (client, controller, and 2 engines). Even though the object is "not that big" when you have multple copies around the memory usage gets out of controller. Fernando was seeing his 2 Gb system nearly maxed out. It is even worse though than just 4 copies. If you go through your script you will see that the client and each engine has 2-3 copies of the object. For certain periods of time, the controller can also have multiple copies. Most of these problems (for objects the size of yours) would go away if the controller and engines were all running on different systems. 2. There is no real parallelism being expressed. This is because pushAll sends the entire object to each engine. Are you meaning to use scatterAll? This partitions a sequence and sends the partitions to each engine. 3. One of the golden rules of parallel computing is "never send something over the wire that can easily be recreated in place" The following code is much better: rc.execute(0,'a = N.arange(0,50000)') rc.execute(1,'a = N.arnage(50000, 100000)') rc.executeAll('b = a*a') result = gatherAll('b') # Now gather the result locally With this you have full parallelism and only three small strings (not the actual objects) are sent over the wire. Code like this will scale to many processes. We have really tried to make it easy to send objects between the client and engines. But, this usage pattern is not something we are used to. By this, I mean it is *very* easy (I have done it many times with IPython1 :)) to get carried away with how easy it is to send things around and forget about the realities of what we are really doing underneath the hood. With that said, we are very aware of a need to improve how IPython handles these cases. There are lots of optimizaitons we can and need to do and we also need to provide some protections that prevent crazy things from being done too easily. We have out work cut out or us and having users like yourself hammering on things makes our job easier. Cheers, Brian From fullung at gmail.com Thu Nov 9 17:50:42 2006 From: fullung at gmail.com (Albert Strasheim) Date: Fri, 10 Nov 2006 00:50:42 +0200 Subject: [IPython-dev] Suggestions for implementing parallel algorithms? Message-ID: <004e01c70451$7c37eb10$0a83a8c0@ratbert> Hello all I'm getting started with using IPython1 and mpi4py for implementing parallel algorithms for speaker verification. I'm mostly dealing with data from the NIST Speaker Recognition Evaluation, which entails thousands of speaker models to train and test. There are two expectation-maximization algorithms that are of interest to me: k-means clustering and Gaussian Mixture Model training. Both of these can be implemented in parallel by scattering the training data over a bunch of nodes, calculating some statistics, combining the stats in some way and repeating this process for a maximum number of iterations or until convergence is attained. I currently have these algorithms implemented on top of NumPy. For every algorithm I have a pure-Python version. The Python class implements the public interface and does argument checking and other housekeeping and the defers to "private" methods to get the real work done. To get some decent speed, some of these "private" methods also have C implementations. To get a fast version of the algorithm, I mix a class containing only these "private" methods that call through to C into my pure-Python class. As an example, in the k-means case, this means I end up with two classes, KMeansEstimator (pure Python) and CKMeansEstimator (Python on top with some C mixed in). I would like to adapt these algorithms to run in parallel using IPython1. Some details about my problem: for training my speaker models, I can simply train a number of speaker models per node. This parallelises easily -- different nodes do different speakers. However, for training my world model, I would like to involve all the nodes to work on the same model. This is necessary because to train the world model, I have tens to hunderds of hours of speech whereas the speaker models are adapted from the world model using only a few seconds to a few minutes of speech. A naive way to implement this in parallel would be to copy my existing implementation and call RemoteController's scatterAll/executeAll/gatherAll in places where the original algorithm does the loop over the data (the expectation step). I'm hoping I can avoid this duplication. My first idea is to make something like a LocalController that implements IPython1's IController interface in a way that makes sense for single node operation. This way, I can implement my algorithm once in terms of IController operations, test it easily, and later by simply setting a controller property on a instance of the class implementing the algorithm, decide whether it runs on a single node or in parallel. How do you guys handle these issues in your code? Any suggestions would be appreciated. Cheers, Albert P.S. For an example implementation of k-means on top of MPI, see http://www.cs.umn.edu/~mnjoshi/PKMeans.pdf From ellisonbg.net at gmail.com Fri Nov 10 00:34:49 2006 From: ellisonbg.net at gmail.com (Brian Granger) Date: Thu, 9 Nov 2006 22:34:49 -0700 Subject: [IPython-dev] Suggestions for implementing parallel algorithms? In-Reply-To: <004e01c70451$7c37eb10$0a83a8c0@ratbert> References: <004e01c70451$7c37eb10$0a83a8c0@ratbert> Message-ID: <6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com> Albert, This sounds like a nice application of IPython1. Fernando and I have had a number of talks about this exact issue lately. One things I should say before all else. In spite of there being years of research in parallel computing and algorithms, there is basically no research on *interactive* parallel computing. I am not sure you need your application to be interactive, but if you do, at some level, you are in new territory. With that said, there are some things to keep in mind. First, it is important to realize that all the previous work done on parallel algorithm/application development still applies even though the result can be used interactively. For example, if you need to move data around between the ipython engines, you should still use MPI - and all the guidelines for using MPI still apply. The new and interesting question is really "how and where do you want to interact with the parallel application as a human user?" There are many models of how you would want your application abstracted for interactive usage. And at some level, you may want to have the interactive API very different from the underlying computational model and parallel algorithm. You may want to hide the parallelism or you may find it better to show it explicitely in the API. In my own work, I have tended to factor my application into units that can perform the basic computational tasks for both a serial and parallel versions of the code. I then use these as building blocks to build the parallel and serial version. If the low level components are factored well, the high level algorithm is typically very short and I don't mind maintaining both a serial and a parallel version. For many things I do like the scatterAll/executeAll/gatherAll style of computation - it is extremely lightweight and easy to implement. The one thing to be careful of though is to not use this approach when MPI is more appropriate. Testing the scaling of your application with quickly reveal if there are problems like this. > I'm hoping I can avoid this duplication. My first idea is to make something > like a LocalController that implements IPython1's IController interface in a > way that makes sense for single node operation. This way, I can implement my > algorithm once in terms of IController operations, test it easily, and later > by simply setting a controller property on a instance of the class > implementing the algorithm, decide whether it runs on a single node or in > parallel. I had not thought of that before, but it does make sense. It is sort of similar to building objects that hide whether the object is being used in a parallel/serial context. It is surely worth trying this approach, but I am not sure how it would turn out in your case. I don't know if this helps, but I would love to see what you end up trying and what you find most useful - I am curious about all these things myself. Brian > How do you guys handle these issues in your code? Any suggestions would be > appreciated. > > Cheers, > > Albert > > P.S. For an example implementation of k-means on top of MPI, see > http://www.cs.umn.edu/~mnjoshi/PKMeans.pdf > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/ipython-dev > From gael.varoquaux at normalesup.org Fri Nov 10 03:57:05 2006 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 10 Nov 2006 09:57:05 +0100 Subject: [IPython-dev] %bg %run -i foo.py Message-ID: <20061110085704.GA16897@clipper.ens.fr> It seems this doesn't work. I am missing an obvious way to do this. Elsewhere this would definitely be a nice feature to add. Another nice magic to add would be a way to reload a module in a brutal way. Let me explain. I have all the functions that I use often for interactive work in a few modules. Every once so ever I change one of them. Of course as I am a bad boy, I have imported this module doing a "from foo import *", so I cannot reload it using "reload(foo)". The best way I have found to reload it is something like import foo %run -i foo.__file__ # Well, this doesn't work, you have to copy and paste foo.__file__ here, # as %run cannot take python variables as arguments. I don't know whether I am clear on that, and I don't know if I am the only one who needs such a magic but I for one would love to see this in ipython. Cheers, Ga?l From steve at shrogers.com Fri Nov 10 22:37:50 2006 From: steve at shrogers.com (Steven H. Rogers) Date: Fri, 10 Nov 2006 20:37:50 -0700 Subject: [IPython-dev] IPython1 on PS3/YDL Message-ID: <4555458E.7080607@shrogers.com> The Sony PlayStation 3 running Yellow Dog Linux would seem to be a good platform for IPyothon1. Has anyone given any thought to this? Regards, Steve -- Steven H. Rogers, Ph.D., steve at shrogers.com Weblog: http://shrogers.com/weblog "He who refuses to do arithmetic is doomed to talk nonsense." -- John McCarthy From rocky at panix.com Sat Nov 11 01:21:24 2006 From: rocky at panix.com (R. Bernstein) Date: Sat, 11 Nov 2006 01:21:24 -0500 Subject: [IPython-dev] %bg %run -i foo.py In-Reply-To: <20061110085704.GA16897@clipper.ens.fr> References: <20061110085704.GA16897@clipper.ens.fr> Message-ID: <17749.27620.751949.310833@panix3.panix.com> Seems like no takers on this, so I'll comment. Possibly you know this already though. I don't believe anyone knows how to force an "unimport" in a general way, especially when a module contains some sort of dynamically loadable module (such as a DLL or shared object). So yes, a "restart" is sometimes desirable. I realized this a while ago so in the context of a debugger (e.g pydb). So there is a soft restart (called "run") which preserves state like breakpoints and debugger settings and any variables set. And there is a "hard" restart (called "restart") which in fact does an exec using the parameters it saved for the command-line invocation. In the latter, there is no attempt to save debugger state. In theory, one could write out breakpoints and other state to a file and read that back in after restarting. (Maybe one day I'll add two commands to save/restore debugger state command.) But in the context of a python shell such as ipython, what would such a "restart" mean? Pretty simple and straightforward to add would be a "restart" which does the same thing -- just exec itself using whatever command-line parameters were given before. I suppose the saved history would give *some* access to previous commands, even if objects themselves would be wiped clean. Given this, would that be helpful? I don't know. And if not, how would one mark what to keep and what not to keep? Clearly not everything should or could be kept: some objects might be the result of instantiating various classes which did imports. More ambitious would again be to add two commands to save and restore ipython state for some definition of state, e.g. the value of "%pdb" (on/off) or the current working directory. Gael Varoquaux writes: > It seems this doesn't work. I am missing an obvious way to do this. > Elsewhere this would definitely be a nice feature to add. > > Another nice magic to add would be a way to reload a module in a brutal > way. Let me explain. I have all the functions that I use often for > interactive work in a few modules. Every once so ever I change one of > them. Of course as I am a bad boy, I have imported this module doing a > "from foo import *", so I cannot reload it using "reload(foo)". The best > way I have found to reload it is something like > > import foo > %run -i foo.__file__ > # Well, this doesn't work, you have to copy and paste foo.__file__ here, > # as %run cannot take python variables as arguments. > > I don't know whether I am clear on that, and I don't know if I am the > only one who needs such a magic but I for one would love to see this in > ipython. > > Cheers, > > Ga?l > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/ipython-dev > From steve at shrogers.com Sat Nov 11 07:57:15 2006 From: steve at shrogers.com (Steven H. Rogers) Date: Sat, 11 Nov 2006 05:57:15 -0700 Subject: [IPython-dev] IPython1 on PS3/YDL In-Reply-To: <6ce0ac130611102140i71a7beb3q970a38b6798242e4@mail.gmail.com> References: <4555458E.7080607@shrogers.com> <6ce0ac130611102140i71a7beb3q970a38b6798242e4@mail.gmail.com> Message-ID: <4555C8AB.9030303@shrogers.com> I expect IPython to "just work" on the PPE, which would provide a good environment for exploring the Cell's possibilities. I think it would take some work to enable IPython to manage the six available SPE's. The SPE's need to be programmed in C and standard Python to run efficiently if you did a straight port to the SPE. NumPy and PIL are obvious fits for the SPE architecture, but processing non-array data structures takes some thought. Steve /////////////////// Brian Granger wrote: > I think it would be fantastic....I just need to get a PS3.... > > It sounds like it should "just work," which would be really awesome. > Let us know if you get to try this out. > > Brian > > On 11/10/06, Steven H. Rogers wrote: >> The Sony PlayStation 3 running Yellow Dog Linux would seem to be a good >> platform for IPyothon1. Has anyone given any thought to this? >> >> Regards, >> Steve >> >> -- >> Steven H. Rogers, Ph.D., steve at shrogers.com >> Weblog: http://shrogers.com/weblog >> "He who refuses to do arithmetic is doomed to talk nonsense." >> -- John McCarthy >> >> >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://projects.scipy.org/mailman/listinfo/ipython-dev >> > > -- Steven H. Rogers, Ph.D., steve at shrogers.com Weblog: http://shrogers.com/weblog "He who refuses to do arithmetic is doomed to talk nonsense." -- John McCarthy From walter at livinglogic.de Mon Nov 13 09:48:30 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 13 Nov 2006 15:48:30 +0100 Subject: [IPython-dev] wxPython GUI for ipipe Message-ID: <455885BE.8080503@livinglogic.de> Hello all! Unfortunately the Google Summer of Code project didn't work out, so we decided to do our own little Summer of Code. LivingLogic is sponsoring a student to implement a wxPython GUI for ipipe. In the coming weeks and months Nikolas Tautenhahn will be implementing a wxPython based GUI for ipipe that provides the same functionality as ibrowse. The first results of this can be seen here: http://styx.livinglogic.de/~walter/IPython/igrid.py For this to work, wxPython has to be installed and the current IPython head is required. Doing "from igrid import *" sets igrid as the default ipipe display handler. A screenshot can be found here: http://styx.livinglogic.de/~walter/IPython/igrid.gif Servus, Walter From fullung at gmail.com Mon Nov 13 19:23:48 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 14 Nov 2006 02:23:48 +0200 Subject: [IPython-dev] Suggestions for implementing parallel algorithms? In-Reply-To: <6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com> References: <004e01c70451$7c37eb10$0a83a8c0@ratbert> <6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com> Message-ID: <20061114002348.GA18968@dogbert.sdsl.sun.ac.za> Hello all On Thu, 09 Nov 2006, Brian Granger wrote: > This sounds like a nice application of IPython1. Fernando and I have > had a number of talks about this exact issue lately. One things I > should say before all else. In spite of there being years of research > in parallel computing and algorithms, there is basically no research > on *interactive* parallel computing. I am not sure you need your > application to be interactive, but if you do, at some level, you are > in new territory. My application probably doesn't fall into the interactive category at present. But some speaker verification and identification systems could definately be used interactively. > With that said, there are some things to keep in mind. > > First, it is important to realize that all the previous work done on > parallel algorithm/application development still applies even though > the result can be used interactively. For example, if you need to > move data around between the ipython engines, you should still use MPI > - and all the guidelines for using MPI still apply. Okay. This makes sense. > The new and interesting question is really "how and where do you want > to interact with the parallel application as a human user?" There are > many models of how you would want your application abstracted for > interactive usage. And at some level, you may want to have the > interactive API very different from the underlying computational model > and parallel algorithm. You may want to hide the parallelism or you > may find it better to show it explicitely in the API. Interesting. I see what you're saying. For speaker verification systems, you could have an interactive parallel application in the case where you provide a system for a bunch of operators to investigate phone calls with. The operators might be wearing trenchcoats. ;-) > In my own work, I have tended to factor my application into units that > can perform the basic computational tasks for both a serial and > parallel versions of the code. I then use these as building blocks to > build the parallel and serial version. If the low level components > are factored well, the high level algorithm is typically very short > and I don't mind maintaining both a serial and a parallel version. What is still unclear to me is how you call these components? I'm trying to avoid putting too much code in strings that get passed to executeAll and the like. If you have something foo that does N things on each node, how do set things up so that you don't have to write N lines in N strings to N executeAll calls? Presumably one would want to to just write executeAll('foo(a,b,c)') where you wrote foo as a normal Python function. Something like this: In [17]: f = lambda x: x In [18]: rc.pushAll(f=f) In [19]: rc.pushAll('f(10)') However, lambda functions don't pickle: Object cannot be serialized: f Can't pickle : attribute lookup __builtin__.function failed Out[18]: False or something like this: In [36]: rc.getIDs() Out[36]: [0, 1] In [37]: def f(x): return 10*x ....: In [38]: rc.pushAll(f=f) Out[38]: True However, this nukes the engines: exceptions.AttributeError: 'module' object has no attribute 'f' If I distribute a module containing the functions I want to execute to all the nodes beforehand, it might be able to unpickle the function on the engines. Still have to try this. I wonder if it would work with functions declared inside instance methods. Where would the engines find these functions? My thinking here is that you could write parallel algorithms like this: class ParallelFoo: def parallelop(self, data, controller): def something_to_execute_everywhere(x): return 10*x controller.scatterAll('x', data) controller.pushAll(f=something_to_execute_everywhere) controller.executeAll('f(x)') Make sense? Any suggestions for doing this kind of thing? > For many things I do like the scatterAll/executeAll/gatherAll style of > computation - it is extremely lightweight and easy to implement. The > one thing to be careful of though is to not use this approach when MPI > is more appropriate. Testing the scaling of your application with > quickly reveal if there are problems like this. For my application, getting the data to the engines probably needs some thought. Prior to training the world model with K-means and GMM EM I need to spread out tens to hundreds of hours of speech between the engines. Correct me if I'm wrong, but since the client and controller aren't part of the MPI universe, I have to use something like scatterAll here? On my client I would typically have a directory with a few hundred files that I want to distribute to the engines. A quick 'n dirty implementation could read all these files into memory and scatter them to the engines. This approach will run into problems when my dataset becomes bigger than the memory on my client machine (reasonably likely) and is probably going to be reasonably (very?) slow anyway. Next idea: scatter filenames (or some other kind of data ID) and let the engines query a central server for the data, maybe via HTTP or FTP (or maybe do something that exposes a PyTables database to the network). Alternatively, keep all the data on each engine machine. My data is probably too big for this approach. Next idea after that: since I have bunch of machines with disk space to spare, I could spread out multiple copies of my dataset across these machines. Then I could still scatter names and let the engines do some kind of lookup to find machines that have the data it is looking for and have it then get it from one of these at random. Basically the previous idea + load balancing. Next idea after that: in a separate process on each engine machine, run something akin to a BitTorrent client that downloads files from a torrent as the local engine needs them. When this starts up, the client machine could seed the data until there is at least one copy of each file distributed across the engine machines. Next idea after that: figure out a way to prefer scattering of data to kernels that already have the data available. I think the BOINC folks call this concept locality scheduling: http://boinc.berkeley.edu/sched_locality.php Other idea: instead of all this network I/O, keep a subset of the data on each engine machine and use locality scheduling to ensure that only machines that have certain data get work related to that data scattered to them. At this point, scatter probably isn't the right word anymore. How fast this stuff is going to be... we'll see. :-) Do you guys have large datasets to deal with? Any thoughts on doing this kind of thing? > >I'm hoping I can avoid this duplication. My first idea is to make something > >like a LocalController that implements IPython1's IController interface in > >a > >way that makes sense for single node operation. This way, I can implement > >my > >algorithm once in terms of IController operations, test it easily, and > >later > >by simply setting a controller property on a instance of the class > >implementing the algorithm, decide whether it runs on a single node or in > >parallel. > > I had not thought of that before, but it does make sense. It is sort > of similar to building objects that hide whether the object is being > used in a parallel/serial context. It is surely worth trying this > approach, but I am not sure how it would turn out in your case. I'd like to explore this idea further if I can figure out the "right" way for algorithms to tell the engines how to do a piece of work from inside a method. > I don't know if this helps, but I would love to see what you end up > trying and what you find most useful - I am curious about all these > things myself. Thanks for your inputs. Much appreciated. Cheers, Albert From gael.varoquaux at normalesup.org Wed Nov 15 14:17:28 2006 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 15 Nov 2006 20:17:28 +0100 Subject: [IPython-dev] %bg %run -i foo.py In-Reply-To: <20061110085704.GA16897@clipper.ens.fr> References: <20061110085704.GA16897@clipper.ens.fr> Message-ID: <20061115191724.GA21139@clipper.ens.fr> On Fri, Nov 10, 2006 at 09:57:05AM +0100, Gael Varoquaux wrote: > It seems this doesn't work. I am missing an obvious way to do this. > Elsewhere this would definitely be a nice feature to add. OK, revisiting my original mail I see that I wasn't clear: I would like to do something like "%bg %run -i foo.py" but this doesn't work. Is there a way to do this (I may have missed something in the doc), elsewhere this would be a nice feature to add. Cheers, Ga?l From vivainio at gmail.com Thu Nov 16 01:26:45 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Thu, 16 Nov 2006 08:26:45 +0200 Subject: [IPython-dev] %bg %run -i foo.py In-Reply-To: <20061115191724.GA21139@clipper.ens.fr> References: <20061110085704.GA16897@clipper.ens.fr> <20061115191724.GA21139@clipper.ens.fr> Message-ID: <46cb515a0611152226q39a6ebf7lb91e4e1f2a2d522e@mail.gmail.com> On 11/15/06, Gael Varoquaux wrote: > On Fri, Nov 10, 2006 at 09:57:05AM +0100, Gael Varoquaux wrote: > > It seems this doesn't work. I am missing an obvious way to do this. > > Elsewhere this would definitely be a nice feature to add. > > OK, revisiting my original mail I see that I wasn't clear: > > I would like to do something like > > "%bg %run -i foo.py" Try something like %bg _ip.magic('run -i foo.py') In general, the _ip object object (an instance of "IPApi") is often a good place to look for stuff. Try %history and you'll see that all magics are expanded to _ip.magic('blah'). (An unrelated note - I'm moving and w/o home internet connection until the end of this month, so have some patience). -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From gael.varoquaux at normalesup.org Thu Nov 16 02:21:05 2006 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 16 Nov 2006 08:21:05 +0100 Subject: [IPython-dev] %bg %run -i foo.py Message-ID: <20061116072104.GC5448@clipper.ens.fr> On Thu, Nov 16, 2006 at 08:26:45AM +0200, Ville M. Vainio wrote: > %bg _ip.magic('run -i foo.py') Thanks ! I added this to the wiki > (An unrelated note - I'm moving and w/o home internet connection until > the end of this month, so have some patience). Of course, and thank you for your time spent on ipython, ipython rocks my days ! Ga?l From fperez.net at gmail.com Thu Nov 16 02:30:36 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 16 Nov 2006 00:30:36 -0700 Subject: [IPython-dev] Suggestions for implementing parallel algorithms? In-Reply-To: <20061114002348.GA18968@dogbert.sdsl.sun.ac.za> References: <004e01c70451$7c37eb10$0a83a8c0@ratbert> <6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com> <20061114002348.GA18968@dogbert.sdsl.sun.ac.za> Message-ID: Hey Albert, On 11/13/06, Albert Strasheim wrote: > Hello all > > On Thu, 09 Nov 2006, Brian Granger wrote: > > > This sounds like a nice application of IPython1. Fernando and I have > > had a number of talks about this exact issue lately. One things I > > should say before all else. In spite of there being years of research > > in parallel computing and algorithms, there is basically no research > > on *interactive* parallel computing. I am not sure you need your > > application to be interactive, but if you do, at some level, you are > > in new territory. > > My application probably doesn't fall into the interactive category at > present. But some speaker verification and identification systems could > definately be used interactively. Sorry for the silence... I'm currently working specifically on all of this, for a large production code in molecular electronic structure problems. My private copy of ipython1 has some fixes and enhancements to make this easier, and I'm writing a little toy example which I'll commit shortly, and which I'm using to find out for myself what the 'best practices' should be. We're all learning here, so I don't claim these solutions will be better than anything else, but at least they'll come from a production problem, and lots and lots of talks on the phone and face to face with Brian. Feel free to ping me of off-SVN copies of any of this if you want them early. I'm starting to think of using HG for off-SVN development, so we can all share dev trees more fluidly while SVN is used only for commits ready for public consumption... Would you like that? Cheers, f From ellisonbg.net at gmail.com Thu Nov 16 02:44:51 2006 From: ellisonbg.net at gmail.com (Brian Granger) Date: Thu, 16 Nov 2006 00:44:51 -0700 Subject: [IPython-dev] Suggestions for implementing parallel algorithms? In-Reply-To: References: <004e01c70451$7c37eb10$0a83a8c0@ratbert> <6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com> <20061114002348.GA18968@dogbert.sdsl.sun.ac.za> Message-ID: <6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com> Sorry I have been so silent as well. I have a proposal (about IPython) due in the next few days so I have been focusing on that. I will reply early next week once things settle down. I definitely have some ideas (and practical experience) in dealing with large datasets. But Fernando, do you think that Hg and subversion would play well together? I don't know much about Hg. Brian On 11/16/06, Fernando Perez wrote: > Hey Albert, > > On 11/13/06, Albert Strasheim wrote: > > Hello all > > > > On Thu, 09 Nov 2006, Brian Granger wrote: > > > > > This sounds like a nice application of IPython1. Fernando and I have > > > had a number of talks about this exact issue lately. One things I > > > should say before all else. In spite of there being years of research > > > in parallel computing and algorithms, there is basically no research > > > on *interactive* parallel computing. I am not sure you need your > > > application to be interactive, but if you do, at some level, you are > > > in new territory. > > > > My application probably doesn't fall into the interactive category at > > present. But some speaker verification and identification systems could > > definately be used interactively. > > Sorry for the silence... I'm currently working specifically on all of > this, for a large production code in molecular electronic structure > problems. My private copy of ipython1 has some fixes and enhancements > to make this easier, and I'm writing a little toy example which I'll > commit shortly, and which I'm using to find out for myself what the > 'best practices' should be. We're all learning here, so I don't claim > these solutions will be better than anything else, but at least > they'll come from a production problem, and lots and lots of talks on > the phone and face to face with Brian. > > Feel free to ping me of off-SVN copies of any of this if you want them early. > > I'm starting to think of using HG for off-SVN development, so we can > all share dev trees more fluidly while SVN is used only for commits > ready for public consumption... > Would you like that? > > Cheers, > > f > From fperez.net at gmail.com Thu Nov 16 03:00:50 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 16 Nov 2006 01:00:50 -0700 Subject: [IPython-dev] Suggestions for implementing parallel algorithms? In-Reply-To: <6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com> References: <004e01c70451$7c37eb10$0a83a8c0@ratbert> <6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com> <20061114002348.GA18968@dogbert.sdsl.sun.ac.za> <6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com> Message-ID: On 11/16/06, Brian Granger wrote: > Sorry I have been so silent as well. I have a proposal (about > IPython) due in the next > few days so I have been focusing on that. I will reply early next > week once things settle down. > > I definitely have some ideas (and practical experience) in dealing > with large datasets. > > But Fernando, do you think that Hg and subversion would play well > together? I don't know much about Hg. Well, I've lightly used bzr on top of an SVN tree without too many problems. Though in that case, I've always used the bzr tree for fast-paced work and made the SVN commits as separate operations, not by trying to generate a commit from a bzr changeset. Recently (both SAGE and Brett Cannon's tests) have me leaning more towards hg than bzr, but it's probably a bit of a toss-up between those two. The main point is just to give each of us an easy way to keep local version control we can share for pulling changes, while SVN remains the 'official' tree for commits meant for 'final' distribution. Just an idea... Cheers f From stefan at sun.ac.za Thu Nov 16 03:50:29 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 16 Nov 2006 10:50:29 +0200 Subject: [IPython-dev] Suggestions for implementing parallel algorithms? In-Reply-To: References: <004e01c70451$7c37eb10$0a83a8c0@ratbert> <6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com> <20061114002348.GA18968@dogbert.sdsl.sun.ac.za> <6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com> Message-ID: <20061116085028.GF1975@mentat.za.net> On Thu, Nov 16, 2006 at 01:00:50AM -0700, Fernando Perez wrote: > On 11/16/06, Brian Granger wrote: > > Sorry I have been so silent as well. I have a proposal (about > > IPython) due in the next > > few days so I have been focusing on that. I will reply early next > > week once things settle down. > > > > I definitely have some ideas (and practical experience) in dealing > > with large datasets. > > > > But Fernando, do you think that Hg and subversion would play well > > together? I don't know much about Hg. > > Well, I've lightly used bzr on top of an SVN tree without too many > problems. Though in that case, I've always used the bzr tree for > fast-paced work and made the SVN commits as separate operations, not > by trying to generate a commit from a bzr changeset. > > Recently (both SAGE and Brett Cannon's tests) have me leaning more > towards hg than bzr, but it's probably a bit of a toss-up between > those two. Interesting, I'm interested to know why. I switched from hg to bzr a year or so ago, having had problems with things like cherry picking. I believe Lele's 'tailor' can be useful for the problem mentioned above: http://www.darcs.net/DarcsWiki/Tailor Regards St?fan From prabhu at aero.iitb.ac.in Thu Nov 16 04:57:02 2006 From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran) Date: Thu, 16 Nov 2006 15:27:02 +0530 Subject: [IPython-dev] Suggestions for implementing parallel algorithms? In-Reply-To: <6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com> References: <004e01c70451$7c37eb10$0a83a8c0@ratbert> <6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com> <20061114002348.GA18968@dogbert.sdsl.sun.ac.za> <6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com> Message-ID: <17756.13806.598304.997831@prpc.aero.iitb.ac.in> >>>>> "Brian" == Brian Granger writes: Brian> But Fernando, do you think that Hg and subversion would Brian> play well together? I don't know much about Hg. What about bzr? cheers, prabhu From vivainio at gmail.com Thu Nov 16 05:58:46 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Thu, 16 Nov 2006 12:58:46 +0200 Subject: [IPython-dev] Suggestions for implementing parallel algorithms? In-Reply-To: <17756.13806.598304.997831@prpc.aero.iitb.ac.in> References: <004e01c70451$7c37eb10$0a83a8c0@ratbert> <6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com> <20061114002348.GA18968@dogbert.sdsl.sun.ac.za> <6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com> <17756.13806.598304.997831@prpc.aero.iitb.ac.in> Message-ID: <46cb515a0611160258u5a9c477eq5779345a80599c2c@mail.gmail.com> On 11/16/06, Prabhu Ramachandran wrote: > >>>>> "Brian" == Brian Granger writes: > > Brian> But Fernando, do you think that Hg and subversion would > Brian> play well together? I don't know much about Hg. Hg and SVN should play as well together as local directory and SVN, in any case. :-) > What about bzr? At this moment, googling around would indicate that Hg is technically better than bzr. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From prabhu at aero.iitb.ac.in Thu Nov 16 07:59:39 2006 From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran) Date: Thu, 16 Nov 2006 18:29:39 +0530 Subject: [IPython-dev] Suggestions for implementing parallel algorithms? In-Reply-To: <46cb515a0611160258u5a9c477eq5779345a80599c2c@mail.gmail.com> References: <004e01c70451$7c37eb10$0a83a8c0@ratbert> <6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com> <20061114002348.GA18968@dogbert.sdsl.sun.ac.za> <6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com> <17756.13806.598304.997831@prpc.aero.iitb.ac.in> <46cb515a0611160258u5a9c477eq5779345a80599c2c@mail.gmail.com> Message-ID: <17756.24763.991616.543830@prpc.aero.iitb.ac.in> >>>>> "Ville" == Ville M Vainio writes: Ville> On 11/16/06, Prabhu Ramachandran Ville> wrote: >> >>>>> "Brian" == Brian Granger >> writes: >> Brian> But Fernando, do you think that Hg and subversion would Brian> play well together? I don't know much about Hg. Ville> Hg and SVN should play as well together as local directory Ville> and SVN, in any case. :-) >> What about bzr? Ville> At this moment, googling around would indicate that Hg is Ville> technically better than bzr. bzr is slow but that isn't an issue for smaller projects. IIRC Hg OTOH can't do directory renames yet which bzr can. That seems the difference to me. I believe the bzr folks are working on speed and that the Hg folks on the rename. Personally I can put up with a slower speed for a small tree but renames are really important which is why I suggested bzr... cheers, prabhu From stefan at sun.ac.za Thu Nov 16 08:17:50 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 16 Nov 2006 15:17:50 +0200 Subject: [IPython-dev] Suggestions for implementing parallel algorithms? In-Reply-To: <46cb515a0611160258u5a9c477eq5779345a80599c2c@mail.gmail.com> References: <004e01c70451$7c37eb10$0a83a8c0@ratbert> <6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com> <20061114002348.GA18968@dogbert.sdsl.sun.ac.za> <6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com> <17756.13806.598304.997831@prpc.aero.iitb.ac.in> <46cb515a0611160258u5a9c477eq5779345a80599c2c@mail.gmail.com> Message-ID: <20061116131750.GH1975@mentat.za.net> On Thu, Nov 16, 2006 at 12:58:46PM +0200, Ville M. Vainio wrote: > On 11/16/06, Prabhu Ramachandran wrote: > > > >>>>> "Brian" == Brian Granger writes: > > > > Brian> But Fernando, do you think that Hg and subversion would > > Brian> play well together? I don't know much about Hg. > > Hg and SVN should play as well together as local directory and SVN, in > any case. :-) > > > > What about bzr? > > At this moment, googling around would indicate that Hg is technically > better than bzr. Can you provide a URL? I found several reviews (based on older versions) which all claim that the two are basically equal, except when it comes to speed. In that regard bzr has been vastly improved with 1.0. The comparisons are further muddled by some referring to bzr as in arch/tla. As for my previous question about mercurial and cherry-picking, it seems they have started development on a plugin in August: http://www.selenic.com/mercurial/wiki/index.cgi/TransplantExtension http://hg.kublai.com/mercurial/transplant Cheers St?fan From fperez.net at gmail.com Thu Nov 16 11:31:18 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 16 Nov 2006 09:31:18 -0700 Subject: [IPython-dev] Suggestions for implementing parallel algorithms? In-Reply-To: <20061116131750.GH1975@mentat.za.net> References: <004e01c70451$7c37eb10$0a83a8c0@ratbert> <6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com> <20061114002348.GA18968@dogbert.sdsl.sun.ac.za> <6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com> <17756.13806.598304.997831@prpc.aero.iitb.ac.in> <46cb515a0611160258u5a9c477eq5779345a80599c2c@mail.gmail.com> <20061116131750.GH1975@mentat.za.net> Message-ID: On 11/16/06, Stefan van der Walt wrote: > On Thu, Nov 16, 2006 at 12:58:46PM +0200, Ville M. Vainio wrote: > > On 11/16/06, Prabhu Ramachandran wrote: > > > > > >>>>> "Brian" == Brian Granger writes: > > > > > > Brian> But Fernando, do you think that Hg and subversion would > > > Brian> play well together? I don't know much about Hg. > > > > Hg and SVN should play as well together as local directory and SVN, in > > any case. :-) > > > > > > > What about bzr? > > > > At this moment, googling around would indicate that Hg is technically > > better than bzr. > > Can you provide a URL? I found several reviews (based on older > versions) which all claim that the two are basically equal, except > when it comes to speed. In that regard bzr has been vastly improved > with 1.0. The comparisons are further muddled by some referring to > bzr as in arch/tla. Here's something I recently read: http://sayspy.blogspot.com/2006/11/bazaar-vs-mercurial-unscientific.html This was one of my sources for both saying that they were probably very close, and that hg seemed marginally preferable. Another reason for my slight bias towards hg is that SAGE chose it (for speed reasons, mostly), and I occasionally need to interact with SAGE's dev code. I've heard about Tailor before, but haven't actually ever used it. If your commits just involve file updates, there's no real need for it and I prefer to make up a new log message that summarizes the work. But if you've done structural operations in your hg/bzr work, then you /really/ want something like Tailor to propagate that correctly into the SVN operations. I have no idea how well that actually works in practice, I can easily imagine it being 'below its advertised performance' :) Cheers, f From fullung at gmail.com Thu Nov 16 16:22:41 2006 From: fullung at gmail.com (Albert Strasheim) Date: Thu, 16 Nov 2006 23:22:41 +0200 Subject: [IPython-dev] Suggestions for implementing parallel algorithms? In-Reply-To: References: <004e01c70451$7c37eb10$0a83a8c0@ratbert> <6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com> <20061114002348.GA18968@dogbert.sdsl.sun.ac.za> Message-ID: <20061116212241.GA21364@dogbert.sdsl.sun.ac.za> Hello all On Thu, 16 Nov 2006, Fernando Perez wrote: > On 11/13/06, Albert Strasheim wrote: > >On Thu, 09 Nov 2006, Brian Granger wrote: > >> This sounds like a nice application of IPython1. Fernando and I have > >> had a number of talks about this exact issue lately. One things I > >> should say before all else. In spite of there being years of research > >> in parallel computing and algorithms, there is basically no research > >> on *interactive* parallel computing. I am not sure you need your > >> application to be interactive, but if you do, at some level, you are > >> in new territory. > > > >My application probably doesn't fall into the interactive category at > >present. But some speaker verification and identification systems could > >definately be used interactively. > > Sorry for the silence... I'm currently working specifically on all of > this, for a large production code in molecular electronic structure > problems. My private copy of ipython1 has some fixes and enhancements > to make this easier, and I'm writing a little toy example which I'll > commit shortly, and which I'm using to find out for myself what the > 'best practices' should be. We're all learning here, so I don't claim > these solutions will be better than anything else, but at least > they'll come from a production problem, and lots and lots of talks on > the phone and face to face with Brian. > > Feel free to ping me of off-SVN copies of any of this if you want them > early. I would love to look at any code you'd be willing to offer. I too am trying to figure out the best practices. Thanks in advance! > I'm starting to think of using HG for off-SVN development, so we can > all share dev trees more fluidly while SVN is used only for commits > ready for public consumption... > Would you like that? Sounds like a great plan. Cheers, Albert From smelkov at mph1.phys.spbu.ru Sun Nov 19 12:32:16 2006 From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov) Date: Sun, 19 Nov 2006 20:32:16 +0300 Subject: [IPython-dev] [patch 0/9] IPython patches from kirr Message-ID: <20061119173216.561631860@roro-510.zxlink> Hello up there! While refreshing IPython recently, I've made some patches against ipython--trunk. Please apply. -- ????? ????????, ??????. http://landau.phys.spbu.ru/~kirr/aiv/ From smelkov at mph1.phys.spbu.ru Sun Nov 19 12:32:19 2006 From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov) Date: Sun, 19 Nov 2006 20:32:19 +0300 Subject: [IPython-dev] [patch 3/9] Kill old bits from %prun doc. References: <20061119173216.561631860@roro-510.zxlink> Message-ID: <20061119173434.055551881@roro-510.zxlink> An embedded and charset-unspecified text was scrubbed... Name: prun_kill-old-doc.patch URL: From smelkov at mph1.phys.spbu.ru Sun Nov 19 12:32:20 2006 From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov) Date: Sun, 19 Nov 2006 20:32:20 +0300 Subject: [IPython-dev] [patch 4/9] The manual mentions %ds along %pushd and %popd, but %ds is not a valid magic. Maybe it should be %dhist? References: <20061119173216.561631860@roro-510.zxlink> Message-ID: <20061119173434.540630873@roro-510.zxlink> An embedded and charset-unspecified text was scrubbed... Name: doc-ds-2-dhist.patch URL: From smelkov at mph1.phys.spbu.ru Sun Nov 19 12:32:18 2006 From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov) Date: Sun, 19 Nov 2006 20:32:18 +0300 Subject: [IPython-dev] [patch 2/9] Try to use cProfile module (Python-2.5) instead of old profile. References: <20061119173216.561631860@roro-510.zxlink> Message-ID: <20061119173433.579561522@roro-510.zxlink> An embedded and charset-unspecified text was scrubbed... Name: cprofile.patch URL: From smelkov at mph1.phys.spbu.ru Sun Nov 19 12:32:21 2006 From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov) Date: Sun, 19 Nov 2006 20:32:21 +0300 Subject: [IPython-dev] [patch 5/9] ScientificPython webpage has moved to:: http://dirac.cnrs-orleans.fr/ScientificPython/ References: <20061119173216.561631860@roro-510.zxlink> Message-ID: <20061119173435.031683281@roro-510.zxlink> An embedded and charset-unspecified text was scrubbed... Name: doc-ScientificPython-updates.patch URL: From smelkov at mph1.phys.spbu.ru Sun Nov 19 12:32:22 2006 From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov) Date: Sun, 19 Nov 2006 20:32:22 +0300 Subject: [IPython-dev] [patch 6/9] Fix %config crash when invoked without args. References: <20061119173216.561631860@roro-510.zxlink> Message-ID: <20061119173435.516700533@roro-510.zxlink> An embedded and charset-unspecified text was scrubbed... Name: fix-config.patch URL: From smelkov at mph1.phys.spbu.ru Sun Nov 19 12:32:17 2006 From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov) Date: Sun, 19 Nov 2006 20:32:17 +0300 Subject: [IPython-dev] [patch 1/9] Fix typos here and there... References: <20061119173216.561631860@roro-510.zxlink> Message-ID: <20061119173433.114457240@roro-510.zxlink> An embedded and charset-unspecified text was scrubbed... Name: typos.patch URL: From smelkov at mph1.phys.spbu.ru Sun Nov 19 12:32:25 2006 From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov) Date: Sun, 19 Nov 2006 20:32:25 +0300 Subject: [IPython-dev] [patch 9/9] doc: point bug-tracker URL to IPythons trac-tickets. References: <20061119173216.561631860@roro-510.zxlink> Message-ID: <20061119173436.959526650@roro-510.zxlink> An embedded and charset-unspecified text was scrubbed... Name: doc-tickets.patch URL: From smelkov at mph1.phys.spbu.ru Sun Nov 19 12:32:23 2006 From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov) Date: Sun, 19 Nov 2006 20:32:23 +0300 Subject: [IPython-dev] [patch 7/9] Implement %page -r (page in raw mode) References: <20061119173216.561631860@roro-510.zxlink> Message-ID: <20061119173436.007608230@roro-510.zxlink> An embedded and charset-unspecified text was scrubbed... Name: page-raw.patch URL: From smelkov at mph1.phys.spbu.ru Sun Nov 19 12:32:24 2006 From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov) Date: Sun, 19 Nov 2006 20:32:24 +0300 Subject: [IPython-dev] [patch 8/9] Fix wrong Qt+IPython+Designer link in documentation. References: <20061119173216.561631860@roro-510.zxlink> Message-ID: <20061119173436.483781295@roro-510.zxlink> An embedded and charset-unspecified text was scrubbed... Name: doc-qt+ipython.patch URL: From vivainio at gmail.com Sun Nov 19 15:12:29 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Sun, 19 Nov 2006 22:12:29 +0200 Subject: [IPython-dev] [patch 0/9] IPython patches from kirr In-Reply-To: <20061119173216.561631860@roro-510.zxlink> References: <20061119173216.561631860@roro-510.zxlink> Message-ID: <46cb515a0611191212q17b55ddfs1a7a169463990dc9@mail.gmail.com> On 11/19/06, Kirill Smelkov wrote: > While refreshing IPython recently, I've made some patches against > ipython--trunk. Please apply. Will do (on tuesday). Many thanks! -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From fperez.net at gmail.com Mon Nov 20 01:53:17 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 19 Nov 2006 23:53:17 -0700 Subject: [IPython-dev] [patch 0/9] IPython patches from kirr In-Reply-To: <20061119173216.561631860@roro-510.zxlink> References: <20061119173216.561631860@roro-510.zxlink> Message-ID: On 11/19/06, Kirill Smelkov wrote: > Hello up there! > > While refreshing IPython recently, I've made some patches against > ipython--trunk. Please apply. Thanks! We'll have a look (Ville already replied). Much appreciated. Ville - let me know if there's any one you may have a question with. Upon a quick look, I noticed that in patch 2/9 would be cleaner by simply having two separate try/excepts to deal with the profile-not-in-debian and the cProfile issue separately. It's cleaner than retrying the failed import, and the intent is more obvious. Also, patch 4/9: the right magic is '%dirs' (for "directory stack"), not %dhist. Cheers, f From vivainio at gmail.com Tue Nov 21 01:37:47 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Tue, 21 Nov 2006 08:37:47 +0200 Subject: [IPython-dev] [patch 0/9] IPython patches from kirr In-Reply-To: References: <20061119173216.561631860@roro-510.zxlink> Message-ID: <46cb515a0611202237q4d2d478al78cef9f578f06db2@mail.gmail.com> On 11/20/06, Fernando Perez wrote: > Thanks! We'll have a look (Ville already replied). Much appreciated. > > Ville - let me know if there's any one you may have a question with. The %config one. I still think %config is redundant and should be removed, _ip.options already works, is stable (as a part of ipapi) and is handier anyway (tab completion works etc.). -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From vivainio at gmail.com Tue Nov 21 15:44:02 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Tue, 21 Nov 2006 22:44:02 +0200 Subject: [IPython-dev] [patch 0/9] IPython patches from kirr In-Reply-To: <20061121190806.GA5431@landau.phys.spbu.ru> References: <20061119173216.561631860@roro-510.zxlink> <20061121190806.GA5431@landau.phys.spbu.ru> Message-ID: <46cb515a0611211244n2d14aa7ctfc42cc34f0c49679@mail.gmail.com> On 11/21/06, Kirill Smelkov wrote: > Thanks for your acknowledgement! It's great that some people pay attention to things such as obsolete documentation, they are all too easily left dangling... > This one is ok? No, your patch can't be applied for some reason. Please send a new one against the current svn, which has the other patches (apart from the %config one, because I'd like to remove %config altogether) applied. And what happened to patch 8/9? -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From smelkov at mph1.phys.spbu.ru Wed Nov 22 16:34:00 2006 From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov) Date: Thu, 23 Nov 2006 00:34:00 +0300 Subject: [IPython-dev] [patch 1/2] Try to use cProfile module (Python-2.5) instead of old profile. References: <20061122213359.515229954@roro-510.zxlink> Message-ID: <20061122213502.540288964@roro-510.zxlink> An embedded and charset-unspecified text was scrubbed... Name: cprofile.patch URL: From smelkov at mph1.phys.spbu.ru Wed Nov 22 16:34:01 2006 From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov) Date: Thu, 23 Nov 2006 00:34:01 +0300 Subject: [IPython-dev] [patch 2/2] Fix wrong Qt+IPython+Designer link in documentation. References: <20061122213359.515229954@roro-510.zxlink> Message-ID: <20061122213503.012163136@roro-510.zxlink> An embedded and charset-unspecified text was scrubbed... Name: doc-qt+ipython.patch URL: From smelkov at mph1.phys.spbu.ru Wed Nov 22 16:33:59 2006 From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov) Date: Thu, 23 Nov 2006 00:33:59 +0300 Subject: [IPython-dev] [patch 0/2] IPython patches from kirr, iter2 Message-ID: <20061122213359.515229954@roro-510.zxlink> I'm resending 2 patches out of 3 left unapplied from the first iteration. -- All the best, Kirill. http://landau.phys.spbu.ru/~kirr/aiv/ From rocky at panix.com Wed Nov 22 23:04:52 2006 From: rocky at panix.com (R. Bernstein) Date: Wed, 22 Nov 2006 23:04:52 -0500 Subject: [IPython-dev] making %pydb be more pythonic (was: Extensions/pydb_ipy.py added) In-Reply-To: <17731.31618.644784.714712@panix3.panix.com> References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com> <17731.8786.756329.573337@panix3.panix.com> <46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com> <17731.15518.833930.352892@panix3.panix.com> <46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com> <17731.31618.644784.714712@panix3.panix.com> Message-ID: <17765.7652.52085.132851@panix3.panix.com> About a month back, there was this discussion: R. Bernstein writes: > Ville M. Vainio writes: > > On 10/28/06, R. Bernstein wrote: > > > > > But one other thing regarding call_pydb(). And I'm not really sure how > > > to fix. So if anyone has suggestions... > > > > > > pydb.runv() should be using the ipython Pdb class which has pdef, pdoc > > > commands added and not the pydb one. > > > > Would it make sense to import pydb.pydb module in pydb_ipy.py, and > > inject the ipython version of Pdb class to the pydb module namespace? > > That way main() would instantiate the ipython version... > > Good idea! That's be great! Nothing has happened on this and I suspect because it wasn't doable. (At least I couldn't figure out how it would be done. But I figured if it could that *could* it would be a great thing to do.) So recently, I've added a parameter to pydb so that runv() can get passed with an Pdb object extended in the way that ipython wants to extend it. That will be in the next release of pydb (which again can be timed so that it is no later than the next ipython release). I've modified ipy_pydb.py so that it now passes in such a Pdb object. This was the debug session can make use of pdoc and pinfo. It also uses the ipython debugger prompt (which personally I'm not a fan of but I guess it makes it ipython consistent.) There is now the possibility of using the ipython color scheme, but I couldn't figure out how to get that passed when creating the Debugger.Pdb object. And in going over IPython/Extensions/ipy_pydb.py I've noticed a couple of other things. First there should be a check that pydb can be imported and that there is a late enough version of pydb. Below is a patch. -------------- next part -------------- A non-text attachment was scrubbed... Name: ipy_pydb.py.diff Type: application/octet-stream Size: 1185 bytes Desc: Changes to have %pydb be more ipythonic. URL: From vivainio at gmail.com Thu Nov 23 14:06:01 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Thu, 23 Nov 2006 21:06:01 +0200 Subject: [IPython-dev] [patch 1/2] Try to use cProfile module (Python-2.5) instead of old profile. In-Reply-To: <20061122213502.540288964@roro-510.zxlink> References: <20061122213359.515229954@roro-510.zxlink> <20061122213502.540288964@roro-510.zxlink> Message-ID: <46cb515a0611231106n20ed0cf2ld8d75acd6591f5e4@mail.gmail.com> On 11/22/06, Kirill Smelkov wrote: > rationale: cProfile is faster. > try: > import profile,pstats > except ImportError: > - profile = pstats = None > + pass > + > +# cProfile was added in Python2.5 > +# if present, we use it instead of old profile, because cProfile is faster > +try: > + import cProfile as profile > + import pstats > +except ImportError: > + pass New you import both profile and cProfile, if both exist. That's a bit wasteful; why not import cProfile first, and only if that raises ImportError import profile? -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From vivainio at gmail.com Thu Nov 23 14:48:05 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Thu, 23 Nov 2006 21:48:05 +0200 Subject: [IPython-dev] making %pydb be more pythonic (was: Extensions/pydb_ipy.py added) In-Reply-To: <17765.7652.52085.132851@panix3.panix.com> References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com> <17731.8786.756329.573337@panix3.panix.com> <46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com> <17731.15518.833930.352892@panix3.panix.com> <46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com> <17731.31618.644784.714712@panix3.panix.com> <17765.7652.52085.132851@panix3.panix.com> Message-ID: <46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com> On 11/23/06, R. Bernstein wrote: > > > Would it make sense to import pydb.pydb module in pydb_ipy.py, and > > > inject the ipython version of Pdb class to the pydb module namespace? > > > That way main() would instantiate the ipython version... > > > > Good idea! That's be great! > > Nothing has happened on this and I suspect because it wasn't > doable. (At least I couldn't figure out how it would be done. But I > figured if it could that *could* it would be a great thing to do.) This could have worked (in ipy_pydb - untried code): ----- import pydb.pydb import Debugger pydb.pydb.Pdb = Debugger.Pdb ------ I didn't try it myself because I figured you could have given it more testing mileage. Additionally, I was off the net for a long while due to moving. > So recently, I've added a parameter to pydb so that runv() can get > passed with an Pdb object extended in the way that ipython wants to > extend it. That will be in the next release of pydb (which again can > be timed so that it is no later than the next ipython release). > > I've modified ipy_pydb.py so that it now passes in such a Pdb > object. This was the debug session can make use of pdoc and pinfo. It > also uses the ipython debugger prompt (which personally I'm not a fan > of but I guess it makes it ipython consistent.) There is now the > possibility of using the ipython color scheme, but I couldn't figure > out how to get that passed when creating the Debugger.Pdb object. > > And in going over IPython/Extensions/ipy_pydb.py I've noticed a couple > of other things. First there should be a check that pydb can be > imported and that there is a late enough version of pydb. > > Below is a patch. Not a bad solution, I committed the patch. In general, ipy_pydb.py is "your territory" and I'll gladly commit every patch I get for it immediately. In theory something like ipy_pydb.py could even be bundled w/ pydb, just like python-mode.el is bundled with python... -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From smelkov at mph1.phys.spbu.ru Thu Nov 23 16:44:05 2006 From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov) Date: Fri, 24 Nov 2006 00:44:05 +0300 Subject: [IPython-dev] [patch 1/2] Try to use cProfile module (Python-2.5) instead of old profile. In-Reply-To: <46cb515a0611231106n20ed0cf2ld8d75acd6591f5e4@mail.gmail.com> References: <20061122213359.515229954@roro-510.zxlink> <20061122213502.540288964@roro-510.zxlink> <46cb515a0611231106n20ed0cf2ld8d75acd6591f5e4@mail.gmail.com> Message-ID: <20061123214404.GA4614@landau.phys.spbu.ru> On Thu, Nov 23, 2006 at 09:06:01PM +0200, Ville M. Vainio wrote: > On 11/22/06, Kirill Smelkov wrote: > > >rationale: cProfile is faster. > > try: > > import profile,pstats > > except ImportError: > >- profile = pstats = None > >+ pass > >+ > >+# cProfile was added in Python2.5 > >+# if present, we use it instead of old profile, because cProfile is faster > >+try: > >+ import cProfile as profile > >+ import pstats > >+except ImportError: > >+ pass > > New you import both profile and cProfile, if both exist. That's a bit > wasteful; why not import cProfile first, and only if that raises > ImportError import profile? That's what I originally did (see [patch 2/9], buf Fernando objected, I can't understand why... -- ????? ????????, ??????. http://landau.phys.spbu.ru/~kirr/aiv/ From vivainio at gmail.com Thu Nov 23 16:52:42 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Thu, 23 Nov 2006 23:52:42 +0200 Subject: [IPython-dev] [patch 1/2] Try to use cProfile module (Python-2.5) instead of old profile. In-Reply-To: <20061123214404.GA4614@landau.phys.spbu.ru> References: <20061122213359.515229954@roro-510.zxlink> <20061122213502.540288964@roro-510.zxlink> <46cb515a0611231106n20ed0cf2ld8d75acd6591f5e4@mail.gmail.com> <20061123214404.GA4614@landau.phys.spbu.ru> Message-ID: <46cb515a0611231352s2c58d47ej5699ad70b448aaf9@mail.gmail.com> On 11/23/06, Kirill Smelkov wrote: > That's what I originally did (see [patch 2/9], buf Fernando objected, I > can't understand why... Oh, I see, he probably just accidentally misread your patch. I'll go ahead and apply your original patch then (tomorrow). -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From rocky at panix.com Fri Nov 24 00:39:20 2006 From: rocky at panix.com (R. Bernstein) Date: Fri, 24 Nov 2006 00:39:20 -0500 Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was: Extensions/pydb_ipy.py added) In-Reply-To: <46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com> References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com> <17731.8786.756329.573337@panix3.panix.com> <46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com> <17731.15518.833930.352892@panix3.panix.com> <46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com> <17731.31618.644784.714712@panix3.panix.com> <17765.7652.52085.132851@panix3.panix.com> <46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com> Message-ID: <17766.34184.188910.580317@panix3.panix.com> Thanks for applying the patch! I am currently out of town, but when I get back I'll try as you suggest below. I have other patches sitting on a disk somewhere too. I don't have a problem adding ipy_pydb.py in the pydb package. But the one thing though that I think would somehow be nice is that if one *didn't* have to do run "import ipy_pydb" to allow %pydb to work. Given that one can put this in a "try" block, I don't see any downside of doing this. A cool and more general mechanism for ipython extensions would be something along the lines of Emacs Lisp "autoload". Say if I run %xxx ... then if that's not registered, try to run "import ipy_xxx" (where for example xxx = pydb), and then run %xxx .... Comments? Ville M. Vainio writes: > On 11/23/06, R. Bernstein wrote: > > > > > Would it make sense to import pydb.pydb module in pydb_ipy.py, and > > > > inject the ipython version of Pdb class to the pydb module namespace? > > > > That way main() would instantiate the ipython version... > > > > > > Good idea! That's be great! > > > > Nothing has happened on this and I suspect because it wasn't > > doable. (At least I couldn't figure out how it would be done. But I > > figured if it could that *could* it would be a great thing to do.) > > This could have worked (in ipy_pydb - untried code): > > ----- > import pydb.pydb > import Debugger > > pydb.pydb.Pdb = Debugger.Pdb > ------ > > I didn't try it myself because I figured you could have given it more > testing mileage. Additionally, I was off the net for a long while due > to moving. > > > So recently, I've added a parameter to pydb so that runv() can get > > passed with an Pdb object extended in the way that ipython wants to > > extend it. That will be in the next release of pydb (which again can > > be timed so that it is no later than the next ipython release). > > > > I've modified ipy_pydb.py so that it now passes in such a Pdb > > object. This was the debug session can make use of pdoc and pinfo. It > > also uses the ipython debugger prompt (which personally I'm not a fan > > of but I guess it makes it ipython consistent.) There is now the > > possibility of using the ipython color scheme, but I couldn't figure > > out how to get that passed when creating the Debugger.Pdb object. > > > > And in going over IPython/Extensions/ipy_pydb.py I've noticed a couple > > of other things. First there should be a check that pydb can be > > imported and that there is a late enough version of pydb. > > > > Below is a patch. > > > Not a bad solution, I committed the patch. In general, ipy_pydb.py is > "your territory" and I'll gladly commit every patch I get for it > immediately. In theory something like ipy_pydb.py could even be > bundled w/ pydb, just like python-mode.el is bundled with python... > > -- > Ville M. Vainio - vivainio.googlepages.com > blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' > From Fernando.Perez at colorado.edu Fri Nov 24 02:59:40 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Fri, 24 Nov 2006 00:59:40 -0700 Subject: [IPython-dev] [patch 1/2] Try to use cProfile module (Python-2.5) instead of old profile. In-Reply-To: <46cb515a0611231352s2c58d47ej5699ad70b448aaf9@mail.gmail.com> References: <20061122213359.515229954@roro-510.zxlink> <20061122213502.540288964@roro-510.zxlink> <46cb515a0611231106n20ed0cf2ld8d75acd6591f5e4@mail.gmail.com> <20061123214404.GA4614@landau.phys.spbu.ru> <46cb515a0611231352s2c58d47ej5699ad70b448aaf9@mail.gmail.com> Message-ID: <4566A66C.9020008@colorado.edu> Ville M. Vainio wrote: > On 11/23/06, Kirill Smelkov wrote: > >> That's what I originally did (see [patch 2/9], buf Fernando objected, I >> can't understand why... > > Oh, I see, he probably just accidentally misread your patch. I'll go > ahead and apply your original patch then (tomorrow). Yes, I did misread the original patch. I looked at it again just now, and it's fine. Thanks again Kirill for these contributions. Cheers, f From vivainio at gmail.com Fri Nov 24 11:27:57 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Fri, 24 Nov 2006 18:27:57 +0200 Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was: Extensions/pydb_ipy.py added) In-Reply-To: <17766.34184.188910.580317@panix3.panix.com> References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com> <17731.8786.756329.573337@panix3.panix.com> <46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com> <17731.15518.833930.352892@panix3.panix.com> <46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com> <17731.31618.644784.714712@panix3.panix.com> <17765.7652.52085.132851@panix3.panix.com> <46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com> <17766.34184.188910.580317@panix3.panix.com> Message-ID: <46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com> On 11/24/06, R. Bernstein wrote: > I don't have a problem adding ipy_pydb.py in the pydb package. But the one thing though that I think would somehow be nice is that if one *didn't* have to do run "import ipy_pydb" to allow %pydb to work. Given that one can put this in a "try" block, I don't see any downside of doing this. It's just that having to import ipy_pydb would make enabling pydb a configuration option. Sometimes one may wish to use standard pdb even when pydb is installed. And the import need not be manual - it's quite easy to add "import ipy_pydb" to your personal ipy_user_conf.py. > A cool and more general mechanism for ipython extensions would be something along the lines of Emacs Lisp "autoload". Say if I run %xxx ... then if that's not registered, try to run "import ipy_xxx" (where for example xxx = pydb), and then run %xxx .... Comments? Well, this might save us some startup time & memory footprint. It should be quite easy to do, esp. for magics (and completers). We can easily inject stub callables in place of magics, which 1) import a specified module and 2) try to run the magic again. I think a good first line of action would be to add an ipapi function that permanently registers a magic name / completer trigger (or whatever) with ipython. Here's what one could do e.g. in setup.py for pydb: import IPython.ipapi ip = IPython.ipapi.get() ip.register_autoload("%pydb", "pydb.ipy_pydb") -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From smelkov at mph1.phys.spbu.ru Fri Nov 24 16:29:19 2006 From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov) Date: Sat, 25 Nov 2006 00:29:19 +0300 Subject: [IPython-dev] [patch 1/2] Try to use cProfile module (Python-2.5) instead of old profile. In-Reply-To: <4566A66C.9020008@colorado.edu> References: <20061122213359.515229954@roro-510.zxlink> <20061122213502.540288964@roro-510.zxlink> <46cb515a0611231106n20ed0cf2ld8d75acd6591f5e4@mail.gmail.com> <20061123214404.GA4614@landau.phys.spbu.ru> <46cb515a0611231352s2c58d47ej5699ad70b448aaf9@mail.gmail.com> <4566A66C.9020008@colorado.edu> Message-ID: <20061124212919.GA4802@landau.phys.spbu.ru> On Fri, Nov 24, 2006 at 12:59:40AM -0700, Fernando Perez wrote: > Ville M. Vainio wrote: > >On 11/23/06, Kirill Smelkov wrote: > > > >>That's what I originally did (see [patch 2/9], buf Fernando objected, I > >>can't understand why... > > > >Oh, I see, he probably just accidentally misread your patch. I'll go > >ahead and apply your original patch then (tomorrow). > > Yes, I did misread the original patch. I looked at it again just now, and > it's fine. Thanks. Can happen with anyone... -- ????? ????????, ??????. http://landau.phys.spbu.ru/~kirr/aiv/ From rocky at panix.com Sat Nov 25 18:06:30 2006 From: rocky at panix.com (R. Bernstein) Date: Sat, 25 Nov 2006 18:06:30 -0500 Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was: Extensions/pydb_ipy.py added) In-Reply-To: <46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com> References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com> <17731.8786.756329.573337@panix3.panix.com> <46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com> <17731.15518.833930.352892@panix3.panix.com> <46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com> <17731.31618.644784.714712@panix3.panix.com> <17765.7652.52085.132851@panix3.panix.com> <46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com> <17766.34184.188910.580317@panix3.panix.com> <46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com> Message-ID: <17768.52342.879269.903593@panix3.panix.com> Ville M. Vainio writes: ... > Well, this might save us some startup time & memory footprint. Good! > It should be quite easy to do, esp. for magics (and completers). We can > easily inject stub callables in place of magics, which 1) import a > specified module and 2) try to run the magic again. Good! > > I think a good first line of action would be to add an ipapi function > that permanently registers a magic name / completer trigger (or > whatever) with ipython. So this ipapi function doesn't exist yet, right? > > Here's what one could do e.g. in setup.py for pydb: > > import IPython.ipapi > ip = IPython.ipapi.get() > > ip.register_autoload("%pydb", "pydb.ipy_pydb") I'm guessing from the above that the ipapi function that needs to get added is called "register_autoload", correct? (Right now I don't see "register_autoload" in ipython sources.) The other thing that is not clear to me where or when this register_autoload is invoked. It has to have been invoked before running the first time %pydb is entered by a user since the autoload indicates what to import. It doesn't make sense to do anything when installing pydb. And the goal here was to be able to run "%pydb" without having to run an "import" first. And so it doesn't make sense to put this as the initialization of an import, either. The two natural places then would be in ipy_user_conf.py or ipy_system_conf.py. And it looks like all of the other extensions in ipython are imported in the latter. From rocky at panix.com Sat Nov 25 18:22:38 2006 From: rocky at panix.com (R. Bernstein) Date: Sat, 25 Nov 2006 18:22:38 -0500 Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was: Extensions/pydb_ipy.py added) In-Reply-To: <46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com> References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com> <17731.8786.756329.573337@panix3.panix.com> <46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com> <17731.15518.833930.352892@panix3.panix.com> <46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com> <17731.31618.644784.714712@panix3.panix.com> <17765.7652.52085.132851@panix3.panix.com> <46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com> <17766.34184.188910.580317@panix3.panix.com> <46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com> Message-ID: <17768.53310.125276.760922@panix3.panix.com> Ville M. Vainio writes: > It's just that having to import ipy_pydb would make enabling pydb a > configuration option. Sometimes one may wish to use standard pdb even > when pydb is installed. But one can't run %pydb and get pdb, so there's no ambiguity here. (For pdb one needs "%run -d") > > And the import need not be manual - it's quite easy to add "import > ipy_pydb" to your personal ipy_user_conf.py. Okay, will do for now. Thanks. From vivainio at gmail.com Sun Nov 26 03:43:13 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Sun, 26 Nov 2006 10:43:13 +0200 Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was: Extensions/pydb_ipy.py added) In-Reply-To: <17768.52342.879269.903593@panix3.panix.com> References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com> <46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com> <17731.15518.833930.352892@panix3.panix.com> <46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com> <17731.31618.644784.714712@panix3.panix.com> <17765.7652.52085.132851@panix3.panix.com> <46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com> <17766.34184.188910.580317@panix3.panix.com> <46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com> <17768.52342.879269.903593@panix3.panix.com> Message-ID: <46cb515a0611260043l5052ff22tc2d624676298dfa8@mail.gmail.com> On 11/26/06, R. Bernstein wrote: > > I think a good first line of action would be to add an ipapi function > > that permanently registers a magic name / completer trigger (or > > whatever) with ipython. > > So this ipapi function doesn't exist yet, right? Right. > > Here's what one could do e.g. in setup.py for pydb: > > > > import IPython.ipapi > > ip = IPython.ipapi.get() > > > > ip.register_autoload("%pydb", "pydb.ipy_pydb") > > I'm guessing from the above that the ipapi function that needs to get > added is called "register_autoload", correct? (Right now I don't see > "register_autoload" in ipython sources.) Yes. register_autoload doesn't exist yet, I'm still merely branistroming some ideas. > indicates what to import. It doesn't make sense to do anything when > installing pydb. And the goal here was to be able to run "%pydb" Why doesn't it make sense to do this when installing pydb? Isn't installation the exact time when things like this should be done? > The two natural places then would be in ipy_user_conf.py or > ipy_system_conf.py. And it looks like all of the other extensions in These are imported every time ipython starts. register_autoload would be a persistent operation, i.e. it should only be run only once and ipython stores the autoload map to somewhere in the ".ipython" directory which it reads on startup. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From vivainio at gmail.com Sun Nov 26 03:43:55 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Sun, 26 Nov 2006 10:43:55 +0200 Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was: Extensions/pydb_ipy.py added) In-Reply-To: <17768.53310.125276.760922@panix3.panix.com> References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com> <46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com> <17731.15518.833930.352892@panix3.panix.com> <46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com> <17731.31618.644784.714712@panix3.panix.com> <17765.7652.52085.132851@panix3.panix.com> <46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com> <17766.34184.188910.580317@panix3.panix.com> <46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com> <17768.53310.125276.760922@panix3.panix.com> Message-ID: <46cb515a0611260043w6dee36eek9881b7f937aea92d@mail.gmail.com> On 11/26/06, R. Bernstein wrote: > Ville M. Vainio writes: > > It's just that having to import ipy_pydb would make enabling pydb a > > configuration option. Sometimes one may wish to use standard pdb even > > when pydb is installed. > > But one can't run %pydb and get pdb, so there's no ambiguity > here. (For pdb one needs "%run -d") But for post-mortem, there's no such choice. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From vivainio at gmail.com Sun Nov 26 05:23:34 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Sun, 26 Nov 2006 12:23:34 +0200 Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was: Extensions/pydb_ipy.py added) In-Reply-To: <46cb515a0611260043l5052ff22tc2d624676298dfa8@mail.gmail.com> References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com> <17731.15518.833930.352892@panix3.panix.com> <46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com> <17731.31618.644784.714712@panix3.panix.com> <17765.7652.52085.132851@panix3.panix.com> <46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com> <17766.34184.188910.580317@panix3.panix.com> <46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com> <17768.52342.879269.903593@panix3.panix.com> <46cb515a0611260043l5052ff22tc2d624676298dfa8@mail.gmail.com> Message-ID: <46cb515a0611260223o29b3bff9qd607b6dfa8456385@mail.gmail.com> On 11/26/06, Ville M. Vainio wrote: > Why doesn't it make sense to do this when installing pydb? Isn't > installation the exact time when things like this should be done? Answering myself - The registration should probably be performed both during installation and import of the module. In installation the registration only applies to the user that happens to run setup.py, but other users can do the registration manually by just importing the module once. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From rocky at panix.com Sun Nov 26 10:01:48 2006 From: rocky at panix.com (R. Bernstein) Date: Sun, 26 Nov 2006 10:01:48 -0500 Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was: Extensions/pydb_ipy.py added) In-Reply-To: <46cb515a0611260223o29b3bff9qd607b6dfa8456385@mail.gmail.com> References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com> <17731.15518.833930.352892@panix3.panix.com> <46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com> <17731.31618.644784.714712@panix3.panix.com> <17765.7652.52085.132851@panix3.panix.com> <46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com> <17766.34184.188910.580317@panix3.panix.com> <46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com> <17768.52342.879269.903593@panix3.panix.com> <46cb515a0611260043l5052ff22tc2d624676298dfa8@mail.gmail.com> <46cb515a0611260223o29b3bff9qd607b6dfa8456385@mail.gmail.com> Message-ID: <17769.44124.934324.948419@panix3.panix.com> Ville M. Vainio writes: > On 11/26/06, Ville M. Vainio wrote: > > > Why doesn't it make sense to do this when installing pydb? Isn't > > installation the exact time when things like this should be done? > > Answering myself - > > The registration should probably be performed both during installation > and import of the module. In installation the registration only > applies to the user that happens to run setup.py, but other users can > do the registration manually by just importing the module once. What happens if I install pydb first and then install ipython? And as for doing somehting ipython-wise inside a pydb import, you have the same issue as you brought up before: I may import pydb outside of ipython, and/or may not want it to get used automatically or every time inside ipython. For now, ipy_user_conf.py seems like right place to put an "import ipy_pydb" to indicate you want %pydb to work. From rocky at panix.com Sun Nov 26 10:06:40 2006 From: rocky at panix.com (R. Bernstein) Date: Sun, 26 Nov 2006 10:06:40 -0500 Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was: Extensions/pydb_ipy.py added) In-Reply-To: <46cb515a0611260043w6dee36eek9881b7f937aea92d@mail.gmail.com> References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com> <46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com> <17731.15518.833930.352892@panix3.panix.com> <46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com> <17731.31618.644784.714712@panix3.panix.com> <17765.7652.52085.132851@panix3.panix.com> <46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com> <17766.34184.188910.580317@panix3.panix.com> <46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com> <17768.53310.125276.760922@panix3.panix.com> <46cb515a0611260043w6dee36eek9881b7f937aea92d@mail.gmail.com> Message-ID: <17769.44416.663746.783851@panix3.panix.com> Ville M. Vainio writes: > On 11/26/06, R. Bernstein wrote: > > > Ville M. Vainio writes: > > > It's just that having to import ipy_pydb would make enabling pydb a > > > configuration option. Sometimes one may wish to use standard pdb even > > > when pydb is installed. > > > > But one can't run %pydb and get pdb, so there's no ambiguity > > here. (For pdb one needs "%run -d") > > But for post-mortem, there's no such choice. Yes, and the way things are done right now in ipython that is a completly separate issue. This thread was about how not require an "import ipy_pydb" if someone wants to use %pydb. From vivainio at gmail.com Sun Nov 26 10:20:37 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Sun, 26 Nov 2006 17:20:37 +0200 Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was: Extensions/pydb_ipy.py added) In-Reply-To: <17769.44124.934324.948419@panix3.panix.com> References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com> <17731.31618.644784.714712@panix3.panix.com> <17765.7652.52085.132851@panix3.panix.com> <46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com> <17766.34184.188910.580317@panix3.panix.com> <46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com> <17768.52342.879269.903593@panix3.panix.com> <46cb515a0611260043l5052ff22tc2d624676298dfa8@mail.gmail.com> <46cb515a0611260223o29b3bff9qd607b6dfa8456385@mail.gmail.com> <17769.44124.934324.948419@panix3.panix.com> Message-ID: <46cb515a0611260720q68745e6pbc8147f329901f17@mail.gmail.com> On 11/26/06, R. Bernstein wrote: > What happens if I install pydb first and then install ipython? And as Then you'll need to run "import ipy_pydb" once in ipython. > for doing somehting ipython-wise inside a pydb import, you have the > same issue as you brought up before: I may import pydb outside of > ipython, and/or may not want it to get used automatically or every > time inside ipython. IPython.ipapi.get() returns None if you are not running in IPython. This, and "except ImportError" can be done to safely determine whether you should do ipython related stuff. > For now, ipy_user_conf.py seems like right place to put an "import > ipy_pydb" to indicate you want %pydb to work. Exactly. And it will stay as the recommended place to load extensions. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From vivainio at gmail.com Sun Nov 26 10:25:29 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Sun, 26 Nov 2006 17:25:29 +0200 Subject: [IPython-dev] Mercurial queues Message-ID: <46cb515a0611260725j5d1a8bb4v75c7503c1e9348f8@mail.gmail.com> A blast from the past - Stefan Reich?r wrote: > I also think that bzr is a good solution for that kind of stuff. > However, I use Mercurial Queues (mq) for tracking svn repositories and > generating patches. I took a look at mq myself, and it seems like a very good idea when working off-SVN: http://www.selenic.com/mercurial/wiki/index.cgi/MqExtension All in all, Mercurial looks / feels very promising... -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From vivainio at gmail.com Sun Nov 26 12:41:07 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Sun, 26 Nov 2006 19:41:07 +0200 Subject: [IPython-dev] Mercurial queues In-Reply-To: <46cb515a0611260725j5d1a8bb4v75c7503c1e9348f8@mail.gmail.com> References: <46cb515a0611260725j5d1a8bb4v75c7503c1e9348f8@mail.gmail.com> Message-ID: <46cb515a0611260941v45b9639ax2b325b3e4c51de5@mail.gmail.com> On 11/26/06, Ville M. Vainio wrote: > All in all, Mercurial looks / feels very promising... And on that tangent, there's now a stock completer for "hg" which you can invoke by pressing tab after typing the hg command: [IPython]|8> hg q qapplied qdiff qimport qpop qrename qseries qclone qfold qinit qprev qrestore qtop qcommit qguard qnew qpush qsave qunapplied qdelete qheader qnext qrefresh qselect [IPython]|8> hg q -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From vivainio at gmail.com Sun Nov 26 16:34:30 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Sun, 26 Nov 2006 23:34:30 +0200 Subject: [IPython-dev] svnversion used for Release.py Message-ID: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com> I've added a new tool, tools/update_revnum.py that modifies Release.py (gets the revision number with svnversion command; svn doesn't support getting global revision command via $Revision$ like tags). Now the distribution names of svn versions are handier: http://vivainio.googlepages.com/ipython-0.7.3.svn.r1937.win32.exe I think we could put out a beta / whatever release soon. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From stefan at sun.ac.za Mon Nov 27 03:08:03 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Mon, 27 Nov 2006 10:08:03 +0200 Subject: [IPython-dev] Mercurial queues In-Reply-To: <46cb515a0611260941v45b9639ax2b325b3e4c51de5@mail.gmail.com> References: <46cb515a0611260725j5d1a8bb4v75c7503c1e9348f8@mail.gmail.com> <46cb515a0611260941v45b9639ax2b325b3e4c51de5@mail.gmail.com> Message-ID: <20061127080803.GB30805@mentat.za.net> Hi Ville Please find a patch for bzr completion attached. Regards St?fan On Sun, Nov 26, 2006 at 07:41:07PM +0200, Ville M. Vainio wrote: > On 11/26/06, Ville M. Vainio wrote: > > > All in all, Mercurial looks / feels very promising... > > And on that tangent, there's now a stock completer for "hg" which you > can invoke by pressing tab after typing the hg command: > > [IPython]|8> hg q > qapplied qdiff qimport qpop qrename qseries > qclone qfold qinit qprev qrestore qtop > qcommit qguard qnew qpush qsave qunapplied > qdelete qheader qnext qrefresh qselect > [IPython]|8> hg q -------------- next part -------------- Index: IPython/Extensions/ipy_stock_completers.py =================================================================== --- IPython/Extensions/ipy_stock_completers.py (revision 1942) +++ IPython/Extensions/ipy_stock_completers.py (working copy) @@ -113,6 +113,42 @@ ip.set_hook('complete_command', hg_completer, str_key = 'hg') +bzr_commands = """ +add annotate bind branch break-lock bundle-revisions cat check +checkout commit conflicts deleted diff export gannotate gbranch +gcommit gdiff help ignore ignored info init init-repository inventory +log merge missing mkdir mv nick pull push reconcile register-branch +remerge remove renames resolve revert revno root serve sign-my-commits +status testament unbind uncommit unknowns update upgrade version +version-info visualise whoami +""" + +def bzr_completer(self,event): + """ Completer for bazaar commands """ + cmd_param = event.line.split() + if event.line.endswith(' '): + cmd_param.append('') + + if len(cmd_param) > 2: + cmd = cmd_param[1] + param = cmd_param[-1] + output_file = (param == '--output=') + if cmd == 'help': + return bzr_commands.split() + elif cmd in ['bundle-revisions','conflicts', + 'deleted','nick','register-branch', + 'serve','unbind','upgrade','version', + 'whoami'] and not output_file: + return [] + else: + # the rest are probably file names + return ip.IP.Completer.file_matches(event.symbol) + + return bzr_commands.split() + +ip.set_hook('complete_command', bzr_completer, str_key = 'bzr') + + def runlistpy(self, event): comps = shlex.split(event.line) relpath = (len(comps) > 1 and comps[-1] or '') From vivainio at gmail.com Mon Nov 27 06:25:28 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Mon, 27 Nov 2006 13:25:28 +0200 Subject: [IPython-dev] Mercurial queues In-Reply-To: <20061127105147.GA32697@mentat.za.net> References: <46cb515a0611260725j5d1a8bb4v75c7503c1e9348f8@mail.gmail.com> <46cb515a0611260941v45b9639ax2b325b3e4c51de5@mail.gmail.com> <20061127080803.GB30805@mentat.za.net> <46cb515a0611270149s200603a6n9a1c49d04a656bab@mail.gmail.com> <20061127105147.GA32697@mentat.za.net> Message-ID: <46cb515a0611270325y2314ad6am2780b02914400e1a@mail.gmail.com> On 11/27/06, Stefan van der Walt wrote: > What in the world happened there?! The file is fine on my machine, > but I also see the line-feeds in the incoming message. > > Here it is again, gzipped. It worked, thanks! I wonder whether we could someday add also options for the commands (e.g. svn import --username) - though I suppose it would have to be after we get the autoload stuff in (post-0.7.3), to avoid too large databases. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From stefan at sun.ac.za Mon Nov 27 07:12:45 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Mon, 27 Nov 2006 14:12:45 +0200 Subject: [IPython-dev] Mercurial queues In-Reply-To: <46cb515a0611270325y2314ad6am2780b02914400e1a@mail.gmail.com> References: <46cb515a0611260725j5d1a8bb4v75c7503c1e9348f8@mail.gmail.com> <46cb515a0611260941v45b9639ax2b325b3e4c51de5@mail.gmail.com> <20061127080803.GB30805@mentat.za.net> <46cb515a0611270149s200603a6n9a1c49d04a656bab@mail.gmail.com> <20061127105147.GA32697@mentat.za.net> <46cb515a0611270325y2314ad6am2780b02914400e1a@mail.gmail.com> Message-ID: <20061127121245.GC32697@mentat.za.net> Hi Ville On Mon, Nov 27, 2006 at 01:25:28PM +0200, Ville M. Vainio wrote: > On 11/27/06, Stefan van der Walt wrote: > > >What in the world happened there?! The file is fine on my machine, > >but I also see the line-feeds in the incoming message. > > > >Here it is again, gzipped. > > It worked, thanks! I wonder whether we could someday add also options > for the commands (e.g. svn import --username) - though I suppose it > would have to be after we get the autoload stuff in (post-0.7.3), to > avoid too large databases. I thought about this too. I was trying to imaging what kind of structure one would need to describe the version control completer. One would intuitively build something like vc_commands = ['info','update', 'bundle', ...] vc_subcommands = {'bundle' : ['--output','-r'], ...} vc_subcommand_expand = {'--output': file_complete, '-r': revision_complete, ...} def revision_complete(self, input): rev_re = re.compile('...') cur_rev = rev_re.match(commands.getoutput('bzr log -r -1')...) return ['BASE',cur_rev,'TIP',...] On the other hand, how do we keep such structures up to date? It would be ideal if the program itself could provide a clue to its completion. I think these structures already exist in bzr and hg. In fact, looking in the directory tree, I think we are looking for bzrlib/shellcomplete.py I'll grab the latest copy of bzr and play around. Cheers St?fan From vivainio at gmail.com Mon Nov 27 07:34:50 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Mon, 27 Nov 2006 14:34:50 +0200 Subject: [IPython-dev] Mercurial queues In-Reply-To: <20061127121245.GC32697@mentat.za.net> References: <46cb515a0611260725j5d1a8bb4v75c7503c1e9348f8@mail.gmail.com> <46cb515a0611260941v45b9639ax2b325b3e4c51de5@mail.gmail.com> <20061127080803.GB30805@mentat.za.net> <46cb515a0611270149s200603a6n9a1c49d04a656bab@mail.gmail.com> <20061127105147.GA32697@mentat.za.net> <46cb515a0611270325y2314ad6am2780b02914400e1a@mail.gmail.com> <20061127121245.GC32697@mentat.za.net> Message-ID: <46cb515a0611270434u7aab0449x3f3492116c1a2126@mail.gmail.com> On 11/27/06, Stefan van der Walt wrote: > I thought about this too. I was trying to imaging what kind of > structure one would need to describe the version control completer. > One would intuitively build something like > > vc_commands = ['info','update', 'bundle', ...] > vc_subcommands = {'bundle' : ['--output','-r'], ...} > vc_subcommand_expand = {'--output': file_complete, > '-r': revision_complete, > ...} > > def revision_complete(self, input): > rev_re = re.compile('...') > cur_rev = rev_re.match(commands.getoutput('bzr log -r -1')...) > > return ['BASE',cur_rev,'TIP',...] I think you might be overengineering a little bit. Just looking at the command and yielding ['--output','-r'] + file_completions would be just fine for most purposes. That way, if the user types - and presses tab, she gets the list of options for that command. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From fperez.net at gmail.com Tue Nov 28 09:27:07 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 28 Nov 2006 07:27:07 -0700 Subject: [IPython-dev] svnversion used for Release.py In-Reply-To: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com> References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com> Message-ID: On 11/26/06, Ville M. Vainio wrote: > I've added a new tool, tools/update_revnum.py that modifies Release.py > (gets the revision number with svnversion command; svn doesn't support > getting global revision command via $Revision$ like tags). > > Now the distribution names of svn versions are handier: > > http://vivainio.googlepages.com/ipython-0.7.3.svn.r1937.win32.exe Great, thanks. I'd been wanting this for a long time. > I think we could put out a beta / whatever release soon. +1. I'll be with limited online activity this week, and pretty much offline the next. We could target a release for the end of the year. Feel free to put out the betas sooner though, or let me know and I can upload one to the dist/testing dir anytime (before Saturday when I go offline). Cheers, f From fperez.net at gmail.com Wed Nov 29 01:00:04 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 28 Nov 2006 23:00:04 -0700 Subject: [IPython-dev] svnversion used for Release.py In-Reply-To: <17772.56028.653337.85310@panix3.panix.com> References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com> <17772.56028.653337.85310@panix3.panix.com> Message-ID: On 11/28/06, R. Bernstein wrote: > Before you go offline and before the next ipython release comes out, > it would be good to get this from > http://projects.scipy.org/pipermail/ipython-dev/2006-October/002413.html > cleared up. Thanks ... > > ---- > > One last strongly related topic which includes Fernando's comment about > the necessity of Python 2.4 support. It was the following readline > workaround in Debugger.py around line 68 that caused me to initially > decided to punt on Python < 2.5: > > > if sys.version[:3] >= '2.5': > def __init__(self,color_scheme='NoColor',completekey=None, > stdin=None, stdout=None): > ... # I'll come back to this below > else: > # Ugly hack: for Python 2.3-2.4, we can't call the parent constructor, > # because it binds readline and breaks tab-completion. This means we > # have to COPY the constructor here. > def __init__(self,color_scheme='NoColor'): > ... #1 > > I think the intent at #1 above is basically to copy the code in pdb's > __init__ constructor omitting "import readline" and the code associated > with implications of that. > > So it looks to me like the test on version 2.5 has more to do with > testing the version of *pdb* used than anything else version 2.4 or > version 2.5. Is this correct? > > Assuming this is the case, when I look the 2.4 pdb versus the 2.5 pdb, > I see THAT THEY BOTH IMPORT READLINE. In other words the 2.5 branch > ("I'll come back to this below") should have pdb code copied minus the > import readline. This has to be a separate branch still because it is > different code. But none of it appears, so I'd be curious to learn why > it's not needed in the 2.5 branch. No, the problem is a bit more subtle than that: it's not the 'import readline' that causes problems, but rather the fact that the constructor fires this call: 2.4: class Pdb(bdb.Bdb, cmd.Cmd): def __init__(self): bdb.Bdb.__init__(self) cmd.Cmd.__init__(self) 2.5: class Pdb(bdb.Bdb, cmd.Cmd): def __init__(self, completekey='tab', stdin=None, stdout=None): bdb.Bdb.__init__(self) cmd.Cmd.__init__(self, completekey, stdin, stdout) The critical difference is that the 2.5 constructor exposes the 'completekey' parameter publicly, and correctly passes it down to the cmd.Cmd constructor, so we can nicely override it to None and avoid any readline-related misconfiguration issues. Prior to 2.5, this parameter was not accessible from the Pdb constructor and the default value of 'tab' would always fire. Does that help? Cheers, f From fperez.net at gmail.com Wed Nov 29 01:03:11 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 28 Nov 2006 23:03:11 -0700 Subject: [IPython-dev] svnversion used for Release.py In-Reply-To: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com> References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com> Message-ID: On 11/26/06, Ville M. Vainio wrote: > I've added a new tool, tools/update_revnum.py that modifies Release.py > (gets the revision number with svnversion command; svn doesn't support > getting global revision command via $Revision$ like tags). > > Now the distribution names of svn versions are handier: > > http://vivainio.googlepages.com/ipython-0.7.3.svn.r1937.win32.exe > > I think we could put out a beta / whatever release soon. You may have already noticed, but %pdb is systematically crashing. I'm getting the dreaded: ipdb>q *** ERROR *** This version of pdb has a bug and crashed. Returning to IPython... WARNING: Failure executing file: on 2.4 and 2.5 for even the most trivial tests. I won't be able to debug this before I leave, but I know it was working fine not very long ago, b/c I use %pdb pretty regularly. You might have an idea of what happened here... Cheers, f From vivainio at gmail.com Wed Nov 29 02:43:11 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Wed, 29 Nov 2006 09:43:11 +0200 Subject: [IPython-dev] svnversion used for Release.py In-Reply-To: References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com> Message-ID: <46cb515a0611282343p620c24ebi6ffb2c7ccc2a784c@mail.gmail.com> On 11/29/06, Fernando Perez wrote: > You may have already noticed, but %pdb is systematically crashing. > I'm getting the dreaded: > > > ipdb>q > *** ERROR *** > This version of pdb has a bug and crashed. > Returning to IPython... > WARNING: Failure executing file: > > on 2.4 and 2.5 for even the most trivial tests. I won't be able to > debug this before I leave, but I know it was working fine not very > long ago, b/c I use %pdb pretty regularly. You might have an idea of > what happened here... Yes, it's the old problem with "old_all_completions". It's an easy fix, I'll attend to it today. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From vivainio at gmail.com Wed Nov 29 03:04:00 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Wed, 29 Nov 2006 10:04:00 +0200 Subject: [IPython-dev] svnversion used for Release.py In-Reply-To: <46cb515a0611282343p620c24ebi6ffb2c7ccc2a784c@mail.gmail.com> References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com> <46cb515a0611282343p620c24ebi6ffb2c7ccc2a784c@mail.gmail.com> Message-ID: <46cb515a0611290004s2a03eb5r49dbedddf1a8ddb8@mail.gmail.com> On 11/29/06, Ville M. Vainio wrote: > > on 2.4 and 2.5 for even the most trivial tests. I won't be able to > > debug this before I leave, but I know it was working fine not very > > long ago, b/c I use %pdb pretty regularly. You might have an idea of > > what happened here... > > Yes, it's the old problem with "old_all_completions". It's an easy > fix, I'll attend to it today. Done. rev 1953 is the new "beta 1". -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From rocky at panix.com Wed Nov 29 03:27:59 2006 From: rocky at panix.com (R. Bernstein) Date: Wed, 29 Nov 2006 03:27:59 -0500 Subject: [IPython-dev] svnversion used for Release.py In-Reply-To: <46cb515a0611282343p620c24ebi6ffb2c7ccc2a784c@mail.gmail.com> References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com> <46cb515a0611282343p620c24ebi6ffb2c7ccc2a784c@mail.gmail.com> Message-ID: <17773.17551.326725.76625@panix3.panix.com> Looks like it's fixed in revison 1953. There is still a stray "print" statement I mentioned a while ago. -------------- next part -------------- A non-text attachment was scrubbed... Name: pdb.diff Type: application/octet-stream Size: 447 bytes Desc: Remove a stray debug "print" statement URL: -------------- next part -------------- Thanks. Ville M. Vainio writes: > On 11/29/06, Fernando Perez wrote: > > > You may have already noticed, but %pdb is systematically crashing. > > I'm getting the dreaded: > > > > > > ipdb>q > > *** ERROR *** > > This version of pdb has a bug and crashed. > > Returning to IPython... > > WARNING: Failure executing file: > > > > on 2.4 and 2.5 for even the most trivial tests. I won't be able to > > debug this before I leave, but I know it was working fine not very > > long ago, b/c I use %pdb pretty regularly. You might have an idea of > > what happened here... > > Yes, it's the old problem with "old_all_completions". It's an easy > fix, I'll attend to it today. > > -- > Ville M. Vainio - vivainio.googlepages.com > blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/ipython-dev > From rocky at panix.com Wed Nov 29 03:57:28 2006 From: rocky at panix.com (R. Bernstein) Date: Wed, 29 Nov 2006 03:57:28 -0500 Subject: [IPython-dev] svnversion used for Release.py In-Reply-To: References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com> <17772.56028.653337.85310@panix3.panix.com> Message-ID: <17773.19320.848097.530795@panix3.panix.com> Fernando Perez writes: > No, the problem is a bit more subtle than that: it's not the 'import > readline' that causes problems, but rather the fact that the > constructor fires this call: > > 2.4: > class Pdb(bdb.Bdb, cmd.Cmd): > > def __init__(self): > bdb.Bdb.__init__(self) > cmd.Cmd.__init__(self) > > 2.5: > class Pdb(bdb.Bdb, cmd.Cmd): > > def __init__(self, completekey='tab', stdin=None, stdout=None): > bdb.Bdb.__init__(self) > cmd.Cmd.__init__(self, completekey, stdin, stdout) > > The critical difference is that the 2.5 constructor exposes the > 'completekey' parameter publicly, and correctly passes it down to the > cmd.Cmd constructor, so we can nicely override it to None and avoid > any readline-related misconfiguration issues. Prior to 2.5, this > parameter was not accessible from the Pdb constructor and the default > value of 'tab' would always fire. > > > Does that help? This helps a bit, thanks. A suitably recent version of the pydb follows the Python 2.5 constructor convention - it doesn't matter what the Python version is. If pydb is used, (and we do check for a suitably recent version), then that hack is *not* needed. So class Pdb(OldPdb): """Modified Pdb class, does not load readline.""" if sys.version[:3] >= '2.5' can become if sys.version[:3] >= '2.5' or has_pydb: I've attached a patch for this below. I'm still not clear about one other thing though, and this is not addressed in the patch but perhaps should be. When you create the debugger you turn off any readline completion, right? But is this correct whne pydb is used? pydb sets up command completion correctly for itself, and I'd venture to say that this would be more helpful in the debugger context than the default ipython command completion. -------------- next part -------------- A non-text attachment was scrubbed... Name: pydb-debug.diff Type: application/octet-stream Size: 923 bytes Desc: Don't need Python 2.3, 2.4 hack if pydb is used. URL: From fperez.net at gmail.com Thu Nov 30 00:28:26 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 29 Nov 2006 22:28:26 -0700 Subject: [IPython-dev] svnversion used for Release.py In-Reply-To: <17773.19320.848097.530795@panix3.panix.com> References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com> <17772.56028.653337.85310@panix3.panix.com> <17773.19320.848097.530795@panix3.panix.com> Message-ID: On 11/29/06, R. Bernstein wrote: > This helps a bit, thanks. A suitably recent version of the pydb > follows the Python 2.5 constructor convention - it doesn't matter what > the Python version is. If pydb is used, (and we do check for a > suitably recent version), then that hack is *not* needed. So > > class Pdb(OldPdb): > """Modified Pdb class, does not load readline.""" > > if sys.version[:3] >= '2.5' > > can become > > if sys.version[:3] >= '2.5' or has_pydb: > > I've attached a patch for this below. I think Ville already applied it, thanks. > I'm still not clear about one other thing though, and this is not > addressed in the patch but perhaps should be. > > When you create the debugger you turn off any readline completion, > right? But is this correct whne pydb is used? pydb sets up command > completion correctly for itself, and I'd venture to say that this > would be more helpful in the debugger context than the default ipython > command completion. IPython simply prevents pdb (the builtin one) from mucking around with our own completers (it would otherwise call rlcompleter and completely disable ipython's completions). It's quite possible that this disables pydb's completions, and in that case we should try to accomodate both needs: preventing pdb from breaking ipython's completion, while allowing pydb's completion to function. If that's not the case, patches welcome :) I'll be 90% offline starting now until about Dec. 12. I did manage to get a patch in which I'd wanted for a long time; since it's debugger related, your testing and feedback would be welcome: %debug now activates the interactive debugger (including pydb if available) AFTER an exception has fired, even if you had %pdb off. This lets you inspect a crashed code interactively without having to do the old dance '%pdb on; rerun code; %pdb off'. I like it a lot more. Cheers, f From vivainio at gmail.com Thu Nov 30 02:28:02 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Thu, 30 Nov 2006 09:28:02 +0200 Subject: [IPython-dev] svnversion used for Release.py In-Reply-To: References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com> <17772.56028.653337.85310@panix3.panix.com> <17773.19320.848097.530795@panix3.panix.com> Message-ID: <46cb515a0611292328w7cb979a8vdeeed1a50982c0b3@mail.gmail.com> On 11/30/06, Fernando Perez wrote: > now activates the interactive debugger (including pydb if available) > AFTER an exception has fired, even if you had %pdb off. This lets you > inspect a crashed code interactively without having to do the old > dance '%pdb on; rerun code; %pdb off'. I like it a lot more. Sounds great. If it works in all cases, we could as well deprecate %pdb, which suddenly became quite pointless. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From fperez.net at gmail.com Thu Nov 30 02:35:05 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 30 Nov 2006 00:35:05 -0700 Subject: [IPython-dev] svnversion used for Release.py In-Reply-To: <46cb515a0611292328w7cb979a8vdeeed1a50982c0b3@mail.gmail.com> References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com> <17772.56028.653337.85310@panix3.panix.com> <17773.19320.848097.530795@panix3.panix.com> <46cb515a0611292328w7cb979a8vdeeed1a50982c0b3@mail.gmail.com> Message-ID: On 11/30/06, Ville M. Vainio wrote: > On 11/30/06, Fernando Perez wrote: > > > now activates the interactive debugger (including pydb if available) > > AFTER an exception has fired, even if you had %pdb off. This lets you > > inspect a crashed code interactively without having to do the old > > dance '%pdb on; rerun code; %pdb off'. I like it a lot more. > > Sounds great. If it works in all cases, we could as well deprecate > %pdb, which suddenly became quite pointless. Not quite: if you're in a rapid edit/run/debug cycle, the automatic behavior of %pdb is nice. It's also foolproof: it fires immediately so there's no chance for a typo on your part to clobber sys.last_traceback and make %debug useless. So I think that %debug is what I'll use now 90% of the time, but I still see a (admittedly marginal) use for %pdb. Cheers, f From rocky at panix.com Thu Nov 30 04:21:41 2006 From: rocky at panix.com (R. Bernstein) Date: Thu, 30 Nov 2006 04:21:41 -0500 Subject: [IPython-dev] svnversion used for Release.py In-Reply-To: References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com> <17772.56028.653337.85310@panix3.panix.com> <17773.19320.848097.530795@panix3.panix.com> Message-ID: <17774.41637.724154.424378@panix3.panix.com> Fernando Perez writes: > On 11/29/06, R. Bernstein wrote: > > > This helps a bit, thanks. A suitably recent version of the pydb > > follows the Python 2.5 constructor convention - it doesn't matter what > > the Python version is. If pydb is used, (and we do check for a > > suitably recent version), then that hack is *not* needed. So > > > > class Pdb(OldPdb): > > """Modified Pdb class, does not load readline.""" > > > > if sys.version[:3] >= '2.5' > > > > can become > > > > if sys.version[:3] >= '2.5' or has_pydb: > > > > I've attached a patch for this below. > > I think Ville already applied it, thanks. This part yes. A couple of the others no, and I don't understand why not. One of the patches was to remove a temporary debugging print in IPython/Magic.py that accidentally crept in. I've included it in the patch below. > > > I'm still not clear about one other thing though, and this is not > > addressed in the patch but perhaps should be. > > > > When you create the debugger you turn off any readline completion, > > right? But is this correct whne pydb is used? pydb sets up command > > completion correctly for itself, and I'd venture to say that this > > would be more helpful in the debugger context than the default ipython > > command completion. > > IPython simply prevents pdb (the builtin one) from mucking around with > our own completers (it would otherwise call rlcompleter and completely > disable ipython's completions). I'm having deja vu. This issue isn't strictly a debugger issue. Any program that uses readline will have this, and ipython users can invoke any such program run say via %run. So ipython needs to protect itself in a general way. A while back Ville noticed command history bleeding between the pydb (or more generally the thing %run'd and ipython). So that was addressed in a more general way than had been addressed via pdb. Well at least from the bleeding into ipython. The bleeding in the other direction, from ipython to the pydb (or any readline application) is still there and Ville seems to prefer that. As for completely disabling ipython's completions, I don't see this happening at least with pydb and the current patch. I run an ipython completion before issuing a %pydb or going into post-mortem. Then I see pydb's completion inside the debugger. I leave that and I get the same completions in ipython that I had before. However since I'm not completely certain of the erroneous situation that was happening before, I am not sure my test is the right one. > It's quite possible that this > disables pydb's completions, and in that case we should try to > accomodate both needs: preventing pdb from breaking ipython's > completion, while allowing pydb's completion to function. I just checked and ipython does disable pydb's completions. In the process, I notice that ipython doesn't allow the completion key [tab] to be specified. Right? (If it does, I would modify the patch slightly) > If that's not the case, patches welcome :) Okay - here is a patch and the current outstanding patches. If there are comments, let me know. -------------- next part -------------- A non-text attachment was scrubbed... Name: pydb-4.patch Type: application/octet-stream Size: 2507 bytes Desc: Another round of pydb patches. Some new, some old URL: -------------- next part -------------- > > I'll be 90% offline starting now until about Dec. 12. > > I did manage to get a patch in which I'd wanted for a long time; since > it's debugger related, your testing and feedback would be welcome: > > %debug > > now activates the interactive debugger (including pydb if available) > AFTER an exception has fired, even if you had %pdb off. This lets you > inspect a crashed code interactively without having to do the old > dance '%pdb on; rerun code; %pdb off'. I like it a lot more. Well, one thing I like a lot more about it is that it doesn't change the debugger prompt. :-) I've never understood the advantage of changing that. And here's why it is unhelpful. If you are running py-shell inside Emacs or perhaps say pick your favorite IDE that allows one to go into ipython as a shell, then that program need to be sensitive to this change. If the enclosing program needed to modify its behavior as a result, I could see the distinction, but my experience has been it doesn't. Yet, in the case of py-shell, it means even more complicated regular expressions. Also note that recursive debug sessions are indicated by enclosing in a set of parenthesis. So while the prompt may start off say "ipdb>" if there is a recursive debug it would become "(ipdb>) ". > > Cheers, > > f > Cheers From vivainio at gmail.com Thu Nov 30 06:59:34 2006 From: vivainio at gmail.com (Ville M. Vainio) Date: Thu, 30 Nov 2006 13:59:34 +0200 Subject: [IPython-dev] svnversion used for Release.py In-Reply-To: <17774.41637.724154.424378@panix3.panix.com> References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com> <17772.56028.653337.85310@panix3.panix.com> <17773.19320.848097.530795@panix3.panix.com> <17774.41637.724154.424378@panix3.panix.com> Message-ID: <46cb515a0611300359v598c4401w7608455832845f9c@mail.gmail.com> On 11/30/06, R. Bernstein wrote: > This part yes. A couple of the others no, and I don't understand why > not. One of the patches was to remove a temporary debugging print in > IPython/Magic.py that accidentally crept in. I've included it in the > patch below. Accidental loss of the patch. I'll apply this patch today. > A while back Ville noticed command history bleeding between the pydb > (or more generally the thing %run'd and ipython). So that was > addressed in a more general way than had been addressed via pdb. Well > at least from the bleeding into ipython. The bleeding in the other > direction, from ipython to the pydb (or any readline application) is > still there and Ville seems to prefer that. Yes, it's the responsibility of called application to clear the history if they want an empty history. > I just checked and ipython does disable pydb's completions. In the > process, I notice that ipython doesn't allow the completion key [tab] > to be specified. Right? (If it does, I would modify the patch > slightly) It's probably a readline config issue, but I think we should not encourage or even allow using non-tab charactersp -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'