From vlada.peric at gmail.com Wed Jun 8 13:07:15 2011 From: vlada.peric at gmail.com (=?UTF-8?Q?Vladimir_Peri=C4=87?=) Date: Wed, 8 Jun 2011 13:07:15 +0200 Subject: [Python-porting] 2to3 speed Message-ID: <BANLkTikBwzNvtb22UacdJLsPLTeTuQeJ7Q@mail.gmail.com> Hello, I'm trying to port SymPy[0] (symbolic mathematics library) to Python 3. The goal is to use a single code base and then run 2to3 automatically when building for Python 3. Unfortunately, I've hit a small problem: running 2to3 takes a _long_ time, somewhere around 10 mins. This is a problem for users, but a particularly big one for me as it makes development last much longer (the change (Py2) code, run 2to3, test cycle takes a long time). Is there a way to speed up 2to3 execution? I do realize we've got quite a large code base. Will running 2to3 with some fixers disabled help with running time (ie. if I know I've fixed reduce and callable before, I might as well disable the fixers)? Any other advice to making it run faster? Thanks a lot! [0] http://sympy.org/ -- Vladimir Peri? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-porting/attachments/20110608/4972894b/attachment.html> From regebro at gmail.com Wed Jun 8 13:20:07 2011 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 8 Jun 2011 13:20:07 +0200 Subject: [Python-porting] 2to3 speed In-Reply-To: <BANLkTikBwzNvtb22UacdJLsPLTeTuQeJ7Q@mail.gmail.com> References: <BANLkTikBwzNvtb22UacdJLsPLTeTuQeJ7Q@mail.gmail.com> Message-ID: <BANLkTim5Body3zCF55P9_tB3g-jAPYJ1oQ@mail.gmail.com> On Wed, Jun 8, 2011 at 13:07, Vladimir Peri? <vlada.peric at gmail.com> wrote: > Hello, > > I'm trying to port SymPy[0] (symbolic mathematics library) to Python 3. The > goal is to use a single code base and then run 2to3 automatically when > building for Python 3. Unfortunately, I've hit a small problem: running 2to3 > takes a _long_ time, somewhere around 10 mins. This is a problem for users, > but a particularly big one for me as it makes development last much longer > (the change (Py2) code, run 2to3, test cycle takes a long time). Is there a > way to speed up 2to3 execution? I do realize we've got quite a large code > base. Will running 2to3 with some fixers disabled help with running time > (ie. if I know I've fixed reduce and callable before, I might as well > disable the fixers)? Any other advice to making it run faster? The major way to speed up 2to3 during development is to only run it on the files that has changed, by comparing timestamps. Running it on one file usually only takes a couple of seconds. More explanation here: http://python3porting.com/strategies.html#converting-to-python-3-with-2to3 Code examples for how to do this exists in distutils build_2to3 command, if I remember correctly, but since SymPy is already using distutils I would recommend using the build_2to3 command distutils has. Or in fact, I'd recommend using Distribute as it's test command already will run the tests in the build directory and hence automatically use 2to3 if you have set it up to do that. See http://packages.python.org/distribute/python3.html for more on that. You can speed it up a bit more by only including fixers that you are using, but for a complex software that is likely to be most of them anyway. //Lennart From benjamin at python.org Wed Jun 8 16:01:02 2011 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 8 Jun 2011 09:01:02 -0500 Subject: [Python-porting] 2to3 speed In-Reply-To: <BANLkTikBwzNvtb22UacdJLsPLTeTuQeJ7Q@mail.gmail.com> References: <BANLkTikBwzNvtb22UacdJLsPLTeTuQeJ7Q@mail.gmail.com> Message-ID: <BANLkTi=zzcZx8oa42DU2PWqcoRdZ_Q_e5Q@mail.gmail.com> 2011/6/8 Vladimir Peri? <vlada.peric at gmail.com>: > Hello, > > I'm trying to port SymPy[0] (symbolic mathematics library) to Python 3. The > goal is to use a single code base and then run 2to3 automatically when > building for Python 3. Unfortunately, I've hit a small problem: running 2to3 > takes a _long_ time, somewhere around 10 mins. This is a problem for users, > but a particularly big one for me as it makes development last much longer > (the change (Py2) code, run 2to3, test cycle takes a long time). Is there a > way to speed up 2to3 execution? I do realize we've got quite a large code > base. Will running 2to3 with some fixers disabled help with running time > (ie. if I know I've fixed reduce and callable before, I might as well > disable the fixers)? Any other advice to making it run faster? What version of 2to3 are you using? 2to3 got a nice performance boost over the summer which should have made it into Python 3.2. -- Regards, Benjamin From vlada.peric at gmail.com Fri Jun 10 23:56:06 2011 From: vlada.peric at gmail.com (=?UTF-8?Q?Vladimir_Peri=C4=87?=) Date: Fri, 10 Jun 2011 23:56:06 +0200 Subject: [Python-porting] 2to3 speed In-Reply-To: <BANLkTikBwzNvtb22UacdJLsPLTeTuQeJ7Q@mail.gmail.com> References: <BANLkTikBwzNvtb22UacdJLsPLTeTuQeJ7Q@mail.gmail.com> Message-ID: <BANLkTim5vKPa9avM=VWpt5SY+9=3=oLL_w@mail.gmail.com> Thanks to both of you! I am currently using tox for testing/development, but I think I understood from your explanation how to make it all work under Python 3 as well. I've also downloaded the 3.2 version of it and it's about 40% faster, so that was a great help as well! I have encountered an interesting issue, though - one of the libraries SymPy bundles is written to support Python 3 in a single code base (without 2to3). Unfortunately, some of the fixes 2to3 makes bungle this up. Now, editing the library is not a good solution, I'd rather keep it as is - is it possible to tell to 2to3 to skip a part of the code base? And, of course, can I do this from distutils/distribute? -- Vladimir Peri? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-porting/attachments/20110610/124e024e/attachment.html> From regebro at gmail.com Sat Jun 11 00:36:46 2011 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 11 Jun 2011 00:36:46 +0200 Subject: [Python-porting] 2to3 speed In-Reply-To: <BANLkTim5vKPa9avM=VWpt5SY+9=3=oLL_w@mail.gmail.com> References: <BANLkTikBwzNvtb22UacdJLsPLTeTuQeJ7Q@mail.gmail.com> <BANLkTim5vKPa9avM=VWpt5SY+9=3=oLL_w@mail.gmail.com> Message-ID: <BANLkTikqFk=2SiVhkv2tOP2K=NY3wnKu0Q@mail.gmail.com> On Fri, Jun 10, 2011 at 23:56, Vladimir Peri? <vlada.peric at gmail.com> wrote: > Thanks to both of you! I am currently using tox for testing/development, but > I think I understood from your explanation how to make it all work under > Python 3 as well. I've also downloaded the 3.2 version of it and it's about > 40% faster, so that was a great help as well! > > I have encountered an interesting issue, though - one of the libraries SymPy > bundles is written to support Python 3 in a single code base (without 2to3). > Unfortunately, some of the fixes 2to3 makes bungle this up. Now, editing the > library is not a good solution, I'd rather keep it as is - is it possible to > tell to 2to3 to skip a part of the code base? And, of course, can I do this > from distutils/distribute? If those are included in package itself, then there isn't. If they are separate packages it will work with no changes.