[Distribute] Python 3 porting effort - plans proposal for 0.7/0.8
Hello, (Following Ronald's mail about porting setuptools to Py3k (http://mail.python.org/pipermail/distutils-sig/2009-June/012108.html) and Lennart's effort for that too) Since setuptools doesn't exist yet under Python 3, I think we can simplify the "porting to Py 3k" problem by starting a python 3 only version for the Distribute project and flip our porting problems into the other way. Here's my proposal, after 0.6 is released : - Let's start a "0.8" branch right now, by removing all the bootstraping code and by splitting and renaming all elements right away. The setuptools package will be renamed in something else, the pkg_resources.py module and easy_install script too. Those will be three distinct distributions. To use them, people will have to change their import lines, and install_requires lines. - Let's then backport the 0.8 version into a 0.7 version, compatbile with Python 2.x and with the required bootstraping so it works in 2.x environments We might end up with a very ugly 0.7, but who cares this would boosts Python 3 adoption imho Opinions ? Cheers Tarek -- Tarek Ziadé | http://ziade.org
2009/7/23 Tarek Ziadé <ziade.tarek@gmail.com>:
- Let's then backport the 0.8 version into a 0.7 version, compatbile with Python 2.x and with the required bootstraping so it works in 2.x environments
There is no reason to have different version numbers for that. In fact, I think that would only be confusing. The Python 3 version should work for Python 2 as well. Otherwise +1 for everything. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On Thu, Jul 23, 2009 at 4:17 PM, Lennart Regebro<regebro@gmail.com> wrote:
2009/7/23 Tarek Ziadé <ziade.tarek@gmail.com>:
- Let's then backport the 0.8 version into a 0.7 version, compatbile with Python 2.x and with the required bootstraping so it works in 2.x environments
There is no reason to have different version numbers for that. In fact, I think that would only be confusing. The Python 3 version should work for Python 2 as well.
I think you don't see my point : making a pure Python 3 version, without any backward compatibility issues / bootstraping , fight against setuptools distribution. Then backporting it in 0.7 But 0.8 would not be 2.x compatible.
Otherwise +1 for everything. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
-- Tarek Ziadé | http://ziade.org
2009/7/23 Tarek Ziadé <ziade.tarek@gmail.com>:
I think you don't see my point : making a pure Python 3 version, without any backward compatibility issues / bootstraping , fight against setuptools distribution. Then backporting it in 0.7
But 0.8 would not be 2.x compatible.
That would mean we would have two branches of the same code that need to be kept compatible. I don't see why that would be a better idea than having one branch and one version. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On Thu, Jul 23, 2009 at 4:37 PM, Lennart Regebro<regebro@gmail.com> wrote:
2009/7/23 Tarek Ziadé <ziade.tarek@gmail.com>:
I think you don't see my point : making a pure Python 3 version, without any backward compatibility issues / bootstraping , fight against setuptools distribution. Then backporting it in 0.7
But 0.8 would not be 2.x compatible.
That would mean we would have two branches of the same code that need to be kept compatible.
That's exactly how Python 2.X / 3.x are developed. And we expect 0.6 then 0.7 to become maintenance branches at some point.
I don't see why that would be a better idea than having one branch and one version.
Like I said the python 3 branch will be pure python 3 code so we won't have to worry about keeping it compatible with both python 2.x and python 3 That makes the code simpler and cleaner. The maintenance effort between the two branches is not hard as long as there's a maintainer that makes sure every commit in the 0.8 branch is backported, like in Python-dev. I am willing to do this maintenance work. And since Python 3 is a setuptools-free land, we won't have to worry in the 0.8 version about a possible setuptools cohabitation. And people that are using setuptools under 2.x will have less pain to switch to different package names and drop their "import setuptools" infavor of something else, for their 3.x versions. In fact, the sooner such a version is out, the better it could be for Py3 adoption. I don't see any benefit in having a mixed code for 2.x/3.x, but troubles ahead; Tarek -- Tarek Ziadé | http://ziade.org
2009/7/23 Tarek Ziadé <ziade.tarek@gmail.com>:
That's exactly how Python 2.X / 3.x are developed.
Python 2.x and 3.x are two different, incompatible versions of one software. Of course you have different branches. We are to create one compatible version for both Python 2 and Python 3. There is no reason to have two different branches for what is for all intents and purposes the same version and software.
Like I said the python 3 branch will be pure python 3 code so we won't have to worry about keeping it compatible with both python 2.x and python 3
Well, you still need to do that, except that you need to write the code twice, and keep it compatible.
That makes the code simpler and cleaner.
But it more than doubles the effort, as you need to re-implement everything, and also keep everything compatible.
And since Python 3 is a setuptools-free land, we won't have to worry in the 0.8 version about a possible setuptools cohabitation.
No, but we need to do that in 0.7 anyway, so that doesn't actually remove the work.
And people that are using setuptools under 2.x will have less pain to switch to different package names and drop their "import setuptools" infavor of something else, for their 3.x versions.
I don't think we should switch the name for the 3.x version. People need to be able to write code that works under both 2.x and 3.x, and we want to make that as painless as possible. And having different names for the same module under 2.x and 3.x doesn't help with that. What the name is doesn't matter, but it needs to be the same under 2.x and 3.x.
I don't see any benefit in having a mixed code for 2.x/3.x, but troubles ahead;
There are no troubles ahead. You don't have to worry.
The maintenance effort between the two branches is not hard as long as there's a maintainer that makes sure every commit in the 0.8 branch is backported, like in Python-dev. I am willing to do this maintenance work.
This has no significant benefits, creates more work and more confusion. There is simply no reason to do this. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On Thu, Jul 23, 2009 at 5:06 PM, Lennart Regebro<regebro@gmail.com> wrote:
We are to create one compatible version for both Python 2 and Python 3. [..] This has no significant benefits, creates more work and more confusion. There is simply no reason to do this.
Sorry, we are not making any progress in this thread :) you don't seem to understand what backporting changes from a pure 3.x branch to a 2.x compatible branch means. That's just a backport maintenance work once the Py3K branch has started + forwardport of part of the work done in 0.6. So there's no "duplicate work". As a developer, I am ok to have a mixed-code branch for a 0.7 version that will not last, but I don't want to work in a 2.x/3.x code soup for the future 0.8 branch. I gave you the list of benefits and I don't see any benefits in what you are describing. I'd be curious to see how "clean" your 2.x/3.x code could be. How do you intend for instance to have a module with named exceptions that works in both versions, without duplicating the code. This work can be done afterwards, once we have a working system under 3k. Cheers Tarek
2009/7/23 Tarek Ziadé <ziade.tarek@gmail.com>:
you don't seem to understand what backporting changes from a pure 3.x branch to a 2.x compatible branch means. That's just a backport maintenance work once the Py3K branch has started + forwardport of part of the work done in 0.6. So there's no "duplicate work".
You want to reimplement each feature in 0.8 for Python3 in 0.7 for Python 2. That *is* duplicate work.
As a developer, I am ok to have a mixed-code branch for a 0.7 version that will not last, but I don't want to work in a 2.x/3.x code soup for the future 0.8 branch.
There is no 2.x/3.x code soup.
I gave you the list of benefits and I don't see any benefits in what you are describing.
I explained why your benefits do not exist. If you can't see the benefit of having one code set for one version of one software instead of two, then I'm stumped.
I'd be curious to see how "clean" your 2.x/3.x code could be.
You can look at it now. It exists. http://code.google.com/p/python-incompatibility/source/browse/#svn/ports/set...
How do you intend for instance to have a module with named exceptions that works in both versions, without duplicating the code.
Perhaps if you explained why you think it would be necessary to duplicate the code, I could answer. But OK, fine. I don't want to have stupid fights over stupid things. The code exists on the link above. Do whatever you want with it. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On Thu, Jul 23, 2009 at 5:51 PM, Lennart Regebro<regebro@gmail.com> wrote:
You want to reimplement each feature in 0.8 for Python3 in 0.7 for Python 2. That *is* duplicate work.
No, I want to backport it in 2.x syntax. Not reimplement it. That's not code duplication. For example if you make a change in 0.8, I'll backport it but make sure its syntax is correct under 2.x, because 0.7 would be Python 2.x That's all.
I'd be curious to see how "clean" your 2.x/3.x code could be.
You can look at it now. It exists. http://code.google.com/p/python-incompatibility/source/browse/#svn/ports/set...
I am not sure what to look at in there, do you have a place in there that is based on setuptools trunk ? Does it work with 2.x and 3.x at the same time ?
How do you intend for instance to have a module with named exceptions that works in both versions, without duplicating the code.
Perhaps if you explained why you think it would be necessary to duplicate the code, I could answer.
For all the backwards incompatibility Python 3 introduced. Here's a small and incomplete list of points : 1/ import statements like this : try: import something # python 2 except ImportError: import somethingelse # python 3 2/ named exceptions (I don't think setuptools have some, but we might need them at some point) except Something, e: vs except Something as e: 3/ bytes vs string issues If we use for instance subprocess.Popen, it return bytes, no strings, which has side effects on the consumers. If we use regular expressions we have to make a difference between string matching and byte matching etc.. 4/ etc... So I am not sure : do you want to share the same code base for 2.x and 3.x and have in the code things like: (in pseudo code) if python 2: python 2. code if python 3: python 3 code I don't want that. What I am calling a "clean" code is a code base that doesn't deal with two incompatible versions of Python eg having a branch for 2.x and one for 3.x instead. And when something is done on 3.x, it's backported wrt the 2.X syntax. So I think the best strategy is to run 2to3 and finish the work on the code to make it pure Python3, then to split the code in several distributions and release it for Python 3 usage.
But OK, fine. I don't want to have stupid fights over stupid things. The code exists on the link above. Do whatever you want with it.
That's not stupid things, and that's not a fight. I don't understand why you are over-reacting like that. I won't do anything with this existing code if you're not helping ;) But It's quite important imho to have the best strategy for next releases, and I can't picture the same branch with bits of 2.x and 3.x syntax in it. I've been backporting and forwardporting distutils between python 2 and 3 for 6+ months now, and I can tell it's not a lot of work, and both sides are clean and concise because they don't try to work for 2.x and 3.x at the same time. Regards Tarek
2009/7/23 Tarek Ziadé <ziade.tarek@gmail.com>:
I am not sure what to look at in there, do you have a place in there that is based on setuptools trunk ? Does it work with 2.x and 3.x at the same time ?
Yes, via 2to3 which for good reasons are the official way of supporting both Python 2.x and 3.x at the same time.
For all the backwards incompatibility Python 3 introduced.
OK, then I understand the question. The answer is: 2to3. Having two branches is still a terrible idea, which just gives the developers more work, and creates lots of code duplication. I understand that this is not how it's done with Python itself, as mentioned, but this is not Python itself. I understand you arguments that it's not "much work". But it's still work that is not necessary.
But It's quite important imho to have the best strategy for next releases, and I can't picture the same branch with bits of 2.x and 3.x syntax in it.
You don't have that. You write for Python 2, and convert it with 2to3. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On Thu, Jul 23, 2009 at 8:56 PM, Lennart Regebro<regebro@gmail.com> wrote:
But It's quite important imho to have the best strategy for next releases, and I can't picture the same branch with bits of 2.x and 3.x syntax in it.
You don't have that. You write for Python 2, and convert it with 2to3.
+1, that's the official way and we should stick to that. And at some point we hopefully get the 3to2 tool and can switch the process around with the 3.x code being the canonical one. Hanno
On Thursday,2009-07-23, at 12:56 , Lennart Regebro wrote:
You don't have that. You write for Python 2, and convert it with 2to3.
+1 I will hopefully contribute some patches/testing/etc. for Distribute, but I have no plans to start using Python 3 in the forseeable future. I will of course learn what I need to know in order to make sure my patches don't cause problems for Python 3 users, but supporting Python 2 is a chief requirement for me, and so I prefer to develop with Python 2. Regards, Zooko
On Thu, Jul 23, 2009 at 8:56 PM, Lennart Regebro<regebro@gmail.com> wrote:
2009/7/23 Tarek Ziadé <ziade.tarek@gmail.com>:
I am not sure what to look at in there, do you have a place in there that is based on setuptools trunk ? Does it work with 2.x and 3.x at the same time ?
Yes, via 2to3 which for good reasons are the official way of supporting both Python 2.x and 3.x at the same time.
Sorry I am not sure to understand, you want to run 2to3 live ? If not there will be two branches nevertheless, even if the second one was generated via 2to3. Or are you thinking about generated the 3k code at release time for the 3.x version ? What would be the precise process ?
2009/7/23 Tarek Ziadé <ziade.tarek@gmail.com>:
Or are you thinking about generated the 3k code at release time for the 3.x version ?
Either release time or install time (although install time is tricky, as setuptools currently depends on itself, so release time is the way to go at least before a refactoring). -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On Thu, Jul 23, 2009 at 10:37 PM, Lennart Regebro<regebro@gmail.com> wrote:
2009/7/23 Tarek Ziadé <ziade.tarek@gmail.com>:
Or are you thinking about generated the 3k code at release time for the 3.x version ?
Either release time or install time (although install time is tricky, as setuptools currently depends on itself, so release time is the way to go at least before a refactoring).
Ok I'm convinced now :) Having this for Distribute 0.6 could make Py3k developer switch on it very soon but on the other hand, it wuld force them to change their imports by the time 0.7 is out. OTOH, if they know it, they would probably be grateful to have something working today under 3.x So what is the work left to be done so it works fully for our code base ?
2009/7/23 Tarek Ziadé <ziade.tarek@gmail.com>:
So what is the work left to be done so it works fully for our code base ?
Making a diff of the work done of the google code repo, and getting it into the hg repository. It's not that much work, but as noted I failed (in several ways) last time. :-/ -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On Thu, Jul 23, 2009 at 10:53 PM, Lennart Regebro<regebro@gmail.com> wrote:
2009/7/23 Tarek Ziadé <ziade.tarek@gmail.com>:
So what is the work left to be done so it works fully for our code base ?
Making a diff of the work done of the google code repo, and getting it into the hg repository. It's not that much work, but as noted I failed (in several ways) last time. :-/
I can help for the hg part, So how this will look at the end ? a conversion function that can be called when releasing for instance ?
-- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
-- Tarek Ziadé | http://ziade.org
2009/7/23 Tarek Ziadé <ziade.tarek@gmail.com>:
So how this will look at the end ? a conversion function that can be called when releasing for instance ?
Something like that. There is a port.sh script that will make a port (from a copy in /tmp/) and I used that to do the Python 3 tgz's before if I remember correctly. There are other options. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On Thu, Jul 23, 2009 at 11:59 PM, Lennart Regebro<regebro@gmail.com> wrote:
2009/7/23 Tarek Ziadé <ziade.tarek@gmail.com>:
So how this will look at the end ? a conversion function that can be called when releasing for instance ?
Something like that. There is a port.sh script that will make a port (from a copy in /tmp/) and I used that to do the Python 3 tgz's before if I remember correctly. There are other options.
Ok then , what about working on that script within the default branch, I don't think it interfers at all with the rest, I have a release.sh script that creates all .egg and the .tar.gz, You could add your things in a separate directory in the repo and hook it in release.sh and possibly in setup.py, so it builds for python 3, If everyone agrees and if you have time to do this, I can postpone the 0.6 release for a week or so, until this is ready, so we can have 0.6 for Python 3. In any case I am going for a week of vacation starting tomorrow, so the 0.6 will not be released before august the 3rd, Tarek -- Tarek Ziadé | http://ziade.org
2009/7/24 Tarek Ziadé <ziade.tarek@gmail.com>:
I can postpone the 0.6 release for a week or so, until this is ready, so we can have 0.6 for Python 3.
I think we should make a 2.x only release first. The changes to support Python 3 are small but numerous and since the test-coverage isn't that great I'm worried it will insert subtle bugs. I would be less worried if we can have a longer test-period after these changes. On the other hand, if we do merge these changes before the 0.6 release they will quickly get tested. ;) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On Fri, Jul 24, 2009 at 3:29 PM, Lennart Regebro<regebro@gmail.com> wrote:
2009/7/24 Tarek Ziadé <ziade.tarek@gmail.com>:
I can postpone the 0.6 release for a week or so, until this is ready, so we can have 0.6 for Python 3.
I think we should make a 2.x only release first. The changes to support Python 3 are small but numerous and since the test-coverage isn't that great I'm worried it will insert subtle bugs. I would be less worried if we can have a longer test-period after these changes.
On the other hand, if we do merge these changes before the 0.6 release they will quickly get tested. ;)
Right, I am +1 for adding it in 0.6, and having a 0.6.1 for thoses fixes,
On Jul 24, 2009, at 10:29 AM, Lennart Regebro wrote:
2009/7/24 Tarek Ziadé <ziade.tarek@gmail.com>:
I can postpone the 0.6 release for a week or so, until this is ready, so we can have 0.6 for Python 3.
I think we should make a 2.x only release first. The changes to support Python 3 are small but numerous and since the test-coverage isn't that great I'm worried it will insert subtle bugs. I would be less worried if we can have a longer test-period after these changes.
On the other hand, if we do merge these changes before the 0.6 release they will quickly get tested. ;)
People using python 3x don't have support for setuptools so they would understand if you release early a beta version. And I think you should, not having setutools is a problem for every package being converted to python 3.1 -- Leonardo Santagada santagada at gmail.com
On Friday, 24 July, 2009, at 03:59PM, "Leonardo Santagada" <santagada@gmail.com> wrote:
On Jul 24, 2009, at 10:29 AM, Lennart Regebro wrote:
2009/7/24 Tarek Ziadé <ziade.tarek@gmail.com>:
I can postpone the 0.6 release for a week or so, until this is ready, so we can have 0.6 for Python 3.
I think we should make a 2.x only release first. The changes to support Python 3 are small but numerous and since the test-coverage isn't that great I'm worried it will insert subtle bugs. I would be less worried if we can have a longer test-period after these changes.
On the other hand, if we do merge these changes before the 0.6 release they will quickly get tested. ;)
People using python 3x don't have support for setuptools so they would understand if you release early a beta version. And I think you should, not having setutools is a problem for every package being converted to python 3.1
The issue with merging python3-related changes is not that the python3 port would have a beta-status, but that there is a risk that this will accidently break python2 support. I agree with Lennart that a 2.x only release would be better, especially because it would be possible to do a 0.7 alpha/beta release short after the stable 0.6 release. Ronald
-- Leonardo Santagada santagada at gmail.com
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Fri, Jul 24, 2009 at 4:06 PM, Ronald Oussoren<ronaldoussoren@mac.com> wrote:
I agree with Lennart that a 2.x only release would be better, especially because it would be possible to do a 0.7 alpha/beta release short after the stable 0.6 release.
Notice that 0.7 will rename the setuptools package, the pkg_resources.py module, and the easy_install script Meaning that if we don't add py3 support in the 0.6.x series, people will *have* to rename their imports if they want to use it under Python 3. Tarek
2009/7/24 Tarek Ziadé <ziade.tarek@gmail.com>:
On Fri, Jul 24, 2009 at 4:06 PM, Ronald Oussoren<ronaldoussoren@mac.com> wrote:
I agree with Lennart that a 2.x only release would be better, especially because it would be possible to do a 0.7 alpha/beta release short after the stable 0.6 release.
Notice that 0.7 will rename the setuptools package, the pkg_resources.py module, and the easy_install script
Meaning that if we don't add py3 support in the 0.6.x series, people will *have* to rename their imports if they want to use it under Python 3.
Yeah, but is that a problem, as the renamed version will exist for Python 2 as well, I assume? In any case, I have no strong opinion on whether 3.x support comes in 0.6 or 0.7. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On Fri, Jul 24, 2009 at 4:33 PM, Lennart Regebro<regebro@gmail.com> wrote:
2009/7/24 Tarek Ziadé <ziade.tarek@gmail.com>:
On Fri, Jul 24, 2009 at 4:06 PM, Ronald Oussoren<ronaldoussoren@mac.com> wrote:
I agree with Lennart that a 2.x only release would be better, especially because it would be possible to do a 0.7 alpha/beta release short after the stable 0.6 release.
Notice that 0.7 will rename the setuptools package, the pkg_resources.py module, and the easy_install script
Meaning that if we don't add py3 support in the 0.6.x series, people will *have* to rename their imports if they want to use it under Python 3.
Yeah, but is that a problem, as the renamed version will exist for Python 2 as well, I assume?
no, it just changes the constraints : 0.6 : python 2 + no renaming 0.7 : python 2 or 3 + renaming or 0.6 : python 2 or 3 + no renaming 0.7 : python 2 or 3 + renaming
In any case, I have no strong opinion on whether 3.x support comes in 0.6 or 0.7.
me neither, so let's drop that for 0.6. The work left for 0.6 is: - writing the zc.buildout bootstrap with a patch (zc.buildout.buildout.boostrap needs to be patched :() - finish the tests under various environments (I didn't test too much under win32) (help welcome :)) So the release should be pushed around the 5th or 6th,
2009/7/24 Stephen Waterbury <waterbug@pangalactic.us>:
Lennart Regebro wrote:
On the other hand, if we do merge these changes before the 0.6 release they will quickly get tested. ;)
Ah, yes: the Microsoft Strategy. ;)
:-) It's also a common OS strategy, for the simple reason that few people seem bothered to test the betas... -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On 23 Jul, 2009, at 16:29, Tarek Ziadé wrote:
On Thu, Jul 23, 2009 at 4:17 PM, Lennart Regebro<regebro@gmail.com> wrote:
2009/7/23 Tarek Ziadé <ziade.tarek@gmail.com>:
- Let's then backport the 0.8 version into a 0.7 version, compatbile with Python 2.x and with the required bootstraping so it works in 2.x environments
There is no reason to have different version numbers for that. In fact, I think that would only be confusing. The Python 3 version should work for Python 2 as well.
I think you don't see my point : making a pure Python 3 version, without any backward compatibility issues / bootstraping , fight against setuptools distribution. Then backporting it in 0.7
But 0.8 would not be 2.x compatible.
At PyCon09 there was some talk about a "3to2" tool, specially for issue: allow package maintainers to work in Python 3 and automaticly generate a Python 2 version. I have no idea if anyone is actually working on such a tool though. Ronald P.S. Sorry about the late answers, I've been travelling and was offline for a while.
participants (7)
-
Hanno Schlichting -
Lennart Regebro -
Leonardo Santagada -
Ronald Oussoren -
Stephen Waterbury -
Tarek Ziadé -
Zooko Wilcox-O'Hearn