Packaging in Python 2 anyone ?

Hi all, I am lacking of time right now to finish an important task before 3.2 final is out: we need to release "packaging" as a standalone release under Python 2.x and Python 3.1, to gather as much feedback as we can from more people. Doing an automated conversion turned out to be a nightmare, and I was about to go ahead and maintain a fork of the packaging package, with the few modules that are needed (sysconfig, etc) within a standalone release. I am looking for someone that has some free time and that is willing to lead this work. 3.2 can go out without this work of course, but it would be *much* better to have that feedback If you are interested, please let me know. Cheers Tarek -- Tarek Ziadé | http://ziade.org

On Sun, Aug 14, 2011 at 7:41 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
If anyone else got at all confused by Tarek's email, s/3.x/3.x+1/ and it will all make sense (the mentioned release numbers in the 3.x series are all one lower than they should be - packaging is planned for 3.3, but a standalone library will allow feedback to be gathered from 2.x and 3.2 users before the API is 'locked in' for 3.3). How far has packaging diverged from distutils2, though? Wasn't that the planned venue for any backports in order to avoid name conflicts? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Aug 15, 2011 at 1:20 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Oh yeah sorry for the version mess up :)
How far has packaging diverged from distutils2, though? Wasn't that the planned venue for any backports in order to avoid name conflicts?
The plan is to provide under earlier versions of Python a standalone project that does not use the "packaging" namespace, but the "distutils2" namespace. IOW, the task to do is: 1/ copy packaging and all its stdlib dependencies in a standalone project 2/ rename packaging to distutils2 3/ make it work under older 2.x and 3.x (2.x would be the priority) <==== 4/ release it, promote its usage 5/ consolidate the API with the feedback received I realize it's by far the less interesting task to do in packaging, but it's by far one of the most important Cheers Tarek -- Tarek Ziadé | http://ziade.org

Tarek Ziadé <ziade.tarek <at> gmail.com> writes:
Okay, I had a bit of spare time today, and here's as far as I've got: Step 1 - done. Step 2 - done. Step 3 - On Python 2.6 most of the tests pass: Ran 322 tests in 12.148s FAILED (failures=3, errors=4, skipped=39) See the detailed test results (for Linux) at https://gist.github.com/1152791 The code is at https://bitbucket.org/vinay.sajip/du2/ stdlib dependency code is either moved to util.py or test/support.py as appropriate. You need to test in a virtualenv with unittest2 installed. No work has been done on packaging the project. I'm not sure if I'll have much more time to spend on this, but it's there in case someone else can look at the remaining test failures, plus Steps 4 and 5; hopefully I've broken the back of it :-) Regards, Vinay Sajip

On Wed, Aug 17, 2011 at 9:15 PM, Chris McDonough <chrism@plope.com> wrote:
I'll throw this out there.. why is it going to have a different name on python2 than on python3?
So it can be a drop-in replacement for the existing distutils2, I'd expect. "packaging" is new with Python3, and is the Guido-approved name. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> "A person who won't read has no advantage over one who can't read." --Samuel Langhorne Clemens

On Thu, Aug 18, 2011 at 12:15 PM, Fred Drake <fdrake@acm.org> wrote:
It's actually for the same reason that unittest changes are backported under the unittest2 name - the distutils2 name can be used in the future to get Python 3.4 packaging features in Python 3.3, but that would be difficult if the backport shadowed the standard library name. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Aug 17, 2011 at 11:00 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Ah, yes... the old "too bad we stuck it in the standard library" problem. For some things, an easy lament, but for foundational packaging-related things, it's hard to get around. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> "A person who won't read has no advantage over one who can't read." --Samuel Langhorne Clemens

On Thu, Aug 18, 2011 at 5:17 AM, Fred Drake <fdrake@acm.org> wrote:
Yeah exactly. And the good thing about packaging and distutils2 is that for the regular usage (package your project) you don't type any code, just define options in setup.cfg. IOW there's no "import packaging" or "import distutils2". Cheers Tarek
-- Tarek Ziadé | http://ziade.org

On Thu, Aug 18, 2011 at 12:30 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote: ...
Okay, I had a bit of spare time today, and here's as far as I've got:
Awesome, thanks a lot !
Thank you very much ! Ideally, if you could push this to hg.python.org/distutils2 (overwriting the existing stuff). Cheers Tarek -- Tarek Ziadé | http://ziade.org

Tarek Ziadé <ziade.tarek <at> gmail.com> writes:
Ideally, if you could push this to hg.python.org/distutils2 (overwriting the existing stuff).
Okay, done. I've overwritten existing files and added new ones, only removing/renaming things like index -> pypi and mkcfg -> create. I haven't touched existing code e.g. the top-level test scripts or the _backport directory. The added test_distutils2.py is what I used to run the tests. Regards, Vinay Sajip

On Thu, Aug 18, 2011 at 11:26 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Frankly, I think there's no best moment for this. We'll need to backport everything we do in packaging/ in distutils2/ (Yeah, painful...)
-- Tarek Ziadé | http://ziade.org

Le 18/08/2011 18:19, Vinay Sajip a écrit :
I will; any help is welcome, especially if you have a machine with the same Windows version (see #12678). I caught Georg’s message on python-committers but could not do anything in time; I only have Internet access at a public library so I can’t be as responsive as I would.
Yes, there are a few dozen bugs that need addressing before 1.0 (i.e. Python 3.3), but there’s time. Alpha and beta releases of distutils2 would be useful. Regards

On Thu, Aug 18, 2011 at 11:16 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Thanks again
-- Tarek Ziadé | http://ziade.org

Hi Tarek,
Can you give us more info? Do you have a repo somewhere, or notes? A related question: what is the minimum 2.x version that we should support? 2.6 would be a dream, thanks to bytes literal and all that, but I’m sure it’s not realistic; 2.5 would be nice for the with statement and hashlib, otherwise 2.4 is okay. When I talked with Łukasz in private email about backports and 3to2, we agreed that there were some serious bugs in 3to2 and we wanted to work on patches. I also wanted to make the command-line driver more flexible, so that it would be easy to run a command to apply only 3.3→3.2 fixes, then another for 3.2→2.7, etc. Maybe your problems were caused by the state of the packaging codebase. The conversion to 3.x was a little messy: in some cases there were parallel code branches for 2.x and 3.x, on other cases 2to3 was run, and in many cases the conversion had to be cleaned up (esp. bytes/str madness). Even now that the code runs and the tests pass, there may still be things in need of a cleanup in the codebase, and maybe they trip up 3to2.
I am looking for someone that has some free time and that is willing to lead this work.
Well, free time is scarce with all these distutils bugs on my plate, but I am definitely interested in heading the backport, as I stated earlier. I think the key point is to avoid making the same work over and over again, and I see a few ways of managing that. The first way is to start with a 2.x-converted codebase (thanks Vinay!) and manually port all cpython/packaging changesets to distutils2, like I used to do. This is just as annoying as backporting to 2.7, and just as simple. The second way is to work on a conversion tool instead of working on changesets. The idea is to make a robust tool based on 3to2 that copies code and converts it. This would not be the easiest way, as shown by your experience, but surely the less cumbersome in the long term. The third way is to use a new Mercurial repo converted from the cpython repo, so that we can run “hg convert” again to pull new changesets. Convert, test and commit. The advantage is that it’s not required to port each changeset: the convert-merge dance can be done once a month, or just for new releases. The fourth way is hybrid: start from a 2.x-converted codebase, and each month, make a diff for cpython/Lib/packaging and apply to distutils2. I fear that such diffs would be painful to apply, and consist mostly of rejects. With idea #3, we get to use a merge tool, which is much better. After writing out these ideas, I think the first one is certainly the simplest thing that could work with minimum pain. Le 18/08/2011 00:30, Vinay Sajip a écrit : put in util.py.
Regards

On Thu, Aug 18, 2011 at 8:27 PM, Éric Araujo <merwok@netwok.org> wrote:
I tried using relative imports, but that made the whole thing complicated and not working under older 2.x then there are a lot of spots where the word 'packaging' is used for other things than modules. then there are spots when we needed to change the bytes/str behavior depending on the py version, making everything complex to maintain I guess it's the addition of the three that made it too complex : transparent renaming + 3to2 + 3.xto3.x
2.5 sounds good. I am sold on dropping 2.4 frankly. Maybe we can drop 2.5 in a few months ;)
I think that's not worth the effort frankly. keeping a clean fully py3 code without worrying about making it 3to2 friendly, make all contributors life easier ihmo. The tradeoff is that we will have to backport to distutils2 changes. That's what was done for a while between the Python trunk and the Py3k branch, so I guess it's doable -- if all packaging contributors agree to do this backport work.
I think so too. The automatic conversion sounded like a great thing, but the nature of the project makes it too hard, Cheers -- Tarek Ziadé | http://ziade.org

Hi, Here’s a status update on distutils2. Vinay did the bulk of the work in his initial commit; we just had to re-add some mistakenly deleted helpers in d2.tests and d2.tests.support, change sysconfig imports and remove duplicate files (sysconfig.*). A contributor did a huge commit to restore 2.4 compatibility. I pulled it, because it was a useful contribution, and am now in the middle of cleaning it: some conversions were not idiomatic or even buggy, just like when we converted from 2.x to 3.x. Alexis and I have been working in parallel, with some unfortunate duplication. We’ve resolved to use the tracker or email to coordinate. When I am finished cleaning up the 2.4 compat changes, I’ll backport all outstanding changesets that were done in packaging, and then I’ll try to fix the few (on linux3^Wlinux) test failures. When the d2 codebase matches packaging's again, it will be easy to keep both codebases in sync. I will edit the wiki page about contributing to state that I will accept patches made against d2 instead of packaging, to lower the contribution bar. It would be very useful to have buildbots. A question: What about distutils2 for Python 3.x? I think we could keep the stdlib codebase compatible with 3.1 and use a semi-automated process to extract cpython3.3/Lib/packaging to distutils2-py3/distutils2 and rename imports. (IIRC PyPI will require us to play games to have both 2.x and 3.x versions of distutils2.) Another question: What about the docs? Can we just point people to docs.python.org and tell them to mentally replace packaging with distutils2? If that is judged unacceptable, then I’ll synchronize the docs in the d2 repo, but that’s hours I won’t spend on bugs or features. Cheers

On 13/09/2011 16:57, Éric Araujo wrote:
What I'm doing for unittest2. 1) I have a script that pulls unittest from mercurial head and then applies patches to it to make it compatible with Python 3.1 - 3.2 and rename it from unittest to unittest2 2) I have a pypi project called unittestpy3k that holds the Python 3 version of unittest2 Projects using unittest2 for Python 3 then have a dependency on unittest2py3k - but the actual Python package name is unittest2. I only need to maintain a set of patches against unittest on head, rather than a whole branch. This works pretty well. All the best, Michael Foord
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Le 13/09/2011 18:34, Michael Foord a écrit :
That’s what I call playing games. I think it would make more sense to push 2.x-compatible and 3.x-compatible sdists to PyPI (with an appropriate 'Programming Language :: Python :: 2' or '3' classifier) and have the download tools be smart. Regards

On Thu, Sep 15, 2011 at 12:23 PM, Éric Araujo <merwok@netwok.org> wrote:
FWIW, I prefer this as well. I'd certainly appreciate the option to do it this way. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> "A person who won't read has no advantage over one who can't read." --Samuel Langhorne Clemens

On 15/09/2011 17:23, Éric Araujo wrote:
Hah, sure. In the meantime my way works *now* and with the existing tools. :-) (But only actually true for the way I make it available from pypi - the rest of the technique is not "playing games", right?) Yes, I would prefer to have a single project name with different distributions for Python 2 and 3 (and I looked into it) - but with the current tools the only way to achieve that is to put both versions into a single distribution. This prevents you from versioning them separately and is a pain to do anyway if the different versions are in different repos. The current tools are a real pain for versioning anyway. If your pypi page even *links* to a page that offers an alpha or beta (in development version) for download then both pip and easy_install will fetch that, in preference to the most recent version on pypi. So yes, I agree there is room for improvement in the current tools. Hopefully distutils2 will fix that. ;-) All the best, Michael Foord
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

On 15/09/2011 19:31, Michael Foord wrote:
I'm pretty sure recent releases of zc.buildout prefer "final" releases by default ;-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

Hi, I caught Tarek on IRC and forced him to answer my questions. Here are the latest news: - I have cleaned up and synchronized the distutils2 codebase with packaging in 3.3. All features and bugs are now identical. The test suite runs with Python 2.4 to 2.7; there are three or four test failures (linux, with threads, UCS4, not shared). Please clone, build (we backported hashlib for 2.4), test and file bugs! We’ll make an alpha4 as soon as all tests pass. - I have started work in a named branch to provide distutils2 for Python 3.1 and 3.2. Patches will flow between packaging, distutils2 and distutils2-py3. I’ll start a discussion on catalog-sig to improve support of parallel releases of 2.x and 3.x-compatible projects. - The docs in the d2 repo will be removed; people will go to docs.python.org and mentally convert packaging to distutils2. I’ll update the PyPI page. Cheers

Éric Araujo <merwok <at> netwok.org> writes:
Well sysconfig.py/sysconfig.cfg have been copied as is. I've only copied over specific things we need from shutil/functools/os, etc. so far to util.py. I haven't looked at 2.4/2.5 support yet: things like hashlib would probably need to be treated the same way Django handles this sort of backport of functionality.
I join my thanks to Tarek’s, and volunteer to follow on :)
That's good news :-) Regards, Vinay Sajip

On 15 August 2011 11:31, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
One thing that I, as a semi-interested bystander, would like to see is sort of a component of 4. Namely, a document somewhere addressing the question of why I, as a current user of distutils (setup.py, etc), should convert my project to use packaging/distutils2 - and what I'd need to do so. At the moment, I see no benefit to me in migrating. New projects, or projects that already know that they want one or more of the benefits that packaging/distutils2/setuptools bring, are a different matter. It's projects with needs satisfied by distutils, and code invested a distutils-based solution, that could do with some persuasion. I checked the docs, and "Distributing Python Modules" is for new projects, and "What's new" basically says "we expect you to migrate" but has no reasons or guidelines. If someone borrows the time machine and makes this already available, so much the better. Pointers would be appreciated! Paul.

I’m working on such a document, first in a doc set outside of the Python docs, and when it’s ready as an official HOWTO. (I’ll send the URL when I finish and publish it.)
I checked the docs, and "Distributing Python Modules" is for new projects,
That doc set is for distutils, unless you meant “Distributing Python Projects”, which is currently under severe updating. Regards

On Sun, Aug 14, 2011 at 7:41 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
If anyone else got at all confused by Tarek's email, s/3.x/3.x+1/ and it will all make sense (the mentioned release numbers in the 3.x series are all one lower than they should be - packaging is planned for 3.3, but a standalone library will allow feedback to be gathered from 2.x and 3.2 users before the API is 'locked in' for 3.3). How far has packaging diverged from distutils2, though? Wasn't that the planned venue for any backports in order to avoid name conflicts? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Aug 15, 2011 at 1:20 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Oh yeah sorry for the version mess up :)
How far has packaging diverged from distutils2, though? Wasn't that the planned venue for any backports in order to avoid name conflicts?
The plan is to provide under earlier versions of Python a standalone project that does not use the "packaging" namespace, but the "distutils2" namespace. IOW, the task to do is: 1/ copy packaging and all its stdlib dependencies in a standalone project 2/ rename packaging to distutils2 3/ make it work under older 2.x and 3.x (2.x would be the priority) <==== 4/ release it, promote its usage 5/ consolidate the API with the feedback received I realize it's by far the less interesting task to do in packaging, but it's by far one of the most important Cheers Tarek -- Tarek Ziadé | http://ziade.org

Tarek Ziadé <ziade.tarek <at> gmail.com> writes:
Okay, I had a bit of spare time today, and here's as far as I've got: Step 1 - done. Step 2 - done. Step 3 - On Python 2.6 most of the tests pass: Ran 322 tests in 12.148s FAILED (failures=3, errors=4, skipped=39) See the detailed test results (for Linux) at https://gist.github.com/1152791 The code is at https://bitbucket.org/vinay.sajip/du2/ stdlib dependency code is either moved to util.py or test/support.py as appropriate. You need to test in a virtualenv with unittest2 installed. No work has been done on packaging the project. I'm not sure if I'll have much more time to spend on this, but it's there in case someone else can look at the remaining test failures, plus Steps 4 and 5; hopefully I've broken the back of it :-) Regards, Vinay Sajip

On Wed, Aug 17, 2011 at 9:15 PM, Chris McDonough <chrism@plope.com> wrote:
I'll throw this out there.. why is it going to have a different name on python2 than on python3?
So it can be a drop-in replacement for the existing distutils2, I'd expect. "packaging" is new with Python3, and is the Guido-approved name. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> "A person who won't read has no advantage over one who can't read." --Samuel Langhorne Clemens

On Thu, Aug 18, 2011 at 12:15 PM, Fred Drake <fdrake@acm.org> wrote:
It's actually for the same reason that unittest changes are backported under the unittest2 name - the distutils2 name can be used in the future to get Python 3.4 packaging features in Python 3.3, but that would be difficult if the backport shadowed the standard library name. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Aug 17, 2011 at 11:00 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Ah, yes... the old "too bad we stuck it in the standard library" problem. For some things, an easy lament, but for foundational packaging-related things, it's hard to get around. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> "A person who won't read has no advantage over one who can't read." --Samuel Langhorne Clemens

On Thu, Aug 18, 2011 at 5:17 AM, Fred Drake <fdrake@acm.org> wrote:
Yeah exactly. And the good thing about packaging and distutils2 is that for the regular usage (package your project) you don't type any code, just define options in setup.cfg. IOW there's no "import packaging" or "import distutils2". Cheers Tarek
-- Tarek Ziadé | http://ziade.org

On Thu, Aug 18, 2011 at 12:30 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote: ...
Okay, I had a bit of spare time today, and here's as far as I've got:
Awesome, thanks a lot !
Thank you very much ! Ideally, if you could push this to hg.python.org/distutils2 (overwriting the existing stuff). Cheers Tarek -- Tarek Ziadé | http://ziade.org

Tarek Ziadé <ziade.tarek <at> gmail.com> writes:
Ideally, if you could push this to hg.python.org/distutils2 (overwriting the existing stuff).
Okay, done. I've overwritten existing files and added new ones, only removing/renaming things like index -> pypi and mkcfg -> create. I haven't touched existing code e.g. the top-level test scripts or the _backport directory. The added test_distutils2.py is what I used to run the tests. Regards, Vinay Sajip

On Thu, Aug 18, 2011 at 11:26 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Frankly, I think there's no best moment for this. We'll need to backport everything we do in packaging/ in distutils2/ (Yeah, painful...)
-- Tarek Ziadé | http://ziade.org

Antoine Pitrou <solipsis <at> pitrou.net> writes:
Plus, there are at least half a dozen issues which would need to be addressed in packaging before final release, but they are not complete show-stoppers and won't preclude 2.x users giving useful feedback. Regards, Vinay Sajip

Le 18/08/2011 18:19, Vinay Sajip a écrit :
I will; any help is welcome, especially if you have a machine with the same Windows version (see #12678). I caught Georg’s message on python-committers but could not do anything in time; I only have Internet access at a public library so I can’t be as responsive as I would.
Yes, there are a few dozen bugs that need addressing before 1.0 (i.e. Python 3.3), but there’s time. Alpha and beta releases of distutils2 would be useful. Regards

On Thu, Aug 18, 2011 at 11:16 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Thanks again
-- Tarek Ziadé | http://ziade.org

Hi Tarek,
Can you give us more info? Do you have a repo somewhere, or notes? A related question: what is the minimum 2.x version that we should support? 2.6 would be a dream, thanks to bytes literal and all that, but I’m sure it’s not realistic; 2.5 would be nice for the with statement and hashlib, otherwise 2.4 is okay. When I talked with Łukasz in private email about backports and 3to2, we agreed that there were some serious bugs in 3to2 and we wanted to work on patches. I also wanted to make the command-line driver more flexible, so that it would be easy to run a command to apply only 3.3→3.2 fixes, then another for 3.2→2.7, etc. Maybe your problems were caused by the state of the packaging codebase. The conversion to 3.x was a little messy: in some cases there were parallel code branches for 2.x and 3.x, on other cases 2to3 was run, and in many cases the conversion had to be cleaned up (esp. bytes/str madness). Even now that the code runs and the tests pass, there may still be things in need of a cleanup in the codebase, and maybe they trip up 3to2.
I am looking for someone that has some free time and that is willing to lead this work.
Well, free time is scarce with all these distutils bugs on my plate, but I am definitely interested in heading the backport, as I stated earlier. I think the key point is to avoid making the same work over and over again, and I see a few ways of managing that. The first way is to start with a 2.x-converted codebase (thanks Vinay!) and manually port all cpython/packaging changesets to distutils2, like I used to do. This is just as annoying as backporting to 2.7, and just as simple. The second way is to work on a conversion tool instead of working on changesets. The idea is to make a robust tool based on 3to2 that copies code and converts it. This would not be the easiest way, as shown by your experience, but surely the less cumbersome in the long term. The third way is to use a new Mercurial repo converted from the cpython repo, so that we can run “hg convert” again to pull new changesets. Convert, test and commit. The advantage is that it’s not required to port each changeset: the convert-merge dance can be done once a month, or just for new releases. The fourth way is hybrid: start from a 2.x-converted codebase, and each month, make a diff for cpython/Lib/packaging and apply to distutils2. I fear that such diffs would be painful to apply, and consist mostly of rejects. With idea #3, we get to use a merge tool, which is much better. After writing out these ideas, I think the first one is certainly the simplest thing that could work with minimum pain. Le 18/08/2011 00:30, Vinay Sajip a écrit : put in util.py.
Regards

On Thu, Aug 18, 2011 at 8:27 PM, Éric Araujo <merwok@netwok.org> wrote:
I tried using relative imports, but that made the whole thing complicated and not working under older 2.x then there are a lot of spots where the word 'packaging' is used for other things than modules. then there are spots when we needed to change the bytes/str behavior depending on the py version, making everything complex to maintain I guess it's the addition of the three that made it too complex : transparent renaming + 3to2 + 3.xto3.x
2.5 sounds good. I am sold on dropping 2.4 frankly. Maybe we can drop 2.5 in a few months ;)
I think that's not worth the effort frankly. keeping a clean fully py3 code without worrying about making it 3to2 friendly, make all contributors life easier ihmo. The tradeoff is that we will have to backport to distutils2 changes. That's what was done for a while between the Python trunk and the Py3k branch, so I guess it's doable -- if all packaging contributors agree to do this backport work.
I think so too. The automatic conversion sounded like a great thing, but the nature of the project makes it too hard, Cheers -- Tarek Ziadé | http://ziade.org

Hi, Here’s a status update on distutils2. Vinay did the bulk of the work in his initial commit; we just had to re-add some mistakenly deleted helpers in d2.tests and d2.tests.support, change sysconfig imports and remove duplicate files (sysconfig.*). A contributor did a huge commit to restore 2.4 compatibility. I pulled it, because it was a useful contribution, and am now in the middle of cleaning it: some conversions were not idiomatic or even buggy, just like when we converted from 2.x to 3.x. Alexis and I have been working in parallel, with some unfortunate duplication. We’ve resolved to use the tracker or email to coordinate. When I am finished cleaning up the 2.4 compat changes, I’ll backport all outstanding changesets that were done in packaging, and then I’ll try to fix the few (on linux3^Wlinux) test failures. When the d2 codebase matches packaging's again, it will be easy to keep both codebases in sync. I will edit the wiki page about contributing to state that I will accept patches made against d2 instead of packaging, to lower the contribution bar. It would be very useful to have buildbots. A question: What about distutils2 for Python 3.x? I think we could keep the stdlib codebase compatible with 3.1 and use a semi-automated process to extract cpython3.3/Lib/packaging to distutils2-py3/distutils2 and rename imports. (IIRC PyPI will require us to play games to have both 2.x and 3.x versions of distutils2.) Another question: What about the docs? Can we just point people to docs.python.org and tell them to mentally replace packaging with distutils2? If that is judged unacceptable, then I’ll synchronize the docs in the d2 repo, but that’s hours I won’t spend on bugs or features. Cheers

On 13/09/2011 16:57, Éric Araujo wrote:
What I'm doing for unittest2. 1) I have a script that pulls unittest from mercurial head and then applies patches to it to make it compatible with Python 3.1 - 3.2 and rename it from unittest to unittest2 2) I have a pypi project called unittestpy3k that holds the Python 3 version of unittest2 Projects using unittest2 for Python 3 then have a dependency on unittest2py3k - but the actual Python package name is unittest2. I only need to maintain a set of patches against unittest on head, rather than a whole branch. This works pretty well. All the best, Michael Foord
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Le 13/09/2011 18:34, Michael Foord a écrit :
That’s what I call playing games. I think it would make more sense to push 2.x-compatible and 3.x-compatible sdists to PyPI (with an appropriate 'Programming Language :: Python :: 2' or '3' classifier) and have the download tools be smart. Regards

On Thu, Sep 15, 2011 at 12:23 PM, Éric Araujo <merwok@netwok.org> wrote:
FWIW, I prefer this as well. I'd certainly appreciate the option to do it this way. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> "A person who won't read has no advantage over one who can't read." --Samuel Langhorne Clemens

On 15/09/2011 17:23, Éric Araujo wrote:
Hah, sure. In the meantime my way works *now* and with the existing tools. :-) (But only actually true for the way I make it available from pypi - the rest of the technique is not "playing games", right?) Yes, I would prefer to have a single project name with different distributions for Python 2 and 3 (and I looked into it) - but with the current tools the only way to achieve that is to put both versions into a single distribution. This prevents you from versioning them separately and is a pain to do anyway if the different versions are in different repos. The current tools are a real pain for versioning anyway. If your pypi page even *links* to a page that offers an alpha or beta (in development version) for download then both pip and easy_install will fetch that, in preference to the most recent version on pypi. So yes, I agree there is room for improvement in the current tools. Hopefully distutils2 will fix that. ;-) All the best, Michael Foord
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

On 15/09/2011 19:31, Michael Foord wrote:
I'm pretty sure recent releases of zc.buildout prefer "final" releases by default ;-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

Hi, I caught Tarek on IRC and forced him to answer my questions. Here are the latest news: - I have cleaned up and synchronized the distutils2 codebase with packaging in 3.3. All features and bugs are now identical. The test suite runs with Python 2.4 to 2.7; there are three or four test failures (linux, with threads, UCS4, not shared). Please clone, build (we backported hashlib for 2.4), test and file bugs! We’ll make an alpha4 as soon as all tests pass. - I have started work in a named branch to provide distutils2 for Python 3.1 and 3.2. Patches will flow between packaging, distutils2 and distutils2-py3. I’ll start a discussion on catalog-sig to improve support of parallel releases of 2.x and 3.x-compatible projects. - The docs in the d2 repo will be removed; people will go to docs.python.org and mentally convert packaging to distutils2. I’ll update the PyPI page. Cheers

Éric Araujo <merwok <at> netwok.org> writes:
Well sysconfig.py/sysconfig.cfg have been copied as is. I've only copied over specific things we need from shutil/functools/os, etc. so far to util.py. I haven't looked at 2.4/2.5 support yet: things like hashlib would probably need to be treated the same way Django handles this sort of backport of functionality.
I join my thanks to Tarek’s, and volunteer to follow on :)
That's good news :-) Regards, Vinay Sajip

On 15 August 2011 11:31, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
One thing that I, as a semi-interested bystander, would like to see is sort of a component of 4. Namely, a document somewhere addressing the question of why I, as a current user of distutils (setup.py, etc), should convert my project to use packaging/distutils2 - and what I'd need to do so. At the moment, I see no benefit to me in migrating. New projects, or projects that already know that they want one or more of the benefits that packaging/distutils2/setuptools bring, are a different matter. It's projects with needs satisfied by distutils, and code invested a distutils-based solution, that could do with some persuasion. I checked the docs, and "Distributing Python Modules" is for new projects, and "What's new" basically says "we expect you to migrate" but has no reasons or guidelines. If someone borrows the time machine and makes this already available, so much the better. Pointers would be appreciated! Paul.

I’m working on such a document, first in a doc set outside of the Python docs, and when it’s ready as an official HOWTO. (I’ll send the URL when I finish and publish it.)
I checked the docs, and "Distributing Python Modules" is for new projects,
That doc set is for distutils, unless you meant “Distributing Python Projects”, which is currently under severe updating. Regards
participants (11)
-
Antoine Pitrou
-
Chris McDonough
-
Chris Withers
-
Fred Drake
-
Michael Foord
-
Nick Coghlan
-
Paul Moore
-
Tarek Ziadé
-
Vinay Sajip
-
Yuval Greenfield
-
Éric Araujo