Mailman 3 September 2008 - Distutils-SIG

Distribute, what's next
by Tarek Ziadé 27 Sep '08

27 Sep '08

Hello, I am trying to build a page now in Python.org wiki, that synthetizes the current state of packaging in Python, the problems, and what can be done. I'll work on it this week-end, and try to synthetize everything that has been said in the latest threads (and elsewhere). My goal is to gather in this single page everything in a high-level view, so hopefully, so maybe we can find the "path" But I am very Zope-oriented, so If some other people want to help me, I'll be hanging in #distutils on freenode (paris time) http://wiki.python.org/moin/Distribute ++ Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

1 0

Re: [Distutils] [Fwd: Re: Annoucing distribute project]
by Guido van Rossum 26 Sep '08

26 Sep '08

I don't think anyone loves distutils, or even likes it. Unfortunately replacing it will be very tough, because distutils has arcane but important knowledge built into it about many platforms, and a new from-scratch system is unlikely to reach that level of coverage for years. There's also the issue that it's not enough to roll out the replacement with a new version of Python -- it would leave everyone using the old distutils in the lurch. It's a prime example of the network effect: until everyone owns one, nobody will want to own one. The best strategy I can think of would be to gradually rewrite the existing distutils, remaining compatible but also offering new APIs, and 5 years hence maybe the installed base will be close enough to 100% that developers writing setup scripts will start using the new features. That's how AJAX conquered the world. --Guido On Thu, Sep 25, 2008 at 10:18 AM, Blake Winton <bwinton(a)latte.ca> wrote: > Just as a question, is there _any_ chance of a new project to replace > distutils being accepted into Python? > > Thanks, > Blake. > > -------- Original Message -------- > Subject: Re: [Distutils] Annoucing distribute project > Date: Thu, 25 Sep 2008 12:55:36 -0400 > From: Jim Fulton <jim(a)zope.com> > To: Andreas Jung <lists(a)zopyx.com> > CC: distutils-sig(a)python.org > References: <94bdd2610809240432i307ed937rbfb1fd063cec915b(a)mail.gmail.com> > <20080924173708.13DA93A4128(a)sparrow.telecommunity.com> > <94bdd2610809241244p13ad74dbkc13ee5b3ae3e14c8(a)mail.gmail.com> > <48DAA90C.20504(a)free.fr> > <20080924213342.612CC3A4127(a)sparrow.telecommunity.com> > <94bdd2610809241456o20dade8bl4b46f3e0ed7aeb90(a)mail.gmail.com> > <20080924223457.2BD7E3A4127(a)sparrow.telecommunity.com> > <94bdd2610809242252j131d9fbcyd3e629667b455d29(a)mail.gmail.com> > <20080925163034.208613A4072(a)sparrow.telecommunity.com> > <C0E161B65AAB12A92255EE01(a)suxmac.local> > > > On Sep 25, 2008, at 12:46 PM, Andreas Jung wrote: > >> >> >> --On 25. September 2008 12:31:40 -0400 "Phillip J. Eby" >> <pje(a)telecommunity.com > wrote: >> >> >>> Personally, I would rather see not a "fork", but development of an >>> entirely new disutils replacement, and then not worry at all about >>> backward compatibility at the setup.py level. Apart from the >>> pkg_resources module, I would treat setuptools and distutils as legacy >>> infrastructure, and explore better ways to build and manage >>> distributions >>> (including eggs). >> >> We can not restart with something completely different or new. Such an >> approach would be totally unacceptable due to the huge amount of existing >> modules having switched to setuptools. Setuptools or an improved codefork >> must be the tools for managing distibutions in Python. > > I would be willing to start with something new if and only if it was > adopted as part of Python. > > Jim > > -- > Jim Fulton > Zope Corporation > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG(a)python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -- --Guido van Rossum (home page: http://www.python.org/~guido/)

5 8

Annoucing distribute project
by Tarek Ziadé 25 Sep '08

25 Sep '08

Hi, I am launching a fork of setuptools. The name is "Distribute". (not definitive) and will be community-driven This fork will remain 100% compatible with setuptools, and follow closely setuptools evolutions. Roadmap ======= - Distribute 0.1 (09/27/2008) The first release of Distribute will be done at the end of this week, targeting setuptools 0.6x, and will provide all latest bugfixes done in the 0.6 branches of setuptools, like Subversion 1.5 compatibility. - Distribute 0.2 (10/27/2008) The 0.2 release will be done in 1 month, targeting setuptools current trunk, and will focus on providing a maximum of bugfixes. You can provide patches, or aks for a particular bug to be fixed. Use the launchpad bug tracker for that https://bugs.launchpad.net/distribute - Distribute 0.3 The 0.3 release will contain new features and improvments, and its release date decide when 0.2 comes out. Project managment =============== Distribute is using launchpad (https://launchpad.net/distribute) and bzr and anyone that wants to contribute is welcome to provide a branch. The branch are merged if they provide tests and if there is a consensus of the community on it. The process is as follow for a contribution: - create a bzr branch - talk about it on distutils-SIG, using [distribute] in the header, asking for review/merge - it is discussed in the mailing list - if no one objects, and if it has tests, it will be merged by a maintainer I am the maintainer at this point but this role will be extended to other people that want to become a maintainer, so I will not become a botlleneck for this project. A maintainer has to be nominated by 100% of the maintainers, and there will be at most 4 maintainers in this project. Maintainers are added or removed after each release. They work together to release the next milestone. Let me know if you want to become a maintainer. Maintainers need to discuss their changes as well on the mailing list, and their only privilege is being able to merge branches on the main branch, because they are trust to do so. Cheers Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

10 20

Re: [Distutils] Annoucing distribute project
by Phillip J. Eby 25 Sep '08

25 Sep '08

At 07:52 AM 9/25/2008 +0200, Tarek Ziadé wrote: >Anyway, that sounds great indeed. But since setuptools is not on the >top of your priority sometimes (you said it wasn't, and we could see >it wasn't), are those people blessed to release it as well ? I don't see a problem with that, although the actual release process itself is nearly 100% automated. (See the 'release.sh' file.) Assuming that there is a releasable distribution in SVN, it takes maybe 15 minutes for me to cut a release and bump the version, if that long. (And it could probably be further automated.) By the way, you'll notice if you inspect the distutils-sig archives carefully, I'm *extremely* responsive when people post questions or report bugs that haven't been previously answered or fixed. I simply don't have a lot of time to drive new development. There are few people qualified to do the design vetting and testing that needs to happen for major new features to be added atop the existing code base, and that's 100% independent of any forking. If the persons existed who could actually do this, there is no technical reason why they couldn't have been doing it prior to the fork. (Which leads me to believe that they don't exist or have the time, and thus, the fork will not actually help anything.) Personally, I would rather see not a "fork", but development of an entirely new disutils replacement, and then not worry at all about backward compatibility at the setup.py level. Apart from the pkg_resources module, I would treat setuptools and distutils as legacy infrastructure, and explore better ways to build and manage distributions (including eggs). If I had the time, that's what I'd be doing. >I mean, if we have a clear roadmap of the next releases of >setuptools, with dates, and if you are unable to release it >yourself, can we count on them to do so ? If so I'd be more than >happy to stop that fork, and to continue happily to work in the bug >tracker if I have a visible roadmap. Feel free to create such a roadmap; as you can see, I do have time to answer a few emails and give feedback. And if somebody responsible wants to take over the job of deciding whether a release is "ready", I can certainly pull the trigger. The bottleneck is not me per se; it's having qualified parties. In the long run, having more tests would help a lot with that, as I've said before. In the short run, it's people with strong multi-platform, multi-distribution, multi-project, multi-Python-version experience that are sorely lacking in the patch contribution, vetting, and design departments. And right now, the only way for me to evaluate such contributors is on the basis of their submitted patches and design proposals... which are few and far between. Historically, most of the patches I get are poorly thought-out even on the level of what the patch does for that one person; i.e., there will be cases that that very person will have problems with, let alone what problems other people will have. This does not give me great hope for the viability of a fork whose expressed purpose is to accept more patches, faster. :) (This situation is due largely to the flexibility of distutils, by the way. The distutils accepts an utterly ridiculous number of possible source trees, for example. It also supports a ridiculous number of installation targets, formats, etc. On the other hand, this same flexibility also makes it virtually impossible for a competitor to "take over the market" unless it provides near-100% backward compatibility, as setuptools does.)

4 5

Re: [Distutils] Annoucing distribute project
by Phillip J. Eby 25 Sep '08

25 Sep '08

At 01:32 PM 9/24/2008 +0200, Tarek Ziadé wrote: >Hi, > >I am launching a fork of setuptools. The name is "Distribute". (not >definitive) and will be community-driven > >This fork will remain 100% compatible with setuptools, and follow >closely setuptools evolutions. Unfortunately, the only way you can remain 100% compatible with code "in the field" is by calling your distribution "setuptools" also, or by having it also install a dummy setuptools egg or at least .egg-info. Otherwise, packages that declare a dependency to a specific setuptools version will cause it to be installed, thereby overwriting your package. There's also setup scripts that use ez_setup, which you could work around for easy_install scenarios with clever enough monkeypatching, but someone directly running a setup script would still break. If you're going to attempt this, I would consider very carefully how you will address these issues, as it seems to me that it would be very easy for someone to silently hose their system this way, and possibly quite difficult to find out what went wrong or how to fix it.

4 6

Re: [Distutils] Annoucing distribute project
by Phillip J. Eby 25 Sep '08

25 Sep '08

At 11:56 PM 9/24/2008 +0200, Tarek Ziadé wrote: >On Wed, Sep 24, 2008 at 11:34 PM, Phillip J. Eby ><<mailto:pje@telecommunity.com>pje(a)telecommunity.com> wrote: >In other words, I don't recommend mixing fork goals. If the goal is >merely to have more-frequent releases of setuptools, it would be >better to have a snapshot facility for same, and release dev-tagged >eggs. Conversely, if the goal is to prototype new features, then it >doesn't make a lot of sense to advertise it as a stable or >up-to-date setuptools. > >So, my personal hope is that the persons doing the fork will make it >clear to their users which it is: unstable feature prototype, or >simple release snapshots? (Where the latter could just as easily be >automated.) > >The point is that the stability you are claiming means that the >whole community depends on your actions. The fact that you were busy >lately made things even slower. Actually, it was the community that slowed down 0.6c9, in that I would have released it last month, except that certain contributors were not ready. As it is, I ended up giving up waiting for one of them to come through -- ironically, it was someone who used to complain a lot about delays. ;-) > That is my whole point. Even if you set up a snapshot facility, > the only person that is able to make changes in setuptools is you > and only you. Sigh. That's simply not true, as has been discussed here before -- more than once. If you check the commit logs, you'll see that other people HAVE made changes to setuptools, and are still quite able to do so. Jim Fulton is already a "blessed" committer, and I'm quite close to blessing P. Jenvey (if he wants it). And, when somebody else steps up with a series of quality patches (backed with pre-reviewed specs in the case of features with significant interactions), then that person can get blessed too. >So I'll make it clear to our users: it will be a project with stable >releases, and unstable features in branches, and a trunk to merge >everything, like all the open source project out there. Great. Just please make sure that you ask your contributors and users NOT to expect support from me, and to not use the setuptools bug tracker for non-setuptools issues. It's not reasonable to expect me to support the fork in addition to the original version. (I'm also somewhat concerned about what issue the OS vendors will blame me or "Python eggs" in general for next, even though I've got nothing to do with the fork.)

2 1

Re: [Distutils] Can we prevent setuptools from writing to stdout
by Phillip J. Eby 24 Sep '08

24 Sep '08

At 02:24 PM 9/24/2008 +0100, Fadhley Salim wrote: >Is it possible to prevent setuptools from writing anything to stdout? > >For various inconvenient reasons our package must install silently - >that means the installation process must not write to either stdout >or stderr. Other than hacking distutils / setuptools to ensure that >there's nothing which can write i've not yet been able to come up >with a reliable means to control output. Temporarily replace sys.stdout with another object, such as a StringIO.

1 0

setuptools 0.6c9 released
by Phillip J. Eby 24 Sep '08

24 Sep '08

I've just cut a release of 0.6c9. Some outstanding bugs remain; see the tracker for details. A few (issues 16, 20, and 23) have candidate fixes in the trunk but have not been applied to the branch as yet.

1 0

Setup tools causes problems with orca finding its python packages
by Michael Whapples 24 Sep '08

24 Sep '08

I have noticed that easy_install has a easy-install.pth file which seems to be adding the directories specified in it before any other python package directories, including the environment variable PYTHONPATH. This causes a problem with orca which uses a bash script to start itself, and specifies where its python packages should be found by setting the PYTHONPATH variable. This wouldn't cause problems except when a user has more than one version of orca installed (eg. a stable version from the distribution installed in the standard python site-packages directory and one (SVN trunk version) in a custom directory). I found that easy_install had added the standard python site-packages directory (/usr/lib/python2.5/site-packages on my system) to easy-install.pth so making it impossible to use my custom install of orca as the standard location was first in sys.path. Is what easy_install doing really a standard thing? If not then this should be corrected to get things to behave as they should. If it is standard then either can easy_install prevent the standard python site-packages module from being added to easy-install.pth or can you suggest a way that orca can overcome this problem? Michael Whapples

2 3

Annoucing collective.eggproxy, the smart mirror for PyPI
by Tarek Ziadé 24 Sep '08

24 Sep '08

Hi, I just wanted to announce the release of collective.eggproxy ( http://pypi.python.org/pypi/collective.eggproxy/0.2.0) collective.eggproxy is a smart mirror for PyPI. It will collect packages on PyPI only when a program like easy_install or zc.buildout asks for it. In other words, unlike some mirrors that act like rsync and get the whole PyPI base (more than 5 gigas) collective.eggproxy will only get what you need. At first run collective.eggproxy downloads pypi index and builds a page of links. When a software asks for a specific package, version, etc. collective.eggproxy downloads it if needed and stores it locally. Want to give a try ? try it in two lines with easy_install: $ easy_install collective.eggproxy $ eggproxy_run That's it ! Cheers Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

1 0