Questionnaire: Why do you use setuptools?
Setuptools non-support for Python 3 is currently a serious hindrance towards Python 3 aceptance. I'm trying to figure out what to do as a next step in the Python 3 support for setuptools. And I have encountered some obstacles. The first one is that setuptools requires itself for installing and running tests. That makes it hard to install it under Python 3. There are various solutions to this, but the next obstacle I encounter in choosing the right solution is that the code is hard to understand, and it makes me want to just rip it out and start over, or in even more frustrated moments, avoid the problems by not using setuptools at all. But the third obstacle for that is that I don't actually know what features of setuptools people use. I personally use setuptools for these reasons: 1. When I create projects with paster, it uses setuptools. 2. Setuptools makes it possible to specify requirements, which is then used by buildout. 3. Namespace packages require pkg_resources? 4. The test command. What are the other major reasons people use setuptools? Is there any good reason to not extract the namespace package support into a separate package? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On Sat, Apr 18, 2009 at 11:09 AM, Lennart Regebro <regebro@gmail.com> wrote:
Setuptools non-support for Python 3 is currently a serious hindrance towards Python 3 aceptance. I'm trying to figure out what to do as a next step in the Python 3 support for setuptools. And I have encountered some obstacles. The first one is that setuptools requires itself for installing and running tests. That makes it hard to install it under Python 3. There are various solutions to this, but the next obstacle I encounter in choosing the right solution is that the code is hard to understand, and it makes me want to just rip it out and start over, or in even more frustrated moments, avoid the problems by not using setuptools at all. But the third obstacle for that is that I don't actually know what features of setuptools people use.
I personally use setuptools for these reasons:
1. When I create projects with paster, it uses setuptools. 2. Setuptools makes it possible to specify requirements, which is then used by buildout. 3. Namespace packages require pkg_resources? 4. The test command.
What are the other major reasons people use setuptools?
entry points (in the "how to extend commands" topic I might propose to include it/use it but it would need to be separated from pkg_resources)
Is there any good reason to not extract the namespace package support into a separate package?
It's on its way http://svn.python.org/projects/peps/trunk/pep-0382.txt and PEP 345 is being reworked to include requires http://svn.python.org/projects/peps/branches/jim-update-345/pep-0345.txt Anyways, you can probably help if you worked on pkg_resources, by splitting all its components in pep-8-ied, readable elements in your branch Cheers Tarek
-- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Tarek Ziadé | http://ziade.org
On Sat, Apr 18, 2009 at 11:23, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
It's on its way http://svn.python.org/projects/peps/trunk/pep-0382.txt
Ah, OK, very good. I missed that one.
and PEP 345 is being reworked to include requires http://svn.python.org/projects/peps/branches/jim-update-345/pep-0345.txt
Sure. Do you have any feeling of when these could get released? 1 month? 6 months? 12 months? Five years? :-)
Anyways, you can probably help if you worked on pkg_resources, by splitting all its components in pep-8-ied, readable elements in your branch
Heh. That would require me to actually *understand* the code. :-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On Sat, Apr 18, 2009 at 11:41, Lennart Regebro <regebro@gmail.com> wrote:
On Sat, Apr 18, 2009 at 11:23, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
It's on its way http://svn.python.org/projects/peps/trunk/pep-0382.txt
Ah, OK, very good. I missed that one.
But.... it aims for Python 3.x and possible 2.7. That doesn't help me, as I will need a method that can support at least 2.5 and 2.6 as well. But then again, there is one already, and it seems the pkgutil version of namespaces would work fine, except that it doens't support Zip-files, which is OK, because they are a pain in the ass anyway. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
Tarek, On Sat, Apr 18, 2009 at 2:23 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
On Sat, Apr 18, 2009 at 11:09 AM, Lennart Regebro <regebro@gmail.com> wrote:
Setuptools non-support for Python 3 is currently a serious hindrance towards Python 3 aceptance. I'm trying to figure out what to do as a next step in the Python 3 support for setuptools. And I have encountered some obstacles. The first one is that setuptools requires itself for installing and running tests. That makes it hard to install it under Python 3. There are various solutions to this, but the next obstacle I encounter in choosing the right solution is that the code is hard to understand, and it makes me want to just rip it out and start over, or in even more frustrated moments, avoid the problems by not using setuptools at all. But the third obstacle for that is that I don't actually know what features of setuptools people use.
I personally use setuptools for these reasons:
1. When I create projects with paster, it uses setuptools. 2. Setuptools makes it possible to specify requirements, which is then used by buildout. 3. Namespace packages require pkg_resources? 4. The test command.
What are the other major reasons people use setuptools?
I ideally would like to use setuptools like pkg_resources for establishing the requires functionality to ensure that the versions of the packages I'm dealing with are legitimate and tested. Having a uniform versioning mechanism would greatly simplify bug reporting and triage, not just within the python community, but also in any communities outside of python that use / distribute python utilities / modules.
entry points (in the "how to extend commands" topic I might propose to include it/use it but it would need to be separated from pkg_resources)
Is there any good reason to not extract the namespace package support into a separate package?
It's on its way http://svn.python.org/projects/peps/trunk/pep-0382.txt
and PEP 345 is being reworked to include requires http://svn.python.org/projects/peps/branches/jim-update-345/pep-0345.txt
I realize that PEP 345 isn't your PEP, but if you could relay this to the authors, it would be much appreciated: The section... Supported-Platform (multiple use) Binary distributions containing a PKG-INFO file will use the Supported-Platform field in their metadata to specify the OS and CPU for which the binary package was compiled. The semantics of the Supported-Platform field are not specified in this PEP. Example: Supported-Platform: RedHat 7.2 Supported-Platform: i386-win32-2791 ... isn't 100% complete in my mind. Will version checking of the OS be properly supported so I can say: Supported-Platform: RedHat (>7.2) ... to say that Redhat 7.2+ is supported? I say this because certain versions of FreeBSD have certain featuresets that were introduced in later versions (most notably the ones that are basically fixing pthread and rt support with multiprocessing), and specifying a version range would be helpful. If it's not within the scope of the PEP at all to address this point, this point should be referenced wherever it exists in other documentation.
Anyways, you can probably help if you worked on pkg_resources, by splitting all its components in pep-8-ied, readable elements in your branch
Thanks, -Garrett
Hi all, me again.. I think I have a handle on how to implement package deinstallation in my gui package manager. One thing I came accross is .EGG files... I'm wondering what people think of these? are they good? how do people see them fitting in with distutils.. ? etc Somebody mentioned they don't work to well under gentoo but i don't know what the exact complaint is. Are source distributions (ie distutils) competitive with .EGG packaging? Regards David
At 01:52 AM 4/28/2009 -0400, David Lyon wrote:
Are source distributions (ie distutils) competitive with .EGG packaging?
Yes. Really the main reason to distribute egg files is if you want end-users to be able to download a single file as a plugin for your application, or certain other special cases.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Lennart Regebro wrote:
Setuptools non-support for Python 3 is currently a serious hindrance towards Python 3 aceptance. I'm trying to figure out what to do as a next step in the Python 3 support for setuptools. And I have encountered some obstacles. The first one is that setuptools requires itself for installing and running tests. That makes it hard to install it under Python 3. There are various solutions to this, but the next obstacle I encounter in choosing the right solution is that the code is hard to understand, and it makes me want to just rip it out and start over, or in even more frustrated moments, avoid the problems by not using setuptools at all. But the third obstacle for that is that I don't actually know what features of setuptools people use.
I personally use setuptools for these reasons:
1. When I create projects with paster, it uses setuptools. 2. Setuptools makes it possible to specify requirements, which is then used by buildout. 3. Namespace packages require pkg_resources? 4. The test command.
What are the other major reasons people use setuptools?
- - setuptools.find_pacakges built-in SVN support makes a whole class of packaging errors go away for me. - - virtualenv makes isolation between different applications sane; it installs setuptools, and then makes 'easy_install' a pleasure to use (I still can't believe people use easy_install in their system Python!) - - entry points serve as crude equivalents of "named utilities" in the Zope component architecture; they allow an application to define a class of plug points, which are then filled by other libraries. These plug points can then be configured together declaratively (e.g. in an INI file) using their names. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJ6bdD+gerLs4ltQ4RApw2AKDO9zRTVw5BGYEd0NKCFWdwc0sbFACghRqW 8BqG80vsK6kS8OEcG+UFelw= =fTfI -----END PGP SIGNATURE-----
On Sat, Apr 18, 2009 at 07:19:31AM -0400, Tres Seaver wrote:
What are the other major reasons people use setuptools?
- setuptools.find_pacakges built-in SVN support makes a whole class of packaging errors go away for me.
- virtualenv makes isolation between different applications sane; it installs setuptools, and then makes 'easy_install' a pleasure to use (I still can't believe people use easy_install in their system Python!)
Why not? They're told to. Go to a random Python package's web page and chances are you'll see "to install it, simply run easy_install MyAwesomePackage".
- entry points serve as crude equivalents of "named utilities" in the Zope component architecture; they allow an application to define a class of plug points, which are then filled by other libraries. These plug points can then be configured together declaratively (e.g. in an INI file) using their names.
Marius Gedminas -- Microsoft -- because God hates us.
On Sat, Apr 18, 2009 at 11:09:11AM +0200, Lennart Regebro wrote:
I don't actually know what features of setuptools people use.
1. 'python setup.py sdist register upload' (you could say this is a distutils feature, but my packages tend to unconditionally import setuptools in their setup.py) 2. virtualenv + easy_installing packages inside + getting all the dependencies specified with install_requires in step 1. 2b. same but with zc.buildout 3. entry points (usually for specifying console_scripts) Marius Gedminas -- If you sat a monkey down in front of a keyboard, the first thing typed would be a unix command. -- Bill Lye
On Sat, Apr 18, 2009 at 11:09:11AM +0200, Lennart Regebro wrote:
I don't actually know what features of setuptools people use.
* Dependency specification (install_requires, etc) * Virtualenv + easy_install for installation * Entrypoints, both for specifying console_scripts and as an easy way to provide plugable hooks in an application * Specifying new distutils commands as setuptools entrypoints Ronald
On Sat, Apr 18, 2009 at 4:09 AM, Lennart Regebro <regebro@gmail.com> wrote:
Setuptools non-support for Python 3 is currently a serious hindrance towards Python 3 aceptance. I'm trying to figure out what to do as a next step in the Python 3 support for setuptools. And I have encountered some obstacles. The first one is that setuptools requires itself for installing and running tests. That makes it hard to install it under Python 3. There are various solutions to this, but the next obstacle I encounter in choosing the right solution is that the code is hard to understand, and it makes me want to just rip it out and start over, or in even more frustrated moments, avoid the problems by not using setuptools at all. But the third obstacle for that is that I don't actually know what features of setuptools people use.
I personally use setuptools for these reasons:
1. When I create projects with paster, it uses setuptools.
Of course if you create a project template that uses distutils, then that's what you'll get... just happens no one does that with paster templates.
2. Setuptools makes it possible to specify requirements, which is then used by buildout.
In pip at least it does this with *.egg-info/install_requires.txt, but in easy_install and I'm guessing buildout it's using the pkg_resources.Distribution object.
3. Namespace packages require pkg_resources?
There's a way of doing it with pkgutils, but in some way that I don't understand, pkg_resources does it better.
4. The test command.
What are the other major reasons people use setuptools?
Entry points are the big one for me. There is some other metadata that I use from time to time, but I'm sure I could work around it. Of course there's the specific things pip uses as noted in a previous email. -- Ian Bicking | http://blog.ianbicking.org
On Mon, Apr 20, 2009 at 09:17, Ian Bicking <ianb@colorstudy.com> wrote:
Of course if you create a project template that uses distutils, then that's what you'll get... just happens no one does that with paster templates.
Oh, sure, it's just lazyness from my side, nothing else.
Entry points are the big one for me. There is some other metadata that I use from time to time, but I'm sure I could work around it. Of course there's the specific things pip uses as noted in a previous email.
Yeah, it's clear to me that there is enough different features of setuptools in such widespread usage that replacing it with something else that actually is easily portable to Python 3 isn't feasible. To get more modules running under Python 3 we really have to make setuptools run and install smoothly under Python 3 too. I'll make a post on this. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
At 02:17 AM 4/20/2009 -0500, Ian Bicking wrote:
3. Namespace packages require pkg_resources?
There's a way of doing it with pkgutils, but in some way that I don't understand, pkg_resources does it better.
pkgutil doesn't support zip files (or any other non-filename importers/path strings), later additions to sys.path, nested namespaces... and that may not be a complete list.
Roughly in order of most important to least important: 1. a formal, machine-readable way to specify our dependencies; see XYZ [2] 2. automatic installation of dependencies; This reduces the "dependencies" section in the instructions for installing Tahoe [1] from having sixteen separate software packages with separate versioning requirements to having one: Python 2.4.2 through 2.6. 3. Obviously having even one "dependency" in your installation instructions is too many, so we try to provide .deb's, .rpm's, gentoo ebuilds, Windows installers (e.g. as produced by py2exe or bbfreeze), Macintosh applications, etc. Many of the tools that produce these other formats rely on the above-mentioned feature #1 and/or feature #2. 4. Some people like to use "easy_install $PACKAGE". In fact, quite a lot of people -- if I recall the results of Tarek's packaging survey correctly, it was more popular among the responding Python programmers than any other method of installing packages [3]. 5. Using plugins so that you can add features to your build system, such as revision control integration, unit testing, versioning, pyflakes, uploading, etc. which you re-use as black box modules maintained by someone else instead of copying code into your setup.py or your GNUmakefile. [4, 5, 6, 7, 8]. 6. Using the console scripts feature of setuptools to make it so your scripts that you wrote for Unix also work on Windows. 7. python setup.py sdist upload register 8. revision control integration; I want all files under revision control to be included in packages. Regards, Zooko
Ugh. Sorry -- I was on a plane when I composed this and I accidentally sent it the next time I connected to the Net before adding the references. Here they are: On Apr 20, 2009, at 18:06 PM, Zooko O'Whielacronx wrote:
Roughly in order of most important to least important:
1. a formal, machine-readable way to specify our dependencies; see XYZ [2]
[2] http://allmydata.org/trac/tahoe/browser/_auto_deps.py
2. automatic installation of dependencies; This reduces the "dependencies" section in the instructions for installing Tahoe [1] from having sixteen separate software packages with separate versioning requirements to having one: Python 2.4.2 through 2.6.
[1] http://allmydata.org/source/tahoe/trunk/docs/install.html
3. Obviously having even one "dependency" in your installation instructions is too many, so we try to provide .deb's, .rpm's, gentoo ebuilds, Windows installers (e.g. as produced by py2exe or bbfreeze), Macintosh applications, etc. Many of the tools that produce these other formats rely on the above-mentioned feature #1 and/or feature #2.
4. Some people like to use "easy_install $PACKAGE". In fact, quite a lot of people -- if I recall the results of Tarek's packaging survey correctly, it was more popular among the responding Python programmers than any other method of installing packages [3].
[3] http://tarekziade.wordpress.com/2009/03/26/packaging-survey-first- results
5. Using plugins so that you can add features to your build system, such as revision control integration, unit testing, versioning, pyflakes, uploading, etc. which you re-use as black box modules maintained by someone else instead of copying code into your setup.py or your GNUmakefile. [4, 5, 6, 7, 8].
[4] http://pypi.python.org/pypi/setuptools_darcs [5] http://pypi.python.org/pypi/setuptools_bzr [6] http://pypi.python.org/pypi/setuptools_hg [7] http://pypi.python.org/pypi/setuptools_pyflakes [8] http://pypi.python.org/pypi/setuptools_trial [9] http://pypi.python.org/pypi/collective.releaser [10] http://pypi.python.org/pypi/django-reusableapps [11] http://pypi.python.org/pypi/eggchecker [12] http://pypi.python.org/pypi/iw.releaser [13] http://pypi.python.org/pypi/peggy [14] http://pypi.python.org/pypi/CodeTools [15] http://pypi.python.org/pypi/collective.dist [16] http://pypi.python.org/pypi/disthelper [17] http://pypi.python.org/pypi/bbfreeze [18] http://pypi.python.org/pypi/buildutils [19] http://pypi.python.org/pypi/eggtestinfo [20] http://pypi.python.org/pypi/pip [21] http://pypi.python.org/pypi/pydeploy32 [22] http://pypi.python.org/pypi/py2app [23] http://pypi.python.org/pypi/pyinstall [24] http://pypi.python.org/pypi/pyrun [25] http://pypi.python.org/pypi/PythonEggTools [26] http://pypi.python.org/pypi/setuphelper [27] http://pypi.python.org/pypi/virtualenv [28] http://pypi.python.org/pypi/yolk [29] http://pypi.python.org/pypi/yolk-portage [30] http://pypi.python.org/pypi/zc.buildout [31] http://pypi.python.org/pypi/Enstaller I should emphasize that this list is by no means exhaustive -- I just scanned through the results of searching pypi for the word "setuptools" and pulled out a few that seemed to be vaguely relevant. I'm think that there might a danger of us distutils-sig folks underestimating the value offered by compatibility with setuptools. Most of the authors of these packages are probably not reading distutils-sig, and so most of them are probably not answering the questions like "Why do you use setuptools?". (For one thing, this mailing list is just insanely high volume -- it is hard to make time to keep up.) Many of the authors of these packages and the other packages like these are not going to go to much, if any, effort to change their tools to support the new Distutils. To the degree to which new authors of new projects which use the new Distutils can get, for free, compatibility with projects like these ones that already use setuptools, then everyone can benefit. Regards, Zooko
6. Using the console scripts feature of setuptools to make it so your scripts that you wrote for Unix also work on Windows.
7. python setup.py sdist upload register
8. revision control integration; I want all files under revision control to be included in packages.
Zooko O'Whielacronx wrote:
Many of the authors of these packages and the other packages like these are not going to go to much, if any, effort to change their tools to support the new Distutils.
agreed on this.
To the degree to which new authors of new projects which use the new Distutils can get, for free, compatibility with projects like these ones that already use setuptools, then everyone can benefit.
The current distutils needs to be improved, this at least seems to be consensual. From the Pycon summary (I was not there, just using the wiki and Tarek emails as a reference), it seems that some things from setuptools will be incorporated in concept if not in code (install requirements, better extensibility, etc...). The technical details are not all set, but they will be at some point. The problem with a new "Distutils2" 100 % compatible with the current one is that it is impossible to do without recreating all the things which it does wrong, because distutils has some serious design issues which cannot be solved without breaking things, and some packages depend on distutils internals quite a bit - thing about all the ones which are broken with setuptools for example. Doing something which is incompatible and requires new scripts from scratch is not an option either. Something which could convert simple setup.py files to the new distutils could be more reasonable (a bit like 2to3), and I think it is doable for a significant proportion of current packages, through an extended set of static metadata. Such a proposal is being worked on (http://wiki.python.org/moin/Distutils/StaticMetadata); I think the suggestion of making those metadata independent of distutils design (being more high level and independent of distutils commands) should be given serious thought. It can be done gradually, too. David
On Apr 21, 2009, at 18:54 PM, David Cournapeau wrote:
The problem with a new "Distutils2" 100 % compatible with the current one is that it is impossible to do without recreating all the things which it does wrong,
I'm sorry, I was unclear. I do not think that it is super important for a programmer who currently uses setuptools to package their software to be able to switch to distutils2 to package their software. I do think that it is super important for a programmer who uses distutils2 to be able to easily re-use the software written by *other* people who use setuptools, and vice versa. This means the sdist format should be interoperable, probably the bdist_egg format should be interoperable, and the "install_requires" metadata should be interoperable. It would also be nice if a programmer who is choosing whether to use setuptools or distutils2 to package their own software could consider the library of development tools plugins as equally available for both, instead of thinking "Oh, I could use the new distutils2, which has a cleaner design and is in Python 2.7, but if I use setuptools then I can continue to use all these development plugins that already exist". So if distutils2 can re-use extant setuptools plugins, then that would be handy. Regards, Zooko
Zooko O'Whielacronx wrote:
On Apr 21, 2009, at 18:54 PM, David Cournapeau wrote:
The problem with a new "Distutils2" 100 % compatible with the current one is that it is impossible to do without recreating all the things which it does wrong,
I'm sorry, I was unclear. I do not think that it is super important for a programmer who currently uses setuptools to package their software to be able to switch to distutils2 to package their software.
Ah, ok, then I indeed partly misunderstood what you said.
I do think that it is super important for a programmer who uses distutils2 to be able to easily re-use the software written by *other* people who use setuptools, and vice versa. This means the sdist format should be interoperable, probably the bdist_egg format should be interoperable, and the "install_requires" metadata should be interoperable.
Oh, yes, definitely. I would not even imagine designing something where eggs or sdist formats are not *exactly* the same, whether they are produced by distutils, setuptools, distutils2 or whatever. This is why the formats have to be specified, and the code to produce them separated from the rest (e.g. there should be a library to produce an egg from the metadata without using the rest of setuptools, same for sdist, etc...). And this can be done for the current distutils as well anyway. Ideally, the static metadata format should be the same, so that you could really, practically, support both systems 'for free'. At least for simple packages (by simple I mean packages without difficult C code, or more generally specificities which require setup magic, etc...).
It would also be nice if a programmer who is choosing whether to use setuptools or distutils2 to package their own software could consider the library of development tools plugins as equally available for both, instead of thinking "Oh, I could use the new distutils2, which has a cleaner design and is in Python 2.7, but if I use setuptools then I can continue to use all these development plugins that already exist". So if distutils2 can re-use extant setuptools plugins, then that would be handy.
This seems more difficult to do (although I have never thought about this problem to be honest, contrary to the other ones mentioned above - I don't use setuptools that much myself). Many plugins work on the assumption of commands - and if I were to design a new tool, commands would be *the* thing I would like to get rid of (it does not play well with the idea of handling dependencies, for once). cheers, David
Zooko O'Whielacronx <zooko@zooko.com> writes:
Roughly in order of most important to least important:
My response is essentially a subset of Zooko's.
1. a formal, machine-readable way to specify our dependencies […]
2. automatic installation of dependencies […]
3. Obviously having even one "dependency" in your installation instructions is too many, so we try to provide [system packages] […] Many of the tools that produce these other formats rely on the above-mentioned feature #1 and/or feature #2. […]
7. python setup.py sdist upload register
All of the above. -- \ “Odious ideas are not entitled to hide from criticism behind | `\ the human shield of their believers' feelings.” —Richard | _o__) Stallman | Ben Finney
participants (12)
-
Ben Finney -
David Cournapeau -
David Lyon -
Garrett Cooper -
Ian Bicking -
Lennart Regebro -
Marius Gedminas -
P.J. Eby -
Ronald Oussoren -
Tarek Ziadé -
Tres Seaver -
Zooko O'Whielacronx