Re: [Distutils] Some negative press for easy_install
At 11:10 AM 2/9/2006 -0500, Kevin Dangoor wrote:
On 2/9/06, Ben Bangert <ben@groovie.org> wrote:
It's mainly because Routes is relied on by quite a few other setuptools-enabled packages, so being able to easy install it was necessary. I didn't have a non-setuptools build mainly because I couldn't see how to setup a setup.py file in such a way that I could make both versions at once. I'm assuming I'd need two setup.py's and to swap them in the build depending on if it was a setuptools build or not.
Could you do something like this:
try: from setuptools import setup except ImportError: from distutils.core import setup
On your system, you'd then be able to build eggs at will. Other people who download an sdist but don't have setuptools will just get normal sdist-like behavior.
Almost. You'll have to use MANIFEST.in or they won't be able to do bdist_rpm or certain other commands from your sdist. That is, you can't use setuptools' revision control features, since they won't have them available. Aside from that minor point, this is the approach I'd recommend to authors who are only using setuptools in order to build eggs and to upload stuff to PyPI, and aren't pointing their users to the docs. You will have to forego all of the normal conveniences of setuptools, however. Personally, however, I think that the short statement somebody else posted about "hey, I use setuptools, and if you're installing anywhere besides your system's site-packages directory you should check out this link" (to the Custom Installation Locations doc) is more than sufficient. I've been keeping setuptools in "alpha" mainly for API-change reasons, but it seems to me that the massive exposure to eggs given by TurboGears has shaken out all but the "new user hasn't read the doc and needs a custom install" problems. What impressed me most about Joe's blog post is not his installation failed, but just how close he was to actual success without even reading the documentation. So, as far as I'm concerned, as soon as it's at the point where he'd have succeeded in what he did, setuptools should qualify as beta quality. And that's just a bit of spit-and-polish away. So even though I'm not thrilled about the blog post, it's actually a pretty positive milestone in two ways: 1. It highlights how little there is left to go to get to a quality installation experience 2. It highlights how popular setuptools has already become.
Phillip J. Eby wrote:
At 11:10 AM 2/9/2006 -0500, Kevin Dangoor wrote:
Could you do something like this:
try: from setuptools import setup except ImportError: from distutils.core import setup
On your system, you'd then be able to build eggs at will. Other people who download an sdist but don't have setuptools will just get normal sdist-like behavior.
Almost. You'll have to use MANIFEST.in or they won't be able to do bdist_rpm or certain other commands from your sdist. That is, you can't use setuptools' revision control features, since they won't have them available.
Note that matplotlib tried essentially this for a while, but apparently some folks really didn't like it. I'm not sure what exactly broke on their systems (they didn't complain to the mailing list), but when setup.py reverted to a plain distutils script, they cheered. Matplotlib now has setupegg.py: from setuptools import setup execfile('setup.py', {'additional_params' : {'namespace_packages' : ['matplotlib.toolkits']}}) I guess the point is that just because someone has setuptools installed on their system doesn't mean they want to use it. Another option, for a single setup.py, is something like the following: if 'setuptools' in sys.modules: use_setuptools_options() This way, setup.py can be setuptools-aware without doing 'import setuptools', but the user would have to do: python -c "import setuptools; execfile('setup.py')"
Andrew Straw wrote:
Note that matplotlib tried essentially this for a while, but apparently some folks really didn't like it. I'm not sure what exactly broke on their systems (they didn't complain to the mailing list), but when setup.py reverted to a plain distutils script, they cheered.
That's right, they complained to me, instead. :-) The issue was that they had setuptools on their system because they were building eggs for another project, but specifically *didn't* want to install matplotlib as an egg.
Matplotlib now has setupegg.py:
from setuptools import setup execfile('setup.py', {'additional_params' : {'namespace_packages' : ['matplotlib.toolkits']}})
I guess the point is that just because someone has setuptools installed on their system doesn't mean they want to use it.
Another option, for a single setup.py, is something like the following:
if 'setuptools' in sys.modules: use_setuptools_options()
This way, setup.py can be setuptools-aware without doing 'import setuptools', but the user would have to do: python -c "import setuptools; execfile('setup.py')"
And this is my preferred method for those packages that want to provide the appropriate information for egg builds but not (yet) *require* setuptools or egg builds. It means that "easy_install somepackage==dev" works (a failing of the setupegg.py approach) and allows "python setup.py install" to work as standard distutils does. -- Robert Kern robert.kern@gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter
At 01:31 PM 2/9/2006 -0600, Robert Kern wrote:
Andrew Straw wrote:
Note that matplotlib tried essentially this for a while, but apparently some folks really didn't like it. I'm not sure what exactly broke on their systems (they didn't complain to the mailing list), but when setup.py reverted to a plain distutils script, they cheered.
That's right, they complained to me, instead. :-)
The issue was that they had setuptools on their system because they were building eggs for another project, but specifically *didn't* want to install matplotlib as an egg.
setuptools-based packages can be forced to install the old-fashioned way using: setup.py install --single-version-externally-managed as long as you also specify a --root directory or a --record file. This is of course not upgradeable or uninstallable without help from a packaging tool that can utilize the results of --root or --record. (This is covered in the "What Your Users Should Know" section of the setuptools manual, btw, under the subheading "Creating System Packages".) If somebody could write a standalone version of the What Your Users Should Know section that everybody can link to, that would likely help resolve these problems sooner. (I'll get to it eventually if nobody else does.) Setuptools now has most of the features that are needed to get used in these situations, but the information that end users need isn't quickly accessible from the current docs. This is a really good place to be, relatively speaking, as it means a lot of progress has been made: the stuff works and it's actually documented, too! It's just buried in reference docs and developer guides for people who actively *want* to use the tools, and not pulled out in a quick reference form good enough for busy folks who just want to *use* a package that just happens to be based on setuptools.
Phillip J. Eby wrote:
At 01:31 PM 2/9/2006 -0600, Robert Kern wrote:
Andrew Straw wrote:
Note that matplotlib tried essentially this for a while, but apparently some folks really didn't like it. I'm not sure what exactly broke on their systems (they didn't complain to the mailing list), but when setup.py reverted to a plain distutils script, they cheered.
That's right, they complained to me, instead. :-)
The issue was that they had setuptools on their system because they were building eggs for another project, but specifically *didn't* want to install matplotlib as an egg.
setuptools-based packages can be forced to install the old-fashioned way using:
setup.py install --single-version-externally-managed
as long as you also specify a --root directory or a --record file. This is of course not upgradeable or uninstallable without help from a packaging tool that can utilize the results of --root or --record.
And this particular user did not want that, either. -- Robert Kern robert.kern@gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter
At 02:10 PM 2/9/2006 -0600, Robert Kern wrote:
Phillip J. Eby wrote:
At 01:31 PM 2/9/2006 -0600, Robert Kern wrote:
Andrew Straw wrote:
Note that matplotlib tried essentially this for a while, but apparently some folks really didn't like it. I'm not sure what exactly broke on their systems (they didn't complain to the mailing list), but when setup.py reverted to a plain distutils script, they cheered.
That's right, they complained to me, instead. :-)
The issue was that they had setuptools on their system because they were building eggs for another project, but specifically *didn't* want to install matplotlib as an egg.
setuptools-based packages can be forced to install the old-fashioned way using:
setup.py install --single-version-externally-managed
as long as you also specify a --root directory or a --record file. This is of course not upgradeable or uninstallable without help from a packaging tool that can utilize the results of --root or --record.
And this particular user did not want that, either.
I'm confused. They didn't want to be able to uninstall? Didn't want to point --record to /dev/null, or --root to /? Or something else?
Phillip J. Eby wrote:
At 02:10 PM 2/9/2006 -0600, Robert Kern wrote:
Phillip J. Eby wrote:
setuptools-based packages can be forced to install the old-fashioned way using:
setup.py install --single-version-externally-managed
as long as you also specify a --root directory or a --record file. This is of course not upgradeable or uninstallable without help from a packaging tool that can utilize the results of --root or --record.
And this particular user did not want that, either.
I'm confused. They didn't want to be able to uninstall? Didn't want to point --record to /dev/null, or --root to /? Or something else?
Didn't want setuptools involved at all. He's wasted more hours on it than he ever really wanted to in the days before non-root installs were reasonably documented. He just doesn't trust it. He only has setuptools installed at all because I've convinced him to distribute eggs of his project (and that I will help him troubleshoot setuptools issues). -- Robert Kern robert.kern@gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter
Robert Kern wrote:
setuptools-based packages can be forced to install the old-fashioned way using:
setup.py install --single-version-externally-managed
as long as you also specify a --root directory or a --record file. This is of course not upgradeable or uninstallable without help from a packaging tool that can utilize the results of --root or --record.
And this particular user did not want that, either.
I'm confused. They didn't want to be able to uninstall? Didn't want to point --record to /dev/null, or --root to /? Or something else?
Didn't want setuptools involved at all. He's wasted more hours on it than he ever really wanted to in the days before non-root installs were reasonably documented. He just doesn't trust it. He only has setuptools installed at all because I've convinced him to distribute eggs of his project (and that I will help him troubleshoot setuptools issues).
Setuptools and people using setuptools can only deal with actual issues, not with vague dislikes. This is open source, issues get resolved, but there's never any guarantee that issues don't *exist*. Without --single-version-externally-managed setuptools acts in ways that can be difficult to handle if .pth files don't work, or you are installing to an odd location, or something like that. It would be nice if all those issues can be documented and have useful/easy-to-resolve failure cases, and all that -- but it's not quite there yet, in part because it's not completely clear how to make all that work. *With* --single-version-externally-managed, it acts just like distutils always has. Right now one project of mine isn't using setuptools exclusively, because it still has Python 2.2 compatibility. But I see no reason to do all sorts of fiddling to support multiple installation techniques, and deal with tensions about what features can and can't be used because of multiple supported installation processes. Right now, the question should be: * Can we provide an experience similar enough to distutils to make conservative users happy? If they weren't happy with distutils, then whatever. I think --single-version-externally-managed does that. * Are we documenting this properly? That includes, are they going to be aware of the appropriate documentation at the right time in their experience. So, knowing about Python setup options before they start. Knowing about setuptools installation issues when they are using ez_setup.py. Knowing about package finding when they are using easy_install, or what they need to understand about eggs and .pth files during actual installation. * Are we providing a cohesive experience for the users who want something better than the previous status quo? Are we guiding them towards good practices? I've started using setuptools features even in packages that don't use plugins or any Egg features. A lot of these will work with distutils, but fail in weird or confusing ways (e.g., package data missing). Or I'll have to maintain an easy and an explicit way to do these things, and they'll get out of sync. I don't think it's going to be a particularly good experience for the conservative users, and it won't accomplish anything. At least when we struggle with setuptools it accomplishes something. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org
Ian Bicking wrote:
Setuptools and people using setuptools can only deal with actual issues, not with vague dislikes. This is open source, issues get resolved, but there's never any guarantee that issues don't *exist*.
Hiya! I really like setuptools and the eggs and stuff. I want to help you out by giving you a specific user case. I don't fully understand setuptools, so I apologize in advanced for any errors. I am a system administrator for several systems. I use a distro to manage most of my software, but obviously some python modules don't come with my distro (too new or the distro doesn't have it). I usually solve this by doing the following as a non-root user: |$ /usr/bin/python setup.py install --prefix=/usr/local/stow/projectname-version $ cd /usr/local/stow $ stow -D projectname-oldversion # optional -- removes old version $ stow projectname-version| This allows me to keep the previous version around in case I need to roll back. Here is a (trunicated) tree listing from trac project: /usr/local/stow/trac-svn-r2824 |-- bin |-- lib | `-- python2.3 | `-- site-packages | `-- trac `-- share |-- man | `-- man1 `-- trac Right now my workaround is to use bdist_egg and then copy the egg into the directory /usr/local/stow/project-version/lib/python2.4/site-packages (which I have to mkdir first). This isn't perfect because I don't get non-egg stuff, if any exists. I hope that this helps you. ^_^ Ciao! -- Probable-Possible, my black hen, She lays eggs in the Relative When. She doesn't lay eggs in the Positive Now Because she's unable to postulate How. -- Frederick Winsor The Doctor What: Need I say more? http://docwhat.gerf.org/ docwhat *at* gerf *dot* org KF6VNC
Christian Holtje wrote:
Ian Bicking wrote:
Setuptools and people using setuptools can only deal with actual issues, not with vague dislikes. This is open source, issues get resolved, but there's never any guarantee that issues don't *exist*.
Hiya! I really like setuptools and the eggs and stuff. I want to help you out by giving you a specific user case. I don't fully understand setuptools, so I apologize in advanced for any errors.
I am a system administrator for several systems. I use a distro to manage most of my software, but obviously some python modules don't come with my distro (too new or the distro doesn't have it).
I usually solve this by doing the following as a non-root user: |$ /usr/bin/python setup.py install --prefix=/usr/local/stow/projectname-version $ cd /usr/local/stow $ stow -D projectname-oldversion # optional -- removes old version $ stow projectname-version|
This allows me to keep the previous version around in case I need to roll back.
The discussion in "A prefix option?" and "DWIM installation with setuptools" is, I think, about similar issues. Well, depending on your granularity. If it is just installation, then setuptools doesn't overwrite old versions, it basically just "activates" a particular installed version of a library by changing easy-install.pth, and to a degree with .egg-link files. I think .egg-link could be used to even more effect, creating something like stow, except without relying on symlinks specifically. Instead of a series of symlinks like stow uses, setuptools changes the package loading and creates one reference to the installed version. So maybe this is already allowed. There aren't (yet) very good tools for actually switching versions and viewing the available installed versions of packages, but the actual mechanism to do so is fairly straight-forward (just edit that easy-install.pth file, view using a wildcard, remove with a delete). The other stuff we've been talking about is creating an environment that contains all of the packages for some "project" -- maybe a particular web application, a user's local (non-root) Python installation, or whatever.
Here is a (trunicated) tree listing from trac project: /usr/local/stow/trac-svn-r2824 |-- bin |-- lib | `-- python2.3 | `-- site-packages | `-- trac `-- share |-- man | `-- man1 `-- trac
Right now my workaround is to use bdist_egg and then copy the egg into the directory /usr/local/stow/project-version/lib/python2.4/site-packages (which I have to mkdir first). This isn't perfect because I don't get non-egg stuff, if any exists.
There shouldn't be non-egg stuff; if there is, then the package isn't really a good setuptools package. I think setuptools catches attempts to write files elsewhere too, and at least warns about it. While I can understand you want to stick to a stow-based system, at this point stow seems unnecessarily indirect. However, you can always give the -d option or one of the other available options to have easy_install put the egg right where you want it. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org
Ian Bicking wrote:
There shouldn't be non-egg stuff; if there is, then the package isn't really a good setuptools package. I think setuptools catches attempts to write files elsewhere too, and at least warns about it. While I can understand you want to stick to a stow-based system, at this point stow seems unnecessarily indirect. However, you can always give the -d option or one of the other available options to have easy_install put the egg right where you want it.
I didn't see the -d option. That does most of what I need right there. Spiffy. I agree, that some sort of UI that I could manage eggs and versions with would be excellent and a happy replacement for stow. I would like to be able to keep these eggs out of the "distribution"'s directories, though. Since I don't want things being replaced on upgrade, etc. Ciao! -- Fallacy #1 You can't manage what you can't measure. -- "Facts and Fallacies of Software Engineering" (Robert L. Glass, 2002) The Doctor What: A Holtje Production http://docwhat.gerf.org/ docwhat *at* gerf *dot* org KF6VNC
Christian Holtje wrote:
Ian Bicking wrote:
There shouldn't be non-egg stuff; if there is, then the package isn't really a good setuptools package. I think setuptools catches attempts to write files elsewhere too, and at least warns about it. While I can understand you want to stick to a stow-based system, at this point stow seems unnecessarily indirect. However, you can always give the -d option or one of the other available options to have easy_install put the egg right where you want it.
I didn't see the -d option. That does most of what I need right there. Spiffy.
I agree, that some sort of UI that I could manage eggs and versions with would be excellent and a happy replacement for stow.
It also occurred to me that --single-version-externally-managed describes exactly what stow implies (stow managing the version). But yes, you don't have to use stow to have stow-like isolation.
I would like to be able to keep these eggs out of the "distribution"'s directories, though. Since I don't want things being replaced on upgrade, etc.
You can enforce that by putting your preferred installation options in /usr/lib/python2.4/distutils/distutils.cfg (sadly you have to put the configuration in that location or your home directory in ~/.pydistutilssomething). -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org
At 10:38 AM 2/9/2006 -0800, Andrew Straw wrote:
Note that matplotlib tried essentially this for a while, but apparently some folks really didn't like it. I'm not sure what exactly broke on their systems (they didn't complain to the mailing list)
I wish more people would complain in useful ways; at least the blog post that started this thread was *specific*, even if it was directed to the wrong forum.
On Feb 9, 2006, at 10:38 AM, Andrew Straw wrote:
This way, setup.py can be setuptools-aware without doing 'import setuptools', but the user would have to do: python -c "import setuptools; execfile('setup.py')"
Maybe we should get an easy_setup that does this? easy_install is fine if you want to install, but not if you want to bdist_egg or use any of the other setuptools-specific features. -bob
participants (6)
-
Andrew Straw -
Bob Ippolito -
Christian Holtje -
Ian Bicking -
Phillip J. Eby -
Robert Kern