Egg support for system packages (including bdist_wininst)
Setuptools in SVN now provides preliminary support for installing eggs in .egg-info directory style, so that setuptools-based projects can be wrapped by system packagers who wish to avoid using .egg files or directories. In addition, you can now use setuptools to install non-setuptools packages in such a way that an .egg-info directory is included, so that setuptools-based packages can detect the presence of the non-setuptools-based package. If you are building packages using a distutils command such as 'bdist_wininst' that internally invokes the distutils "install" command, you do not need to do anything special. Setuptools detects when the "install" command is invoked programmatically (via run_command()) and automatically enters its backward-compatibility mode. However, if you are using a command such as 'bdist_rpm' that generates external scripts which then run "setup.py install", you will need to modify the generated scripts to run this instead: setup.py install --single-version-externally-managed You will also need to include the --root or --record options. For the 'bdist_rpm' command, setuptools automatically adjusts the 'spec' files before they're written, so bdist_rpm *should* work correctly, although I haven't tested it yet. Please let me know if you find any problems, as I have not yet tested this installation mode with anything but bdist_wininst or direct manual installation. For persons like Paul Moore who want to wrap egg-based packages in Windows installers, note that you can now take a setuptools-based package's source and run "setup.py bdist_wininst" and it will build a usable .exe. In addition, you can build Windows installer-wrapped eggs for non-setuptools packages by changing your command line from: python setup.py bdist_wininst to: python -c "import setuptools;execfile('setup.py')" bdist_wininst This will build an .exe that includes .egg-info, so that other packages will be able to detect it at runtime. Similarly, if you are building any other kind of system package using "python setup.py install", you can force .egg-info to be included by using this instead: python -c "import setuptools;execfile('setup.py')" install \ --single-version-externally-managed ... This support for system packages of eggs is still early and experimental; please let me know about any problems you may encounter or any questions you may have. Thanks.
On 12/15/05, Phillip J. Eby <pje@telecommunity.com> wrote:
For persons like Paul Moore who want to wrap egg-based packages in Windows installers, note that you can now take a setuptools-based package's source and run "setup.py bdist_wininst" and it will build a usable .exe.
In addition, you can build Windows installer-wrapped eggs for non-setuptools packages by changing your command line from:
python setup.py bdist_wininst
to: python -c "import setuptools;execfile('setup.py')" bdist_wininst
This will build an .exe that includes .egg-info, so that other packages will be able to detect it at runtime.
Massively cool! Thank you for putting this in place. Paul.
Phillip J. Eby wrote:
Setuptools in SVN now provides preliminary support for installing eggs in .egg-info directory style, so that setuptools-based projects can be wrapped by system packagers who wish to avoid using .egg files or directories. In addition, you can now use setuptools to install non-setuptools packages in such a way that an .egg-info directory is included, so that setuptools-based packages can detect the presence of the non-setuptools-based package.
Thanks for adding this. I'd just wish that this would be the default and the .egg ZIP file installation approach be made an option. setuptools would then finally be compatible to the rest of the distutils world again and avoid all the added overhead and problems of ZIP file imports. Perhaps you could have two commands, e.g. the default install would create the normal package directory (with added .egg-info dir) and a new install_eggfile to install the .egg ZIP file instead. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 22 2005)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
M.-A. Lemburg wrote:
Phillip J. Eby wrote:
Setuptools in SVN now provides preliminary support for installing eggs in .egg-info directory style, so that setuptools-based projects can be wrapped by system packagers who wish to avoid using .egg files or directories. In addition, you can now use setuptools to install non-setuptools packages in such a way that an .egg-info directory is included, so that setuptools-based packages can detect the presence of the non-setuptools-based package.
Thanks for adding this.
I'd just wish that this would be the default and the .egg ZIP file installation approach be made an option.
setuptools would then finally be compatible to the rest of the distutils world again and avoid all the added overhead and problems of ZIP file imports.
Perhaps you could have two commands, e.g. the default install would create the normal package directory (with added .egg-info dir) and a new install_eggfile to install the .egg ZIP file instead.
Somewhat related to this: wouldn't it be possible to build .egg ZIP files using the same layout, ie. with the .egg-info subdir inside the package instead of having a top-level EGG-INFO directory in the ZIP file ?! That way, you could simply unzip the .egg file somewhere on your PYTHONPATH in order to install it and still have the meta information at hand. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 22 2005)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
At 01:52 PM 12/22/2005 +0100, M.-A. Lemburg wrote:
Somewhat related to this: wouldn't it be possible to build .egg ZIP files using the same layout, ie. with the .egg-info subdir inside the package instead of having a top-level EGG-INFO directory in the ZIP file ?!
That way, you could simply unzip the .egg file somewhere on your PYTHONPATH in order to install it and still have the meta information at hand.
And the advantage of manually doing this over having easy_install do it would be...?
Phillip J. Eby wrote:
At 01:52 PM 12/22/2005 +0100, M.-A. Lemburg wrote:
Somewhat related to this: wouldn't it be possible to build .egg ZIP files using the same layout, ie. with the .egg-info subdir inside the package instead of having a top-level EGG-INFO directory in the ZIP file ?!
That way, you could simply unzip the .egg file somewhere on your PYTHONPATH in order to install it and still have the meta information at hand.
And the advantage of manually doing this over having easy_install do it would be...?
* No magic, whatsoever. * No dependencies on a specific package manager. * No commands to learn. * No cluttering up PYTHONPATH. * No mangling of sys.path at runtime. * No Python installation slowdown due to frequent scans of ZIP files. I guess what I want to say is: eggs would become a no-strings-attached, easy to use drop-in install format. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 22 2005)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
At 10:39 PM 12/22/2005 +0100, M.-A. Lemburg wrote:
Phillip J. Eby wrote:
At 01:52 PM 12/22/2005 +0100, M.-A. Lemburg wrote:
Somewhat related to this: wouldn't it be possible to build .egg ZIP files using the same layout, ie. with the .egg-info subdir inside the package instead of having a top-level EGG-INFO directory in the ZIP file ?!
That way, you could simply unzip the .egg file somewhere on your PYTHONPATH in order to install it and still have the meta information at hand.
And the advantage of manually doing this over having easy_install do it would be...?
* No magic, whatsoever. * No dependencies on a specific package manager. * No commands to learn. * No cluttering up PYTHONPATH. * No mangling of sys.path at runtime. * No Python installation slowdown due to frequent scans of ZIP files.
I guess what I want to say is: eggs would become a no-strings-attached, easy to use drop-in install format.
Perhaps I was unclear. My question is what benefit there was, compared to having easy_install (or any other tool) do the unzipping and rename EGG-INFO to Package-Version.egg-info? Note also that unzipping an egg to a directory with the same name as its .egg file produces a similar result, but has the additional advantage of supporting a trivial uninstallation - with no package manager required for either installation or uninstallation. So, the use case of "just unzip and go" actually works already, if you either manually add it to a .pth file or just use require() at runtime.
At 12:48 PM 12/22/2005 +0100, M.-A. Lemburg wrote:
I'd just wish that this would be the default and the .egg ZIP file installation approach be made an option.
setuptools would then finally be compatible to the rest of the distutils world again and avoid all the added overhead and problems of ZIP file imports.
Perhaps you could have two commands, e.g. the default install would create the normal package directory (with added .egg-info dir) and a new install_eggfile to install the .egg ZIP file instead.
The only way this could happen is to add a manifest of installed files, in order to allow uninstallation and upgrades to be made safe in the absence of a packaging system. The .egg file/directory technique currently makes uninstallation as simple as "rm -rf package-version.egg", and multi-version installs work with no special action. Note, too, that using .egg files or directories means that installation *never breaks existing programs* as long as they have specified their dependencies accurately. Traditional distutils installation is destructive in the absence of a system packager. However, if you *have* a system packager, then you should not be using "setup.py install" anyway, so I'm not sure I see how this is an issue. You should be simply building system packages. And it's quite straightforward to make most bdist_* commands work: bdist_wininst and bdist_dumb work with setuptools "out of the box" because they call the "install" command internally; only bdist_rpm required some monkeypatching to add --single-version-externally-managed, because it writes script files that run "install" later. Anyway, I guess what I'm saying is, if you want traditional distutils-style installation, you should use system packages, since they're the only way to allow uninstallation and avoid corrupted upgrades. Otherwise, you should use .egg files or directories, in which case easy_install (and in the future, nest) will be your package manager.
Phillip J. Eby wrote:
At 12:48 PM 12/22/2005 +0100, M.-A. Lemburg wrote:
I'd just wish that this would be the default and the .egg ZIP file installation approach be made an option.
setuptools would then finally be compatible to the rest of the distutils world again and avoid all the added overhead and problems of ZIP file imports.
Perhaps you could have two commands, e.g. the default install would create the normal package directory (with added .egg-info dir) and a new install_eggfile to install the .egg ZIP file instead.
The only way this could happen is to add a manifest of installed files, in order to allow uninstallation and upgrades to be made safe in the absence of a packaging system.
Sure, why not put this manifest into the .egg-info dir ?! The advantage of the .egg-info dir is that it's easy to find using package introspection.
The .egg file/directory technique currently makes uninstallation as simple as "rm -rf package-version.egg", and multi-version installs work with no special action.
Note, too, that using .egg files or directories means that installation *never breaks existing programs* as long as they have specified their dependencies accurately. Traditional distutils installation is destructive in the absence of a system packager.
However, if you *have* a system packager, then you should not be using "setup.py install" anyway, so I'm not sure I see how this is an issue.
The issue is that "setup.py install" should continue to install things the (one-and-only) Python way. Any different way should use a different command name. You are currently overloading the install command with a completely different approach, which is not compatible to the standard distutils system or the Python import mechanism.
You should be simply building system packages. And it's quite straightforward to make most bdist_* commands work: bdist_wininst and bdist_dumb work with setuptools "out of the box" because they call the "install" command internally; only bdist_rpm required some monkeypatching to add --single-version-externally-managed, because it writes script files that run "install" later.
Anyway, I guess what I'm saying is, if you want traditional distutils-style installation, you should use system packages, since they're the only way to allow uninstallation and avoid corrupted upgrades. Otherwise, you should use .egg files or directories, in which case easy_install (and in the future, nest) will be your package manager.
I guess what I want to say is that I only want to be forced to use the easy_install package manager (or any other non-system package manager) if I have a need or requirement for the features it offers. The egg format is not too far away from being a nice drop-in format for Python binary extensions. All it takes is: * making sure that an unzip will create a proper Python package (with the meta information embedded into it) With this changes, I think that eggs could actually become a prime distribution format - even for system extensions since it then no longer interferes with the system installer mechanisms. BTW, in case you still don't believe that placing ZIP files on PYTHONPATH causes a slowdown, just install 10-20 eggs on PYTHONPATH and then do a python -v Python will tell you that it scanned all the ZIP files on startup - even before having imported any of the system lib modules such as warnings. Note that disabling the import of site.py doesn't change this (python -S -v). Here's an example with 20 eggs (using C shell): tmp/eggs/> setenv PYTHONPATH egg1.egg:egg10.egg:egg11.egg:egg12.egg:egg13.egg:egg14.egg:egg15.egg:egg16.egg:egg17.egg:egg18.egg:egg19.egg:egg2.egg:egg20.egg:egg3.egg:egg4.egg:egg5.egg:egg6.egg:egg7.egg:egg8.egg:egg9.egg tmp/eggs> time python -S -c '0' 0.014u 0.006s 0:00.02 50.0% 0+0k 0+0io 0pf+0w tmp/eggs> unsetenv PYTHONPATH tmp/eggs> time python -S -c '0' 0.006u 0.003s 0:00.01 0.0% 0+0k 0+0io 0pf+0w System time for startup doubles. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 22 2005)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
At 11:22 PM 12/22/2005 +0100, M.-A. Lemburg wrote:
Phillip J. Eby wrote:
At 12:48 PM 12/22/2005 +0100, M.-A. Lemburg wrote:
I'd just wish that this would be the default and the .egg ZIP file installation approach be made an option.
setuptools would then finally be compatible to the rest of the distutils world again and avoid all the added overhead and problems of ZIP file imports.
Perhaps you could have two commands, e.g. the default install would create the normal package directory (with added .egg-info dir) and a new install_eggfile to install the .egg ZIP file instead.
The only way this could happen is to add a manifest of installed files, in order to allow uninstallation and upgrades to be made safe in the absence of a packaging system.
Sure, why not put this manifest into the .egg-info dir ?!
Because then people will need a special tool to uninstall it. I'm kind of confused because in one email you argue against having easy_install as a package manager, and then you seem to be arguing for making more package management tools. Which would you prefer? :) Here are the use cases as I see them: Just unzip an egg and use it: Sure, just unzip it as a directory with the .egg name and add it to sys.path or use require(). Flat layout, short sys.path: Use a system package manager like bdist_rpm or bdist_wininst, or some other bdist wrapper. No package manager, or other use cases: Use easy_install as your package manager. Looks to me like everything is covered here.
The advantage of the .egg-info dir is that it's easy to find using package introspection.
I don't understand you. Package != distribution. .egg-info directories go with a project's distribution, and have nothing to do with packages in the import sense. A distribution might contain only a module, for example. So there is no connection between a package and its .egg-info in the sense you seem to be implying.
The .egg file/directory technique currently makes uninstallation as simple as "rm -rf package-version.egg", and multi-version installs work with no special action.
Note, too, that using .egg files or directories means that installation *never breaks existing programs* as long as they have specified their dependencies accurately. Traditional distutils installation is destructive in the absence of a system packager.
However, if you *have* a system packager, then you should not be using "setup.py install" anyway, so I'm not sure I see how this is an issue.
The issue is that "setup.py install" should continue to install things the (one-and-only) Python way.
Which is currently broken without a package manager for uninstallation or upgrade. Your argument implies that: 1. Broken things should not be fixed or improved, ever 2. One must never have more than one implementation of an interface The first one needs no further comment; the second negates the entire purpose of having an interface. (In case it's unclear, I mean that "setup.py install" is an interface that simply promises to make the project available for use, not how or in what format.)
Any different way should use a different command name.
You are currently overloading the install command with a completely different approach, which is not compatible to the standard distutils system or the Python import mechanism.
Please define "compatible" and explain why you believe the above to be true, providing adequate technical detail to demonstrate the supposed incompatibility. When you have enough information to do that, you'll see why it's not true. You'll save some time, however, if you first investigate the 'extra_path' option to 'setup()' that is provided by the distutils. It is well within the power of a normal "setup.py install" to produce *precisely* the same results as a setuptools "setup.py install", and I know this for a fact because early versions of setuptools actually *used* it to bootstrap the installation of setuptools itself as an egg! In other words, there is nothing "completely different" going on here. Nothing. This "non-standard" and "not compatible" nonsense is pure FUD, unless of course the distutils are non-standard and not compatible with themselves. :)
Anyway, I guess what I'm saying is, if you want traditional distutils-style installation, you should use system packages, since they're the only way to allow uninstallation and avoid corrupted upgrades. Otherwise, you should use .egg files or directories, in which case easy_install (and in the future, nest) will be your package manager.
I guess what I want to say is that I only want to be forced to use the easy_install package manager (or any other non-system package manager) if I have a need or requirement for the features it offers.
Which is fine: use your system packager. But people who don't have system packagers need a way to handle upgrades and uninstallation safely - and that *should* be the default behavior. I don't give a flying hoot what the distutils default behavior is; only what it *should* be, which is to say *safe*. The current default behavior is not safe. Resorting to "it's the standard" as an argument therefore carries zero weight for me. Nobody should be using unadorned distutils installation, except for the Python build process and tools that build system packages or other "bdist" formats. Therefore, setuptools does not provide it as a default behavior. Yes, setuptools is valuing safety over performance. If you prefer to be unsafe, you'll have to read the docs and use a command line option to get it the other way. If you want to be fast *and* safe, you need a system package manager.
The egg format is not too far away from being a nice drop-in format for Python binary extensions.
All it takes is:
* making sure that an unzip will create a proper Python package (with the meta information embedded into it)
It does this now; unzip to a directory with a .egg name and add that directory to sys.path.
With this changes, I think that eggs could actually become a prime distribution format - even for system extensions since it then no longer interferes with the system installer mechanisms.
If this is the *only* thing stopping you from liking eggs, I'll certainly consider adding it in 0.7. There may be some backward compatibility issues, in that eggs built with 0.7 would then *not* be usable by older versions of the pkg_resources module, which would not know to look for .egg-info *inside* a zipped egg. It seems to me that it would be much cleaner to have a script that take an .egg and do the equivalent of "install --single-version-externally-managed" with it, than to change the format to support using an arbitrary "unzip" utility. Such an arbitrary utility also won't have any way to uninstall or upgrade the files, either.
Here's an example with 20 eggs (using C shell):
tmp/eggs/> setenv PYTHONPATH egg1.egg:egg10.egg:egg11.egg:egg12.egg:egg13.egg:egg14.egg:egg15.egg:egg16.egg:egg17.egg:egg18.egg:egg19.egg:egg2.egg:egg20.egg:egg3.egg:egg4.egg:egg5.egg:egg6.egg:egg7.egg:egg8.egg:egg9.egg tmp/eggs> time python -S -c '0' 0.014u 0.006s 0:00.02 50.0% 0+0k 0+0io 0pf+0w tmp/eggs> unsetenv PYTHONPATH tmp/eggs> time python -S -c '0' 0.006u 0.003s 0:00.01 0.0% 0+0k 0+0io 0pf+0w
System time for startup doubles.
If I'm reading this correctly, that's 1/100th of a second to scan 20 eggs. If that's a linear progression, one could have *every single project listed on PyPI* as an egg on PYTHONPATH simultaneously, with only 54/100ths of a second being added to startup. That actually sounds *really* good to me for anything but short scripts! Note too that setuptools encourages the use of "virtual" Python installations for doing software development; this makes it really easy to install Python modules for a single project through having its own virtual Python. It's likely that the system Python will have system packages (not affecting sys.path length), while user-installed packages will be more likely to go in a user-private virtual Python or extended PYTHONPATH anyway. Meanwhile, the short scripts will likely use the system Python, or else you'll set up a stripped-down virtual Python just for those scripts if you need one. Since setuptools now supports system packaging, there's no longer any reason to treat sys.path length as an argument against using setuptools. The only reason to use easy_install in the first place is for things your packaging system can't supply, e.g. because a package is too new, or because you're a non-root user who can't install system packages anyway, or because you have application instances with different package needs, you need isolated Python installations for testing, etc. etc. etc. As a general rule, system packagers don't support *any* of these things, so easy_install is really your only practical choice. That's why it exists.
participants (3)
-
M.-A. Lemburg -
Paul Moore -
Phillip J. Eby