Unravelling installation schemes
I'm working on implementing distutils dependencies, as described at: http://www.python.org/cgi-bin/moinmoin/DistutilsDependencies and I've run into a bit of a hitch. In order to run dependencies' setup scripts, I need to be able to pass along command line options from the parent setup script. But, it's not quite that simple. The distutils framework allows people to specify quite a few options at multiple levels. For example, you can specify a different C compiler to the build_lib and build_ext commands, and a third one to the build command itself! Next, there's path munging and alternate installation schemes, along with the 'extra_path' support, install-lib overriding purelib and platlib, the distribution name embedded in the install-headers path, and so on. My brain is approaching critical mass for explosion. :) Now, add to this some additional requirements that are specific to dependency management. I need to know where stuff is being installed, so that when I check for the existence of a dependency, I'm not fooled by it not being on sys.path currently. And, what if a dependency exists on extra_path? [sob, sniff] I guess I'm beginning to see why nobody else has tackled this up to now. :) Of course, for maybe half the installs it would suffice to just run 'setup.py install' on the dependencies. For about half of the remaining situations, it would probably suffice to just pass along whatever options were specified on the parent setup script's command line, verbatim. It's the remaining scenarios that are tough: * Some joker uses a relative path on the command line (could fix, if we know which arguments are directories) * Non-build/non-install commands (could maybe fix by only copying options for build, install, and subcommands thereof) * Somebody uses an explicit --install-lib or install_lib --install-dir (fixable, but it could screw things up if they really should've been specifying platlib or purelib or one of the higher-level options) * Somebody uses an explicit install --install-headers or install_headers --install-dir (aaagh!) So far, I'm thinking that about the only thing to do is: 1. Go through all the command-line options for install, build, and their subcommands 2. If *any* '--install-X' option exists apart from 'install-base' or 'install-platbase', raise holy heck and stop the gravy train when trying to install dependencies. Ask the user to either change their options, or to manually run the child setup with appropriate options, specifying where it is and what to do. 3. Only pass recognized-to-be safe options from build and its subcommands. (Which is probably going to equate to just the compiler option, include-) Fix up directory options from the install command to be absolute. Barf if an inplace build_ext was specified. 4. Don't use any options that came from config files. If it was a global config, it'll apply to the child setup automatically. If it came from the local setup.cfg, it *shouldn't* apply to the child. Whew. So here's the thing... am I making this too hard? Is there a better way to deal with options to child setups? I look at all the zillions of options to the individual commands, and I think about when and why I might specify those options, I think I mostly *wouldn't* want them passing through to the child setups. Maybe the simplest thing to do would be to make a list of "safe" command-line options that will be passed to children, and issue warnings for the rest? Or maybe stop the build if any options would be ignored, unless a "go ahead anyway" option is given? What do y'all think? I guess I'm now leaning towards that last option... simply dump out on-screen all the command-line arguments that aren't safe, and show what *would* be passed to the child setup, and show what arguments would be needed to provide a command line that will force the build to proceed anyway, and also how to run the child setup manually (since the target will have already been downloaded and extracted to the temporary directory). This won't address complex installation options, but then, if somebody has complex installation options, they can probably deal with a little complexity for installing the dependencies, I guess. There are a few other issues floating about, mostly to do with sys.path, .pth files, and installation to locations other than site-packages. But I think I'll leave those thoughts to another post. :)
On Tue, Mar 09, 2004 at 06:57:49PM -0500, Phillip J. Eby wrote:
I guess I'm beginning to see why nobody else has tackled this up to now. :)
My $.02 is, Distutils can take one of 2 forms. It can be automake/autoconf for Python and everyone can install from source, or it can be a package manager. The battle you're fighting seems to me to be from trying to make it do both. If Distutils can make binary packages for most package managers, and it can be extended to include "Provides" and "Requires" in the meta-data, then any OS's native package manager should have all the information it needs to do the right thing at install time.
Of course, for maybe half the installs it would suffice to just run 'setup.py install' on the dependencies. For about half of the remaining situations, it would probably suffice to just pass along whatever options were specified on the parent setup script's command line, verbatim.
Why does everyone seem to think that the prefered method of Distutils package installations is "setup.py install"? What's the point of providing bdist_* commands if we're going to perpetualy re-inforce the "setup.py install" mindset? I'm not trying to be confrontational here, but do we seriously expect everyone who wants to install Distutils packages to have a compiler suite in case a package has C extensions? Even without C extensions, I suspect you can rule out most Windows users by just telling them to "setup.py install". You'll be lucky if you can get them to listen long enough when they ask you to explain how to pass "install" as an option when they click the setup.py icon. I really, really, think Distutils should _produce_binary_packages. For as many platforms and package managers as possible, with as much meta-data as each package manager supports. Then we can just let the package managers worry about dependencies and not re-invent another wheel.
It's the remaining scenarios that are tough:
* Some joker uses a relative path on the command line (could fix, if we know which arguments are directories)
* Non-build/non-install commands (could maybe fix by only copying options for build, install, and subcommands thereof)
* Somebody uses an explicit --install-lib or install_lib --install-dir (fixable, but it could screw things up if they really should've been specifying platlib or purelib or one of the higher-level options)
* Somebody uses an explicit install --install-headers or install_headers --install-dir (aaagh!)
No matter what you do, there will be people who are simply not good at packaging software. Where Distutils shines is with a minimal amount of configuration "setup.py bdist_whatever" will produce a binary usable by anyone with the whatever package manager, even if the developer created their setup.py on a completely different platform. This is the real magic. It's what distinguishes Distutils from CPAN.* You are taking quite a leap when you talk about supporting multiple binary packages from a single source package. I started by abstracting a generic bdist_packager that simply managed the PEP 241 meta-data so bdist_commands could rely on it for metadata information. Subpackages was an extension I never got around to. My point is, _nothing_ you can code will save a bad packager from themself. The penalty for producing bad packages is that no one uses them because they're a pain to install. OTOH, Distutils should do everything it can to make it simple to package a module directories into multiple inter-dependent components so it takes reall effort to screw up the packaging.
There are a few other issues floating about, mostly to do with sys.path, .pth files, and installation to locations other than site-packages. But I think I'll leave those thoughts to another post. :)
Personal whine :( I solved these issues on both Solaris and HP-UX by
making bdist_commands that produced packages that were relocatable by
the native package manager based on where the python installation was
installed on the target and any other criteria required. I'm sorry, I
_still_ waiting on my employer's approval to (re)submit them, but I
strongly believe _that's_ the approach to take. Using native package
managers intelligience and pre and post install scripts lets the person
_installing_ the package make these decisions. That person is the REAL
Distutils "customer". Helping developers make better packages is simply
the most effective way to improve the end user (package installer)
experience.
When Distutils generated packages are universally available via apt/yum
repositories, on the sunfreeware.com site and the HP Porting Archive
Center, _that_ will be day I consider Distutils a true success.
*I know... People point to the popularity of CPAN, which has NO
integration with package managers. My point exactly! If you rebuild a
box, your package manager knows everything that needs to be installed
_except_ stuff CPAN brought down. Please, let's not do this with
Distutils....
At 08:44 PM 3/9/04 -0500, Mark W. Alexander wrote:
On Tue, Mar 09, 2004 at 06:57:49PM -0500, Phillip J. Eby wrote:
I guess I'm beginning to see why nobody else has tackled this up to now. :)
My $.02 is, Distutils can take one of 2 forms. It can be automake/autoconf for Python and everyone can install from source, or it can be a package manager. The battle you're fighting seems to me to be from trying to make it do both.
Actually, it has a lot more to do with the path munging that the 'install' command does, which doesn't preserve a lot of the information in question, coupled with the fact that I'm producing a separate package, rather than hacking the distutils source directly.
If Distutils can make binary packages for most package managers, and it can be extended to include "Provides" and "Requires" in the meta-data, then any OS's native package manager should have all the information it needs to do the right thing at install time.
Well, it sounds like you have more experience with those package managers than I do, so it would be helpful if you could lay out a plan for this with sufficient detail to at least allow me to contemplate implementing it.
Of course, for maybe half the installs it would suffice to just run 'setup.py install' on the dependencies. For about half of the remaining situations, it would probably suffice to just pass along whatever options were specified on the parent setup script's command line, verbatim.
Why does everyone seem to think that the prefered method of Distutils package installations is "setup.py install"?
Er, perhaps because they prefer it? :) I know I do.
What's the point of providing bdist_* commands if we're going to perpetualy re-inforce the "setup.py install" mindset? I'm not trying to be confrontational here, but do we seriously expect everyone who wants to install Distutils packages to have a compiler suite in case a package has C extensions?
That's why I want the dependency system to support installing binary packages as dependencies, at least on Windows.
Even without C extensions, I suspect you can rule out most Windows users by just telling them to "setup.py install". You'll be lucky if you can get them to listen long enough when they ask you to explain how to pass "install" as an option when they click the setup.py icon.
Uh, wouldn't you be supplying your application to those people as a py2exe or other installable with all its dependencies self-contained?
I really, really, think Distutils should _produce_binary_packages. For as many platforms and package managers as possible, with as much meta-data as each package manager supports.
Fantastic, so we can count on your support for specifying what metadata is needed in a way that will work with all these package managers? :) Keep in mind that many developers don't have a clue about these package managers, perhaps not even the one for the system they use! If distutils requires the developer to understand every packaging system in order to release a package, it's useless. However, if all the information needed for packaging can be described in setup.cfg, I imagine most developers would accept patches from people with packaging tool experience to add metadata for the various package managers. Even then, I'm not sure how workable it is, given that dependencies would now need to include many more URLs for where to get all the differently-packaged versions of the packages.
On Wed, Mar 10, 2004 at 11:29:58AM -0500, Phillip J. Eby wrote:
If Distutils can make binary packages for most package managers, and it can be extended to include "Provides" and "Requires" in the meta-data, then any OS's native package manager should have all the information it needs to do the right thing at install time.
Well, it sounds like you have more experience with those package managers than I do, so it would be helpful if you could lay out a plan for this with sufficient detail to at least allow me to contemplate implementing it.
Somewhere I built a table mapping PEP 241 metadata to RPM, SD-UX (HP's manager), and pkgtool (Solaris). I could probably add dpkg now that I've got some experience there. (My HP is slowly rusting, but I still have code I can read.)
Why does everyone seem to think that the prefered method of Distutils package installations is "setup.py install"?
Er, perhaps because they prefer it? :) I know I do.
It's fine if you're only concerned about a machine or 2. If you've got several, having software managed by the native package manager is the ony thing that will keep you sane.
What's the point of providing bdist_* commands if we're going to perpetualy re-inforce the "setup.py install" mindset? I'm not trying to be confrontational here, but do we seriously expect everyone who wants to install Distutils packages to have a compiler suite in case a package has C extensions?
That's why I want the dependency system to support installing binary packages as dependencies, at least on Windows.
OK. I got distracted by your discussion of install commands and didn't see anything regarding binary packages.
Even without C extensions, I suspect you can rule out most Windows users by just telling them to "setup.py install". You'll be lucky if you can get them to listen long enough when they ask you to explain how to pass "install" as an option when they click the setup.py icon.
Uh, wouldn't you be supplying your application to those people as a py2exe or other installable with all its dependencies self-contained?
Actually, Windows is my weakest platform (sorry, no apologies either ;) I don't know how py2exe works, but I'd prefer a method where dependencies where resolved externally so every package that might require package X doesn't have to include package X. A Windows installer that could download and install dependencies would be nice.
I really, really, think Distutils should _produce_binary_packages. For as many platforms and package managers as possible, with as much meta-data as each package manager supports.
Fantastic, so we can count on your support for specifying what metadata is needed in a way that will work with all these package managers? :)
Absolutely! As I said, I had a decent start of an abstract packager used by implementations of bdist_pkgtool and bdist_sdux. I use them in-house, but my employer has not granted me permission to provide the Python Software Contributor agreement required for them to be included. If _anyone_ is interested. contact be off-line and I'll provide a detailed descriptiong that would allow a clean-room re-write.
Keep in mind that many developers don't have a clue about these package managers, perhaps not even the one for the system they use! If distutils requires the developer to understand every packaging system in order to release a package, it's useless.
This much is certain. Packaging is almost a distinct art from coding. Based on what I've done, and what I see that Marc has done with the Egenix packages, I think almost everything that's needed is in place. The only thing possibly missing is "requires" and "provides", and it's been so long since I reviewed a PEP, they may be there for all I know.
However, if all the information needed for packaging can be described in setup.cfg, I imagine most developers would accept patches from people with packaging tool experience to add metadata for the various package managers.
I think it's easier than that. There should be almost no difference in the actual metadata required by the different package managers out there. They just use different names for the fields.
Even then, I'm not sure how workable it is, given that dependencies would now need to include many more URLs for where to get all the differently-packaged versions of the packages.
Couldn't that be handled simply by each bdist_tool knowing how the package-name maps to the binary package name. Different package manager's all have different naming conventions, even if it's only a different extension. mwa -- Mark W. Alexander slash@dotnetslash.net The contents of this message authored by Mark W. Alexander are released under the Creative Commons Attribution-NonCommercial license. Copyright of quoted materials are retained by the original author(s). http://creativecommons.org/licenses/by-nc/1.0/
participants (2)
-
Mark W. Alexander
-
Phillip J. Eby