How we can get rid of eggs for 2.6 and beyond
So, after having some time to absorb the Python-Dev threads about setuptools, bootstrap, and all the rest, I think I see an opportunity to let people route around the "damage" of eggs, while still making it possible for the people who want to use easy_install or to put dependencies in their setup.py to get what they want, too. (And without them needing to install eggs, either.) At the same time, we can address the issues that remain around uninstalling packages, system vs. user packages, PYTHONPATH and site.py woes, and really pretty much every other complaint I've heard in the last few days about setuptools stomping on other people's stuff. (Even Paul's Windows issues, hopefully.) Now, you might be asking, "Okay, so why are you telling me about this? Why not just go fix setuptools?" Well, I *can't*. Not without some help from Python-Dev and the Distutils-SIG, to create an updated standard for installed package metadata, by updating PEP 262 ("A Database of Installed Python Packages") to include input from the system packaging folks, support for namespace packages, and support for setuptools-compatible dependency information. What's that got to do with anything? Well, without it, setuptools can't support uninstall or conflict management without using eggs to compartmentalize the installed files. And because it has to use eggs to do *that*, it has to munge .pth files and install its own site.py when installing to PYTHONPATH. All of this ugliness follows directly from the absence of a PEP 262-style installation database. Sure, setuptools could create its own version of this, and I almost did that four years ago. (If you look at the "open issues" part of PEP 262, you'll see my comments from back then.) I decided not to for two reasons: first, the distutils didn't support it yet, so it didn't help for conflict detection and avoidance in the real world at that point. Second, there were no uninstall tools for it, so I'd have had to write one myself. (Zed's "easy_f'ing_uninstall" to the contrary, it ain't easy, and I have an aversion to deleting stuff on people's systems without knowing what will break. There's a big difference between them typing 'rm -rf' themselves, and me doing it.) However, if tools exist and are distributed for such a "database", and *everybody* agrees to use it as an officially-blessed standard, then it should be possible for setuptools to co-exist with that framework, and we're all happy campers. In particular, the "installing eggs sucks" camp should be happy, because it'll be possible for me (or anyone else) to write a version of easy_install that doesn't install eggs any more, and setuptools-based packages can go back to having "setup.py install" install things the old way by default. So, to accomplish this, we (for some value of "we") need to: 1. Hash out consensus around what changes or enhancements are needed to PEP 262, to resolve the previously-listed open issues, those that have come up since (namespace packages, dependency specifications, canonical name/version forms), and anything else that comes up. 2. Update or replace the implementation as appropriate, and modify the distutils to support it in Python 2.6 and beyond. And "support it" means, "ensure that 'install' and *all* bdist commands update the database". The bdist_rpm, bdist_wininst, and bdist_msi commands, even bdist_dumb. (This should probably also include the add/remove programs stuff in the Windows case.) 3. Create a document for system packagers referencing the PEP and introducing them to what/why/how of the standard, in case they weren't one of the original participants in creating this. It will probably take some non-trivial work to do all this for Python 2.6, but it's probably possible, if we start now. I don't think it's critical to have an uninstall tool distributed with 2.6, as long as there's a reasonable way to bootstrap its installation later. Questions, comments... volunteers? :)
Phillip J. Eby wrote:
Questions, comments... volunteers? :)
This makes a lot of sense. I don't really have anything to add in terms of implementation, but I wonder if we can learn something from how apt or rpms or ports work, and how other programming languages (Ruby gems?) solve this. It's not an easy problem space since you're dealing with disparate and inconsistent target platforms, but it's one that has been solved before by others. Cheers, Martin
Phillip J. Eby wrote:
Second, there were no uninstall tools for it, so I'd have had to write one myself. (Zed's "easy_f'ing_uninstall" to the contrary, it ain't easy, and I have an aversion to deleting stuff on people's systems without knowing what will break. There's a big difference between them typing 'rm -rf' themselves, and me doing it.)
I think, the uninstall should _not_ 'rm -rf' but only 'rm' the files (and 'rmdir' directories, but not recursively) that it created, and that have not been modified in the meantime (after the installation). This can be easily achieved by recording a checksum (eg. md5 or sha) upon installation and only deleting a file if the checksum is correct and only deleting directories when they are empty (after the installed files in them have been deleted). Otherwise, the uninstall should complain and leave the modified files installed. Joachim
Joachim> I think, the uninstall should _not_ 'rm -rf' but only 'rm' the Joachim> files (and 'rmdir' directories, but not recursively) that it Joachim> created, and that have not been modified in the meantime (after Joachim> the installation). That's not sufficient. Suppose file C (e.g. /usr/local/etc/mime.types) is in both packages A and B. Install A - this will create C Install B - this might overwrite C, saving a copy, or it might retain A's copy. Uninstall B - this has to know that C is used by A and not touch it Skip
On Fri, Mar 21, 2008 at 11:21:49AM -0500, skip@pobox.com wrote:
Joachim> I think, the uninstall should _not_ 'rm -rf' but only 'rm' the Joachim> files (and 'rmdir' directories, but not recursively) that it Joachim> created, and that have not been modified in the meantime (after Joachim> the installation).
That's not sufficient. Suppose file C (e.g. /usr/local/etc/mime.types) is in both packages A and B.
Install A - this will create C Install B - this might overwrite C, saving a copy, or it might retain A's copy.
An install should never overwrite an existing file. The sane thing is to fail installing B by default (dpkg does this for example). An option to override this might be useful though. -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
skip@pobox.com writes:
Joachim> I think, the uninstall should _not_ 'rm -rf' but only 'rm' the Joachim> files (and 'rmdir' directories, but not recursively) that it Joachim> created, and that have not been modified in the meantime (after Joachim> the installation).
That's not sufficient. Suppose file C (e.g. /usr/local/etc/mime.types) is in both packages A and B.
Install A - this will create C Install B - this might overwrite C, saving a copy, or it might retain A's copy. Uninstall B - this has to know that C is used by A and not touch it
MacPorts has an expensive but interesting approach. When it finds a file used by another package, it backs it up to sth like $file.`date`.
At 11:21 AM 3/21/2008 -0500, skip@pobox.com wrote:
Joachim> I think, the uninstall should _not_ 'rm -rf' but only 'rm' the Joachim> files (and 'rmdir' directories, but not recursively) that it Joachim> created, and that have not been modified in the meantime (after Joachim> the installation).
That's not sufficient. Suppose file C (e.g. /usr/local/etc/mime.types) is in both packages A and B.
Install A - this will create C Install B - this might overwrite C, saving a copy, or it might retain A's copy. Uninstall B - this has to know that C is used by A and not touch it
Correct. However, in practice, B should not touch C, unless the file is shared between them. This is a key issue for support of namespace packages, at least if we want to avoid using .pth files. (Which is what setuptools-built system packages do for namespace packages currently.) Of course, one possible solution is for both A and B to depend on a "virtual package" that contains C, such that both A and B can install it if it's not there, and list it in their dependencies. But this is one of the handful of open issues that needs to be resolved with Real Life Package Management people, such as Debian, Fedora, etc. Neither overwriting, refusing to install, nor backups will properly address this issue. However, this is properly a topic for the Distutils-SIG or whatever SIG the actual spec goes to. On Python-Dev I'm only looking for a go/no-go on the overall approach.
On Fri, Mar 21, 2008 at 03:02:25PM -0400, Phillip J. Eby wrote:
At 11:21 AM 3/21/2008 -0500, skip@pobox.com wrote:
Joachim> I think, the uninstall should _not_ 'rm -rf' but only 'rm' the Joachim> files (and 'rmdir' directories, but not recursively) that it Joachim> created, and that have not been modified in the meantime (after Joachim> the installation).
That's not sufficient. Suppose file C (e.g. /usr/local/etc/mime.types) is in both packages A and B.
Install A - this will create C Install B - this might overwrite C, saving a copy, or it might retain A's copy. Uninstall B - this has to know that C is used by A and not touch it
Correct. However, in practice, B should not touch C, unless the file is shared between them.
This is a key issue for support of namespace packages, at least if we want to avoid using .pth files. (Which is what setuptools-built system packages do for namespace packages currently.)
Of course, one possible solution is for both A and B to depend on a "virtual package" that contains C, such that both A and B can install it if it's not there, and list it in their dependencies. But this is one of the handful of open issues that needs to be resolved with Real Life Package Management people, such as Debian, Fedora, etc.
Debian (dpkg) does not allow a file to exist in more then one package (distribution in distutils speak) as I said earlier. Directories can however, they will only be removed if they are empty (i.e. when the last package having that directory is uninstalled). I'm not familiar with namespace packages as I've never used one, but I assume it is just an empty directory with one (empty?) __init__.py file in it? The only way I can see Debian handling that is by having a Debian package providing exactly that __init__.py and all other Debian packages needing that namespace package will depend on it. However, this is shouldn't worry anyone other then Debian I think. If a distribution needs to depend on somthing it will depend on whoever provides the module *inside* the namespace package, not on the namespace itself. So the fact that no distribution provides the namespace.__init__.py file shouldn't have to worry other users of the python installdb. Since easy_install and others shouldn't be messing with debian packages in /usr/lib (they should be using /usr/local/lib or ~/.local [1]) this won't be a problem. For them to interoperate in their own module search paths they might need a virtual distribution though (which Debian could use as well then). Regards Floris PS: Thanks to Ben Finney for bringing the discussions here to the attention of debian-python@lists.debian.org. [1] I am aware of problems with this as explained in this thread: http://lists.debian.org/debian-python/2008/03/msg00021.html For the ~/.local, see PEP 370 mentioned here before. Basically the location of locally installed python modules might need to be revised. Could PEP 370 be extended to cover search paths for modules installed by local administrators? For the PEP262-style database this would probably indeed mean the need for a virtual package (err, distribution) in that database.
Neither overwriting, refusing to install, nor backups will properly address this issue. However, this is properly a topic for the Distutils-SIG or whatever SIG the actual spec goes to. On Python-Dev I'm only looking for a go/no-go on the overall approach.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
At 10:17 PM 3/21/2008 +0000, Floris Bruynooghe wrote:
On Fri, Mar 21, 2008 at 03:02:25PM -0400, Phillip J. Eby wrote:
At 11:21 AM 3/21/2008 -0500, skip@pobox.com wrote:
Joachim> I think, the uninstall should _not_ 'rm -rf' but
only 'rm' the
Joachim> files (and 'rmdir' directories, but not recursively) that it Joachim> created, and that have not been modified in the
meantime (after
Joachim> the installation).
That's not sufficient. Suppose file C (e.g. /usr/local/etc/mime.types) is in both packages A and B.
Install A - this will create C Install B - this might overwrite C, saving a copy, or it might retain A's copy. Uninstall B - this has to know that C is used by A and not touch it
Correct. However, in practice, B should not touch C, unless the file is shared between them.
This is a key issue for support of namespace packages, at least if we want to avoid using .pth files. (Which is what setuptools-built system packages do for namespace packages currently.)
Of course, one possible solution is for both A and B to depend on a "virtual package" that contains C, such that both A and B can install it if it's not there, and list it in their dependencies. But this is one of the handful of open issues that needs to be resolved with Real Life Package Management people, such as Debian, Fedora, etc.
Debian (dpkg) does not allow a file to exist in more then one package (distribution in distutils speak) as I said earlier. Directories can however, they will only be removed if they are empty (i.e. when the last package having that directory is uninstalled).
This is a general problem with system package managers, actually. Few allow a file to be provided by more than one package.
I'm not familiar with namespace packages as I've never used one, but I assume it is just an empty directory with one (empty?) __init__.py file in it?
It's not empty; it has to contain a namespace declaration. There's another approach that can work around the absence of the __init__.py, by using .pth files, but it's messy, and I'd like to get rid of it if possible.
The only way I can see Debian handling that is by having a Debian package providing exactly that __init__.py and all other Debian packages needing that namespace package will depend on it.
However, this is shouldn't worry anyone other then Debian I think.
Alas, it'll probably affect most package managers.
If a distribution needs to depend on somthing it will depend on whoever provides the module *inside* the namespace package, not on the namespace itself. So the fact that no distribution provides the namespace.__init__.py file shouldn't have to worry other users of the python installdb.
Well, somebody still has to create the __init__.py, and own it for purposes of uninstallation.
Since easy_install and others shouldn't be messing with debian packages in /usr/lib (they should be using /usr/local/lib or ~/.local [1]) this won't be a problem.
If the new PEP (262 redux) contains anything about actual directory locations on any system of any platform (other than as examples), I'll consider it a failure. Thinking about specific locations while talking about the new PEP should be considered a segmentation fault. :) The only thing that matters for the install db is where things are in relation to an installation target directory. Which reminds me -- the install db should support install-dir-relative paths, and probably should require them for anything that's installed under the library install dir. (Many use cases of this stuff will be *relative*; packaging an application as a giant tarball and unpacking it somewhere else on a system, or copying an application tree from point A to point B, should not render the data therein unusable.)
On Fri, Mar 21, 2008 at 06:30:33PM -0400, Phillip J. Eby wrote:
At 10:17 PM 3/21/2008 +0000, Floris Bruynooghe wrote:
On Fri, Mar 21, 2008 at 03:02:25PM -0400, Phillip J. Eby wrote:
At 11:21 AM 3/21/2008 -0500, skip@pobox.com wrote:
Joachim> I think, the uninstall should _not_ 'rm -rf' but only
'rm' the
Joachim> files (and 'rmdir' directories, but not recursively) that it Joachim> created, and that have not been modified in the
meantime (after
Joachim> the installation).
That's not sufficient. Suppose file C (e.g. /usr/local/etc/mime.types) is in both packages A and B.
Install A - this will create C Install B - this might overwrite C, saving a copy, or it might retain A's copy. Uninstall B - this has to know that C is used by A and not touch it
Correct. However, in practice, B should not touch C, unless the file is shared between them.
This is a key issue for support of namespace packages, at least if we want to avoid using .pth files. (Which is what setuptools-built system packages do for namespace packages currently.)
Of course, one possible solution is for both A and B to depend on a "virtual package" that contains C, such that both A and B can install it if it's not there, and list it in their dependencies. But this is one of the handful of open issues that needs to be resolved with Real Life Package Management people, such as Debian, Fedora, etc. [...] I'm not familiar with namespace packages as I've never used one, but I assume it is just an empty directory with one (empty?) __init__.py file in it?
It's not empty; it has to contain a namespace declaration. There's another approach that can work around the absence of the __init__.py, by using .pth files, but it's messy, and I'd like to get rid of it if possible.
So __init__.py contains some data, but it's static regardless of which sub-packages are installed? So far so good as that doesn't influence things on package-management level. I knew about the .pth hack but as you say it's messy and I'm glad you prefer to get rid of it too :-).
The only way I can see Debian handling that is by having a Debian package providing exactly that __init__.py and all other Debian packages needing that namespace package will depend on it.
However, this is shouldn't worry anyone other then Debian I think.
Alas, it'll probably affect most package managers.
If a distribution needs to depend on somthing it will depend on whoever provides the module *inside* the namespace package, not on the namespace itself. So the fact that no distribution provides the namespace.__init__.py file shouldn't have to worry other users of the python installdb.
Well, somebody still has to create the __init__.py, and own it for purposes of uninstallation.
Ok, true. The virtual distribution thing (or an other solution) might be needed for this. But what I was trying to say was that other package managers should keep their hands off Debian's /usr/lib IMHO.
Since easy_install and others shouldn't be messing with debian packages in /usr/lib (they should be using /usr/local/lib or ~/.local [1]) this won't be a problem.
If the new PEP (262 redux) contains anything about actual directory locations on any system of any platform (other than as examples), I'll consider it a failure. Thinking about specific locations while talking about the new PEP should be considered a segmentation fault. :)
I think I agree that the PEP should not detail locations (and relative paths inside the installdb is a good point), but I do think it's part of this discussion so that the installdb can cope with the different directory situations. You can't expect the system and a user to be able to write to the same installdb. A problem along this line: The system (e.g. Debian) has a namespace package installed and a few of it's sub-packages (/usr/lib). The local admin wants a new sub-pacakge of this namespace which should go into e.g. /usr/local/lib (and is not available in the system). You now have: $system/namespace/__init__.py $system/namespace/sub1/__init__.py $local/namespace/__init__.py $local/namespace/sub2/__init__.py But this doesn't work in Python! The only solution I can think of is that the local admin has to install the entire namespace package again and make sure it ends up before the system one on sys.path. This is slightly ugly I think. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Phillip J. Eby wrote:
At 10:17 PM 3/21/2008 +0000, Floris Bruynooghe wrote:
On Fri, Mar 21, 2008 at 03:02:25PM -0400, Phillip J. Eby wrote:
At 11:21 AM 3/21/2008 -0500, skip@pobox.com wrote:
Joachim> I think, the uninstall should _not_ 'rm -rf' but
only 'rm' the
Joachim> files (and 'rmdir' directories, but not recursively) that it Joachim> created, and that have not been modified in the
meantime (after
Joachim> the installation).
That's not sufficient. Suppose file C (e.g. /usr/local/etc/mime.types) is in both packages A and B.
Install A - this will create C Install B - this might overwrite C, saving a copy, or it might retain A's copy. Uninstall B - this has to know that C is used by A and not touch it
Correct. However, in practice, B should not touch C, unless the file is shared between them.
This is a key issue for support of namespace packages, at least if we want to avoid using .pth files. (Which is what setuptools-built system packages do for namespace packages currently.)
Of course, one possible solution is for both A and B to depend on a "virtual package" that contains C, such that both A and B can install it if it's not there, and list it in their dependencies. But this is one of the handful of open issues that needs to be resolved with Real Life Package Management people, such as Debian, Fedora, etc. Debian (dpkg) does not allow a file to exist in more then one package (distribution in distutils speak) as I said earlier. Directories can however, they will only be removed if they are empty (i.e. when the last package having that directory is uninstalled).
This is a general problem with system package managers, actually. Few allow a file to be provided by more than one package.
I'm not familiar with namespace packages as I've never used one, but I assume it is just an empty directory with one (empty?) __init__.py file in it?
It's not empty; it has to contain a namespace declaration. There's another approach that can work around the absence of the __init__.py, by using .pth files, but it's messy, and I'd like to get rid of it if possible.
If all Python distributions which install PythonPacakges under the "namespace" package are bundeled as DistroPackages, then the content of __init__.py won't really matter, will it? There won't be more than one copy of the "used-to-be namespace pacakge in this scenario. If it *does* matter (e.g., the DistroPackages need to support use of non-DistroPackage PythonPackages which use the same namespace), then all those DistroPackages will need to depend on a nearly-empty, generated-by-the-packager DistroPacakge which supplies only that file, as Florian says next.
The only way I can see Debian handling that is by having a Debian package providing exactly that __init__.py and all other Debian packages needing that namespace package will depend on it.
However, this is shouldn't worry anyone other then Debian I think.
Alas, it'll probably affect most package managers.
Right: RPM isn't happy about having a file owned by multiple packages either.
If a distribution needs to depend on somthing it will depend on whoever provides the module *inside* the namespace package, not on the namespace itself. So the fact that no distribution provides the namespace.__init__.py file shouldn't have to worry other users of the python installdb.
Well, somebody still has to create the __init__.py, and own it for purposes of uninstallation.
I think that will need to be an "artifact" DistroPackage (one which has no corresponding Python source distribution).
Since easy_install and others shouldn't be messing with debian packages in /usr/lib (they should be using /usr/local/lib or ~/.local [1]) this won't be a problem.
- -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH5Fyv+gerLs4ltQ4RAsleAKCyF4qz0lULO1J6vAQMNZyaC+MWfQCbBzfD /eoWYRwwLws24601BAMhdk0= =dNgJ -----END PGP SIGNATURE-----
Phillip J. Eby wrote:
At 11:21 AM 3/21/2008 -0500, skip@pobox.com wrote:
Joachim> I think, the uninstall should _not_ 'rm -rf' but only 'rm' the Joachim> files (and 'rmdir' directories, but not recursively) that it Joachim> created, and that have not been modified in the meantime (after Joachim> the installation).
That's not sufficient. Suppose file C (e.g. /usr/local/etc/mime.types) is in both packages A and B.
Install A - this will create C Install B - this might overwrite C, saving a copy, or it might retain A's copy. Uninstall B - this has to know that C is used by A and not touch it
Correct. However, in practice, B should not touch C, unless the file is shared between them.
This is a key issue for support of namespace packages, at least if we want to avoid using .pth files. (Which is what setuptools-built system packages do for namespace packages currently.)
Of course, one possible solution is for both A and B to depend on a "virtual package" that contains C, such that both A and B can install it if it's not there, and list it in their dependencies. But this is one of the handful of open issues that needs to be resolved with Real Life Package Management people, such as Debian, Fedora, etc.
I've always thought that the right way to handle the dependency DAG is to treat it as a garbage collection problem. Assume that for each package there is a way to derive the following two pieces of information: (a) whether this package was installed explicitly by the user, or implicitly as the result of a dependency, and (b) the set of dependencies for this package. Then, starting with the list of 'explicit' packages as the root set, do a standard mark & sweep; Any package not marked is a candidate for removal. -- Talin
Phillip J. Eby escreveu:
Of course, one possible solution is for both A and B to depend on a "virtual package" that contains C, such that both A and B can install it if it's not there, and list it in their dependencies. No need. Is this is what you want? http://www.debian.org/doc/debian-policy/ap-pkg-diversions.html
I've not-so-recently advocated actually building .debs/.rpm/.whatever instead of reinventing wheels in .eggs. Maybe that would be an worthwhile discussion.
At 05:45 PM 3/25/2008 +0000, Luis Bruno wrote:
Phillip J. Eby escreveu:
Of course, one possible solution is for both A and B to depend on a "virtual package" that contains C, such that both A and B can install it if it's not there, and list it in their dependencies. No need. Is this is what you want? http://www.debian.org/doc/debian-policy/ap-pkg-diversions.html
No. Even aside from the fact that it is debian-specific, it is not intended for the use case of a symmetrically-shared and duplicated file.
On Tue, Mar 25, 2008 at 1:45 PM, Luis Bruno <me@lbruno.org> wrote:
Phillip J. Eby escreveu:
Of course, one possible solution is for both A and B to depend on a "virtual package" that contains C, such that both A and B can install it if it's not there, and list it in their dependencies. No need. Is this is what you want? http://www.debian.org/doc/debian-policy/ap-pkg-diversions.html
I've not-so-recently advocated actually building .debs/.rpm/.whatever instead of reinventing wheels in .eggs. Maybe that would be an worthwhile discussion.
I included this as an alternative in my stab at a rationale and lead-in to a proposal. You can see Floris Bruynooghe's response earlier in the thread, but basically it is an even harder problem to create packages for every major platform automatically than it is to create a simple database of installed packages that plays well with every major package manager. How did you advocate solving this problem? I agree that if every python package provided a MSI/DEB/RPM/MPKG/etc., then there wouldn't be a need to install python packages without a system-level installer. It just doesn't seem practical to achieve that, so there remains, at the very least, the need to provide a programmatic means to determine what versions of what things are currently installed in your python environment. Well, when I say "at the very least" I'm assuming that one can accept that a non-negligible portion of the python community would like to maintain and use a tool to help automate the task of adding or removing python packages not available as system packages. That said, my motivation for including at least briefly a list of alternatives was to encourage others to flesh them out a bit more (or porpose new ones) as maybe one of them really is a better! I personally like the switchable sub-environment idea even though it only side steps the issues instead of solving them.
On 25 Mar, 2008, at 20:11, Alexander Michael wrote:
You can see Floris Bruynooghe's response earlier in the thread, but basically it is an even harder problem to create packages for every major platform automatically than it is to create a simple database of installed packages that plays well with every major package manager.
Why is that? Looking in from the peanut gallery the major problem with automaticly creating OS packages is political, in that some packagers don't believe that developers could create valid OS packages. The technical issues seem small compared to that. Ronald
On Thu, Mar 27, 2008 at 01:55:25PM +0100, Ronald Oussoren wrote:
On 25 Mar, 2008, at 20:11, Alexander Michael wrote:
You can see Floris Bruynooghe's response earlier in the thread, but basically it is an even harder problem to create packages for every major platform automatically than it is to create a simple database of installed packages that plays well with every major package manager.
Why is that? Looking in from the peanut gallery the major problem with automaticly creating OS packages is political, in that some packagers don't believe that developers could create valid OS packages.
The technical issues seem small compared to that.
If all I describe in [1] seems political to you then here some techinical difficulties: Developing the tools required to create MSI databases and CAB files for creating a MS windows install package on any platform python runs on would be a huge project (msilib is implemented by calling the MS C API). Likewise it would be a massive job to create the toolchain in python to create DEBs. I don't know much about RPMs and all other systems out there but if these two systems are any indication about the complexity then supporting *every* OS out there is pretty hard. Not to mention some small distro that has their own format few people care about. Supporing everything in python itself as well as allowing systems to use the python infrastructure is far easier. Regards Floris [1] http://mail.python.org/pipermail/distutils-sig/2008-March/009112.html
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
Phillip J. Eby wrote:
... if tools exist and are distributed for such a [PEP 262] "database", and *everybody* agrees to use it as an officially-blessed standard, then it should be possible for setuptools to co-exist with that framework, and we're all happy campers.
I like this idea and the 3 items proposed to accomplish it.
2. Update or replace the implementation as appropriate ...
After some googling and digging around, I found: <http://svn.python.org/projects/sandbox/trunk/pep262> Is that what you meant by "the implementation"?
Questions, comments... volunteers? :)
I'll try to help, if this is agreed to and if I'm able. Steve
On Fri, Mar 21, 2008 at 2:47 PM, Phillip J. Eby <pje@telecommunity.com> wrote:
Second, there were no uninstall tools for it, so I'd have had to write one myself. (Zed's "easy_f'ing_uninstall" to the contrary, it ain't easy, and I have an aversion to deleting stuff on people's systems without knowing what will break. There's a big difference between them typing 'rm -rf' themselves, and me doing it.)
Agree. I tried a while ago to write a "easy_uninstall" but this is not possible from an application-point of view, to do clean things. zc.buildout resolves this by installing eggs locally, to an isolated environment, so my main Python installation doesn't hold any extensions at all. So if a database of installed package is created, I would be in favor of an application-oriented system where it is possible to create, update, install, a set of packages dedicated to an environment (default would be Python). Maybe by having a namespace tied to a list of versions. In other words; having the same feature virtualenv provides, in Python itself, and define somehow how to switch to it. $ easy_install the.package --environment MyApp
Phillip J. Eby schrieb:
Questions, comments... volunteers? :)
I've yet to read the monster package utils thread so I can't comment on it. However I like to draw some attention to my PEP 370 http://python.org/dev/peps/pep-0370/. It's about a site packages directory in the users home directory. I think it quite related to the discussion. Christian
On Fri, Mar 21, 2008 at 05:59:45PM +0100, Christian Heimes wrote:
Phillip J. Eby schrieb:
Questions, comments... volunteers? :)
I've yet to read the monster package utils thread so I can't comment on it. However I like to draw some attention to my PEP 370 http://python.org/dev/peps/pep-0370/. It's about a site packages directory in the users home directory. I think it quite related to the discussion.
I agree that this is related and valuable too. It would be good to see it integrated. Some thoughts I have from PEP 370: * ~/.local goes against the de-facto standard of just using ~/lib, ~/bin etc in UNIX. The freedesktop.org standard[1] is a very good argument however, but in that case that standard should be followed more closely which means respecting their env variables i.e. maybe putting data in $XDG_DATA_HOME. Also the spec does not mention ~/.local/lib or ~/.local/bin at all, meaning there is no standardised env variable to change it (PEP 370 creates PYTHONUSERBASE to work round that though). * The user site directory would end up before the system site directory (except for suid apps). This will be a problem for Debian etc as bug reports are guaranteed when users install a newer version of a lib locally and accidentally break an app owned by the system. It would be better to leave the user site directory last on sys.path and require overwriting this using PTYHONPATH, so users do it explicitly. * setup.py install's new --user argument. A --home already exists, can the default of that not be changed to the user site directory? Currently you need to do "--home=~" so making it so that "--home" would end up using PYTHONUSERBASE would not break things would it? Regards Floris [1] http://www.freedesktop.org/wiki/Specifications/basedir-spec -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
On 2008-03-21 14:47, Phillip J. Eby wrote:
So, to accomplish this, we (for some value of "we") need to:
1. Hash out consensus around what changes or enhancements are needed to PEP 262, to resolve the previously-listed open issues, those that have come up since (namespace packages, dependency specifications, canonical name/version forms), and anything else that comes up.
2. Update or replace the implementation as appropriate, and modify the distutils to support it in Python 2.6 and beyond. And "support it" means, "ensure that 'install' and *all* bdist commands update the database". The bdist_rpm, bdist_wininst, and bdist_msi commands, even bdist_dumb. (This should probably also include the add/remove programs stuff in the Windows case.)
The bdist commands don't need to touch that database in any way, since they don't install anything, nor do they upload things anywhere. They simply package code and put the result into the dist/ subdir. That's all. What you probably mean is that the installers, pre/post-scripts, etc. run when installing one of those packages should update the database of installed packages. Note that there are several package formats which do not execute any code when installing them - the user simply unzips them in some directory. These packages won't be able to register themselves with a database. I guess the only way to support all of these variants is to use a filesystem based approach, e.g. by placing a file with a special extension into some dir on sys.path. The "database" logic could then scan sys.path for these files, read the data and provide an interface to it. All bdist formats would then have to include these files. distutils already writes .egg-info files when running python setup.py install, so perhaps that's a start (though I'd prefer a three letter extension such as ".pkg"). .egg-info files currently only include the package meta-data (the PKG-INFO section from PEP 262). We'd have to add a list of files making up the package (FILES section in PEP 262) and also some extra information about any extra files the package creates that can safely be removed in the uninstall process (e.g. .pyo and .pyc files, temporary files, database files, configuration data, registry entries, etc.) - this is currently not covered in PEP 262. I don't think the REQUIRES and PROVIDES sections from the PEP 262 are needed. That info can easily go into the PKG-INFO section. A separate FILES section also doesn't seem to be necessary - we could just add one or more entries or the format: CreatesDir abc/ CreatesFile abc/xyz1.py CreatesDir abc/def/ CreatesFile abc/def/xyz2.py CreatesFile abc/def/xyz3.py CreatesFile abc/def/xyz4.ini (BTW: wininst writes such a file for the uninstall process) So to keep things simple, the rfc822 approach defined in PEP 241 would easily cover everything needed and we could trim down the PEP 262 format to a simple rfc822 header list. In other words: the .egg-info files already provide the basis and only need to be extended with a list of created files, directories (and possibly other resources) as well as a list of resources which may be removed even if not installed explicitly such as byte-code files, etc.
3. Create a document for system packagers referencing the PEP and introducing them to what/why/how of the standard, in case they weren't one of the original participants in creating this.
This should probably be a new PEP defining all the bits and pieces making up the installation "database".
It will probably take some non-trivial work to do all this for Python 2.6, but it's probably possible, if we start now. I don't think it's critical to have an uninstall tool distributed with 2.6, as long as there's a reasonable way to bootstrap its installation later.
BTW: There's a simple uninstall command in mxSetup.py that we could contribute to distutils. It works much in the same way as the install command... except that it removes all the files it would have installed. Using pre-built packages, this works without having to rebuild the package just to be able to determine the list of things that need to be removed. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 21 2008)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
At 08:06 PM 3/21/2008 +0100, M.-A. Lemburg wrote:
I guess the only way to support all of these variants is to use a filesystem based approach, e.g. by placing a file with a special extension into some dir on sys.path. The "database" logic could then scan sys.path for these files, read the data and provide an interface to it.
All bdist formats would then have to include these files.
That's the idea behind the current version of PEP 262, yes, and I think it should be kept.
A separate FILES section also doesn't seem to be necessary - we could just add one or more entries or the format:
CreatesDir abc/ CreatesFile abc/xyz1.py CreatesDir abc/def/ CreatesFile abc/def/xyz2.py CreatesFile abc/def/xyz3.py CreatesFile abc/def/xyz4.ini
I actually think the size and hash information is good, in order to be able to tell if you're looking at an original file. I'm not sure how useful the permissions and uid/gid info is. I'm hoping we'll hear from anybody who has a use case for that. And of course, there are still some issues to be resolved regarding requirements, package name/version stuff, etc. But we can hash those out once we reach a quorum on the Distutils-SIG.
On 2008-03-21 22:21, Phillip J. Eby wrote:
At 08:06 PM 3/21/2008 +0100, M.-A. Lemburg wrote:
I guess the only way to support all of these variants is to use a filesystem based approach, e.g. by placing a file with a special extension into some dir on sys.path. The "database" logic could then scan sys.path for these files, read the data and provide an interface to it.
All bdist formats would then have to include these files.
That's the idea behind the current version of PEP 262, yes, and I think it should be kept.
A separate FILES section also doesn't seem to be necessary - we could just add one or more entries or the format:
CreatesDir abc/ CreatesFile abc/xyz1.py CreatesDir abc/def/ CreatesFile abc/def/xyz2.py CreatesFile abc/def/xyz3.py CreatesFile abc/def/xyz4.ini
I actually think the size and hash information is good, in order to be able to tell if you're looking at an original file. I'm not sure how useful the permissions and uid/gid info is. I'm hoping we'll hear from anybody who has a use case for that.
You're heading off in the wrong direction: we should not be trying to rewrite RPM or InnoSetup in Python. Anything more complicated should be left to tools which are specifically written to manage complex software setups. I honestly believe that most people would be happy if we just provide these two things (and no more): * install a package from a local archive, a URL or PyPI * uninstall a package in way that doesn't break other installed packages and whatever the mechanism, avoid making any undercover changes to the Python installation such as adding .pth files, overriding site.py, etc. - these are not needed if the tool keeps to the simple task of installing and uninstalling Python packages. Examples: python pypi.py install mypkg-1.0.tgz python pypi.py install http://www.example.com/mypkg-1.0.tgz python pypi.py install mypkg-1.0 python pypi.py uninstall mypkg If there's a dependency problem, the tool should print the list of other packages it needs. It should not try to install things automagically. If a package needs other modules as well, the package docs can point the user to use e.g. python pypi.py install mydep1-1.3 mydep2-2.3 mydep4-0.3 mypkg-1.0 instead. Anything more complicated should be left to specialized tools such as RPM, apt, MSI or the other such tools out there - after all the tool should be about Python *package* installation, not application installation. We *don't* need the tool to: * support multiple versions of a package (that's just bound to cause problems with pickle, isinstance() etc.) * provide namespace hacking (is a completely separate issue and can be handled by the packages rather than the install tool) * support all kinds of funky version numbers (if a package wants to participate in the system, the author better make sure that the version string fits the standard format) * provide some form of intra-package bus interface (ie. "entry points" as you call them) * provide support for keeping whole packages in ZIP files (doesn't play well with C extensions, clutters up the sys.path, is read-only, needs special importers, etc. etc. ) * try automatic version matching for required packages * download things from SourceForge or other sites with special download mechanisms * scan websites for links * make coffee, clean the house, send the kids to school :-)
And of course, there are still some issues to be resolved regarding requirements, package name/version stuff, etc. But we can hash those out once we reach a quorum on the Distutils-SIG.
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 21 2008)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
On 21/03/2008, M.-A. Lemburg <mal@egenix.com> wrote:
You're heading off in the wrong direction: we should not be trying to rewrite RPM or InnoSetup in Python.
Anything more complicated should be left to tools which are specifically written to manage complex software setups.
I honestly believe that most people would be happy if we just provide these two things (and no more):
* install a package from a local archive, a URL or PyPI
* uninstall a package in way that doesn't break other installed packages
and whatever the mechanism, avoid making any undercover changes to the Python installation such as adding .pth files, overriding site.py, etc. - these are not needed if the tool keeps to the simple task of installing and uninstalling Python packages.
My immediate impression is that I completely agree with this. I'd like to add one capability - to be able to list all installed packages. Anything beyond this, I'd need to be persuaded about. That's not to say that it's not valuable, just that it should be separate from the installer. That's where setuptools falls down, in my view - it tries to do too much all in one package.
We *don't* need the tool to: [...] * make coffee, clean the house, send the kids to school :-)
Well, these would be useful :-) Paul.
At 11:13 PM 3/21/2008 +0100, M.-A. Lemburg wrote:
On 2008-03-21 22:21, Phillip J. Eby wrote:
At 08:06 PM 3/21/2008 +0100, M.-A. Lemburg wrote:
I guess the only way to support all of these variants is to use a filesystem based approach, e.g. by placing a file with a special extension into some dir on sys.path. The "database" logic could then scan sys.path for these files, read the data and provide an interface to it.
All bdist formats would then have to include these files.
That's the idea behind the current version of PEP 262, yes, and I think it should be kept.
A separate FILES section also doesn't seem to be necessary - we could just add one or more entries or the format:
CreatesDir abc/ CreatesFile abc/xyz1.py CreatesDir abc/def/ CreatesFile abc/def/xyz2.py CreatesFile abc/def/xyz3.py CreatesFile abc/def/xyz4.ini
I actually think the size and hash information is good, in order to be able to tell if you're looking at an original file. I'm not sure how useful the permissions and uid/gid info is. I'm hoping we'll hear from anybody who has a use case for that.
You're heading off in the wrong direction: we should not be trying to rewrite RPM or InnoSetup in Python.
I'm making the assumption that the author(s) of PEP 262 had good reason for including what they did, rather than assuming that we should start the entire process over from scratch.
Anything more complicated should be left to tools which are specifically written to manage complex software setups.
Tools which will need this data, in order to do their work. Hence, the reason for standardizing the data, instead of the tool(s).
[snip long list of features, both desired and undesired]
Actually, *all* of these features are out of scope for stdlib development, because I'm not proposing including *any* tools for this in the stdlib, apart from distutils install and bdist_* support. I'm proposing, rather, that we finish the vision of PEP 262, of having a standard specification that *all* tools will abide by -- including rpm, dpkg, and what-have-you. Since *all* of these tools need to abide by that specification, their requirements will need to be considered in the formulation of the spec. And as I said, I'll be happy if all we do is get the distutils to abide by the spec for 2.6, even if it means we don't get an uninstall tool. That can always be installed later using Guido's bootstrap tool. :)
I'm making the assumption that the author(s) of PEP 262 had good reason for including what they did, rather than assuming that we should start the entire process over from scratch.
The objections to the PEP remain the same as they were then, though: In the requirements, it says "we need", without saying why we need. It then goes on saying "we want" (rephrased) "to duplicate APT and RPM", without saying why we want that, or why that brings us closer to what we need. IOW, the PEP is lacking a rationale.
Anything more complicated should be left to tools which are specifically written to manage complex software setups.
Tools which will need this data, in order to do their work. Hence, the reason for standardizing the data, instead of the tool(s).
If there was a chance that the infrastructure being developed actually helps these tools, *that* would be a reasonable goal, IMO. However, I'm extremely skeptical that this can ever succeed to the degree that whoever provides RPMs, .debs, or MSI files will actually use such data, as they will find that the data are incomplete, and they have to redo all of it, anyway.
I'm proposing, rather, that we finish the vision of PEP 262, of having a standard specification that *all* tools will abide by -- including rpm, dpkg, and what-have-you.
Do you also envision the objective of PEP 262, then? I.e. to provide a database of installed packages, in .../install-db?
And as I said, I'll be happy if all we do is get the distutils to abide by the spec for 2.6, even if it means we don't get an uninstall tool. That can always be installed later using Guido's bootstrap tool. :)
I'm even more skeptical here. If the assumption is that the package database allows for uninstallation, I'm -1. IOW, RPM, deb, MSI should then *not* write to that package database, as they also write to a different database, out of the scope of the PEP, and this is what uninstallation should use. Regards, Martin
At 02:31 AM 3/22/2008 +0100, Martin v. Löwis wrote:
I'm making the assumption that the author(s) of PEP 262 had good reason for including what they did, rather than assuming that we should start the entire process over from scratch.
The objections to the PEP remain the same as they were then, though: In the requirements, it says "we need", without saying why we need. It then goes on saying "we want" (rephrased) "to duplicate APT and RPM", without saying why we want that, or why that brings us closer to what we need.
IOW, the PEP is lacking a rationale.
Ok, well, I have a rationale for it: make it possible to get rid of eggs as a mechanism for supporting easy_install. Many people (yourself included) have criticized eggs as an installation mechanism, and this is an alternative that gets rid of .egg files and directories in that case, and most of the need for .pth file usage as well.
If there was a chance that the infrastructure being developed actually helps these tools, *that* would be a reasonable goal, IMO.
Yes, I'm of course primarily interested in Python-specific tools such as virtualenv, easy_install, buildout, and the as-yet-unwritten uninstallers, package listers, etc., that can usefully read or write such data.
However, I'm extremely skeptical that this can ever succeed to the degree that whoever provides RPMs, .debs, or MSI files will actually use such data, as they will find that the data are incomplete, and they have to redo all of it, anyway.
The data isn't for them to use to meet their use cases, it's for them to *provide* so that Python tools don't stomp on, uninstall, or otherwise interfere with files installed by the system. In other words, for system packagers, it's a communication from the system to Python, rather than the other way around. Even though the distutils will build the file in the bdist, the system packaging tools would be free to generate their own file listing and signatures and such.
Do you also envision the objective of PEP 262, then? I.e. to provide a database of installed packages, in .../install-db?
In each directory relative to a given sys.path directory, yes. That is, installing a distutils distribution to any directory would result in a file being added to an install-db within that directory. (Assuming we use the proposed implementation model of PEP 262, which at the moment I don't see any substantial obstacle to.)
And as I said, I'll be happy if all we do is get the distutils to abide by the spec for 2.6, even if it means we don't get an uninstall tool. That can always be installed later using Guido's bootstrap tool. :)
I'm even more skeptical here. If the assumption is that the package database allows for uninstallation, I'm -1. IOW, RPM, deb, MSI should then *not* write to that package database, as they also write to a different database, out of the scope of the PEP, and this is what uninstallation should use.
I probably should have brought this up, in fact, I think I mentioned it in a previous thread, but I would like to see PEP 262 add a way to say "this is a system-installed package, *don't touch*". The idea again is not to do the job of the native packaging system, but rather to ensure that Python-specific tools (e.g. easy_install and friends) do not interfere or conflict with it. A big problem in the early development of easy_install, even using eggs, was that there was no way to tell whether it was safe to delete or overwrite an existing file or directory that was already installed on the system. A mechanism like this would allow tools like easy_install to say, "oh, your system packager has a conflicting package here, you need to use that tool to sort this out if you really want to do something here. I'm not going to touch that." Without something like this, there is no way to tell the difference on many systems between a system package and something the user has put there with "sudo python setup.py install".
On Fri, Mar 21, 2008 at 10:04:45PM -0400, Phillip J. Eby wrote:
At 02:31 AM 3/22/2008 +0100, Martin v. Löwis wrote:
However, I'm extremely skeptical that this can ever succeed to the degree that whoever provides RPMs, .debs, or MSI files will actually use such data, as they will find that the data are incomplete, and they have to redo all of it, anyway.
The data isn't for them to use to meet their use cases, it's for them to *provide* so that Python tools don't stomp on, uninstall, or otherwise interfere with files installed by the system. In other words, for system packagers, it's a communication from the system to Python, rather than the other way around. Even though the distutils will build the file in the bdist, the system packaging tools would be free to generate their own file listing and signatures and such.
As long as systems (dpkg, rpm, ...) install the .egg-info files they do communicate which modules/distributions are installed. The installdb would just duplicate this information (according to the current PEP).
And as I said, I'll be happy if all we do is get the distutils to abide by the spec for 2.6, even if it means we don't get an uninstall tool. That can always be installed later using Guido's bootstrap tool. :)
I'm even more skeptical here. If the assumption is that the package database allows for uninstallation, I'm -1. IOW, RPM, deb, MSI should then *not* write to that package database, as they also write to a different database, out of the scope of the PEP, and this is what uninstallation should use.
I probably should have brought this up, in fact, I think I mentioned it in a previous thread, but I would like to see PEP 262 add a way to say "this is a system-installed package, *don't touch*". The idea again is not to do the job of the native packaging system, but rather to ensure that Python-specific tools (e.g. easy_install and friends) do not interfere or conflict with it.
A big problem in the early development of easy_install, even using eggs, was that there was no way to tell whether it was safe to delete or overwrite an existing file or directory that was already installed on the system. A mechanism like this would allow tools like easy_install to say, "oh, your system packager has a conflicting package here, you need to use that tool to sort this out if you really want to do something here. I'm not going to touch that." Without something like this, there is no way to tell the difference on many systems between a system package and something the user has put there with "sudo python setup.py install".
There is a way of telling if you have to keep you hands off a package (sorry to bring this up again): installation paths. /usr/lib is the system path, the local admin (and hence setuptools) should keep their hands off it at all times (unless requested with a --prefix or so for building the debs or rpms but an uninstall in those cases won't be required from setuptools). What an installdb could help with is tell setuptools to keep it's hands off a distribution manually installed (or by another tool) in the local admin directory (/usr/local) or user directory (~/). Your proposal of an installdb for each directory on sys.path would solve this, but the installdb in /usr/lib will be completely usused. But frankly, for that just an extra field in the .egg-info "Installed-By: setuptools" would do no? Possibly followed by a "X-Setuptools-Installed-Files:" section if you need that. -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
At 11:00 AM 3/22/2008 +0000, Floris Bruynooghe wrote:
As long as systems (dpkg, rpm, ...) install the .egg-info files they do communicate which modules/distributions are installed. The installdb would just duplicate this information (according to the current PEP).
.egg-info/PKG-INFO don't list the specific files, though.
There is a way of telling if you have to keep you hands off a package (sorry to bring this up again): installation paths. /usr/lib is the system path, the local admin (and hence setuptools) should keep their hands off it at all times (unless requested with a --prefix or so for building the debs or rpms but an uninstall in those cases won't be required from setuptools).
As I mentioned previously, if the spec says anything about specific paths, it will be full of fail. The spec MUST be able to work with *any* local policy about where Python packages are to be installed. Otherwise, any tool that wants to work with install-dbs will end up accumulating a long list of paths to be handled specially for each OS vendor and version... and still not handle everything. No can do. This has to be a mechanism, not a policy. Vendors and admins should be able to enforce reasonable policies, without requiring that every tool have those policies built in. For one thing, it's an entry barrier to tools. Basically, what I'm proposing here is like WSGI for package management tools -- and building anything about paths into the spec would be like WSGI spelling out what pages should be at what URLs!
Ok, well, I have a rationale for it: make it possible to get rid of eggs as a mechanism for supporting easy_install. Many people (yourself included) have criticized eggs as an installation mechanism, and this is an alternative that gets rid of .egg files and directories in that case, and most of the need for .pth file usage as well.
How so? I cannot see the need for .egg files or .pth files in the first place. If they exist so that you can import stuff: just install to site-packages, and be done.
The data isn't for them to use to meet their use cases, it's for them to *provide* so that Python tools don't stomp on, uninstall, or otherwise interfere with files installed by the system. In other words, for system packagers, it's a communication from the system to Python, rather than the other way around. Even though the distutils will build the file in the bdist, the system packaging tools would be free to generate their own file listing and signatures and such.
Ok, that's a reasonable requirement. It will be difficult to implement, though, as it will require Linux distributors (in particular) to include the database snippets in their packages. Essentially, one would have to contribute patches to all the distributions (we care about, at least), and then nag the respective maintainers to include these patches.
I probably should have brought this up, in fact, I think I mentioned it in a previous thread, but I would like to see PEP 262 add a way to say "this is a system-installed package, *don't touch*". The idea again is not to do the job of the native packaging system, but rather to ensure that Python-specific tools (e.g. easy_install and friends) do not interfere or conflict with it.
Something like a read-only flag? For those without the read-only flag, the specification should explicitly say what manipulation is allowed. Regards, Martin
On Sat, Mar 22, 2008 at 12:33:49PM +0100, "Martin v. Löwis" wrote:
The data isn't for them to use to meet their use cases, it's for them to *provide* so that Python tools don't stomp on, uninstall, or otherwise interfere with files installed by the system. In other words, for system packagers, it's a communication from the system to Python, rather than the other way around. Even though the distutils will build the file in the bdist, the system packaging tools would be free to generate their own file listing and signatures and such.
Ok, that's a reasonable requirement. It will be difficult to implement, though, as it will require Linux distributors (in particular) to include the database snippets in their packages.
Essentially, one would have to contribute patches to all the distributions (we care about, at least), and then nag the respective maintainers to include these patches.
Not true. You just need to make sure that "setup.py install" creates that database. With the proposed format of the database this is just a file in the correct location - exactly for this reason. Next time the distribution will build the package that database file will be in place. I still maintain that an installdb for the system packages doesn't bring anything to the party as it would be in a system-only directory. But I've argued that in my other emails... Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
Essentially, one would have to contribute patches to all the distributions (we care about, at least), and then nag the respective maintainers to include these patches.
Not true. You just need to make sure that "setup.py install" creates that database. With the proposed format of the database this is just a file in the correct location - exactly for this reason. Next time the distribution will build the package that database file will be in place.
How so? Are you /sure/ the packaging process even *runs* setup.py? And if they do, why do you think they will pick up the database file? Regards, Martin
"Martin v. Löwis" wrote:
Essentially, one would have to contribute patches to all the distributions (we care about, at least), and then nag the respective maintainers to include these patches.
Not true. You just need to make sure that "setup.py install" creates that database. With the proposed format of the database this is just a file in the correct location - exactly for this reason. Next time the distribution will build the package that database file will be in place.
How so? Are you /sure/ the packaging process even *runs* setup.py? And if they do, why do you think they will pick up the database file?
In the case of Fedora rpms, the usual install uses setup.py.
In the case of Fedora rpms, the usual install uses setup.py.
Ok. Does it then also package all files that get installed into the RPM file? If it produces multiple RPMs from a single source package, how does it know which files go into what RPM? Regards, Martin
On Saturday 22 March 2008, Martin v. Löwis wrote:
In the case of Fedora rpms, the usual install uses setup.py.
Ok. Does it then also package all files that get installed into the RPM file? If it produces multiple RPMs from a single source package, how does it know which files go into what RPM?
Regards, Martin
Offhand, I can't think of any examples that produce multiple RPMS, but in general, it doesn't _know_ anything. What goes in what rpm is either 1) Automated using python setup.py install -O1 --root $RPM_BUILD_ROOT --prefix %{_prefix} --record=%{name}.files or 2) Manually listing the files that go into a package.
On Sat, Mar 22, 2008 at 03:14:05PM +0100, "Martin v. Löwis" wrote:
Essentially, one would have to contribute patches to all the distributions (we care about, at least), and then nag the respective maintainers to include these patches.
Not true. You just need to make sure that "setup.py install" creates that database. With the proposed format of the database this is just a file in the correct location - exactly for this reason. Next time the distribution will build the package that database file will be in place.
How so? Are you /sure/ the packaging process even *runs* setup.py? And if they do, why do you think they will pick up the database file?
I speak for Debian, so for Debian: yes. The setup.py would have to be pretty bad for a packager to not use it. There is no reason to re-write upstream's installation procedure as you would have to figure out which files need to be installed where and this would create many bugs. The canonical way is something like this: $ pythonX.Y setup.py --root=$(pwd)/debian/tmp $ # Fixup anything done wrong/badly (for debian) by setup.py $ # Make a tarball of $(pwd)/debian/tmp In reality it's slightly more complicated of course. At [1] there are many packages, paste and parallelpython are good examples if you're interested (look in the debian/rules file). Regards Floris [1] http://svn.debian.org/wsvn/python-modules/packages/?rev=0&sc=0 -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
I speak for Debian, so for Debian: yes. The setup.py would have to be pretty bad for a packager to not use it. There is no reason to re-write upstream's installation procedure as you would have to figure out which files need to be installed where and this would create many bugs.
The canonical way is something like this:
$ pythonX.Y setup.py --root=$(pwd)/debian/tmp $ # Fixup anything done wrong/badly (for debian) by setup.py $ # Make a tarball of $(pwd)/debian/tmp
In reality it's slightly more complicated of course.
More than slightly so, with pycentral, pysupport, and all that. I still doubt that the database would show up in a directory on sys.path. IIUC, things get install into /usr/share/pycentral/package, and then get symlinked into /usr/lib/pythonX.Y/site-packages at installation time. It's not all that clear to me how that deals with "non-regular" files.
At [1] there are many packages, paste and parallelpython are good examples if you're interested (look in the debian/rules file).
I started with nevow. It uses cdbs, so its debian/rules is nearly empty, and does not include a call to setup.py at all. Instead, the distutils support is in /usr/share/cdbs/1/class/python-distutils.mk where setup.py is invoked as ifeq (,$(DEB_PYTHON_REAL_LIB_PACKAGES)) common-install-arch common-install-indep:: common-install-impl common-install-impl:: cd $(DEB_SRCDIR) && python$(DEB_PYTHON_COMPILE_VERSION) $(DEB_PYTHON_SETUP_CMD) install --root=$(DEB_DESTDIR) $(DEB_PYTHON_INSTALL_ARGS_ALL) $(DEB_PYTHON_INSTALL_ARGS_$(cdbs_curpkg)) else $(patsubst %,install/%,$(DEB_PYTHON_REAL_LIB_PACKAGES)) :: install/% : cd $(DEB_SRCDIR) && python$(cdbs_python_ver) $(DEB_PYTHON_SETUP_CMD) install --root=$(CURDIR)/debian/$(cdbs_curpkg) $(DEB_PYTHON_INSTALL_ARGS_ALL) $(DEB_PYTHON_INSTALL_ARGS_$(cdbs_curpkg)) endif $(patsubst %,install/%,$(DEB_PYTHON_SIMPLE_PACKAGES)) :: install/% : cd $(DEB_SRCDIR) && python $(DEB_PYTHON_SETUP_CMD) install --root=$(CURDIR)/debian/$(cdbs_curpkg) $(DEB_PYTHON_INSTALL_ARGS_ALL) $(DEB_PYTHON_INSTALL_ARGS_$(cdbs_curpkg)) I cannot infer from that whether supplying a package database snippet in setup.py would actually work. Regards, Martin
On Sat, Mar 22, 2008 at 04:42:36PM +0100, "Martin v. Löwis" wrote:
I speak for Debian, so for Debian: yes. The setup.py would have to be pretty bad for a packager to not use it. There is no reason to re-write upstream's installation procedure as you would have to figure out which files need to be installed where and this would create many bugs.
The canonical way is something like this:
$ pythonX.Y setup.py --root=$(pwd)/debian/tmp $ # Fixup anything done wrong/badly (for debian) by setup.py $ # Make a tarball of $(pwd)/debian/tmp
In reality it's slightly more complicated of course.
More than slightly so, with pycentral, pysupport, and all that.
I still doubt that the database would show up in a directory on sys.path.
If not it would be a bug in pycentral/pysupport. Only two bugs to file, not that bad I think!
At [1] there are many packages, paste and parallelpython are good examples if you're interested (look in the debian/rules file).
I started with nevow. It uses cdbs, so its debian/rules is nearly empty, and does not include a call to setup.py at all.
Instead, the distutils support is in
/usr/share/cdbs/1/class/python-distutils.mk [...]
Again, that would be a bug in CDBS. For the specific snippet you showed, yes that does essentially "pythonX.Y setup.py --root=$some_alternate_root" as I said above. As an aside I also happen to be in the camp that dislikes CDBS... The specifics and complications don't matter for this discussion I think. If setup.py installs the correct file into the installdb then it will work in almost all cases, Neal Becker confirmed this is true for Fedora as well. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
At 12:33 PM 3/22/2008 +0100, Martin v. Löwis wrote:
I probably should have brought this up, in fact, I think I mentioned it in a previous thread, but I would like to see PEP 262 add a way to say "this is a system-installed package, *don't touch*". The idea again is not to do the job of the native packaging system, but rather to ensure that Python-specific tools (e.g. easy_install and friends) do not interfere or conflict with it.
Something like a read-only flag?
Not exactly. More like, "package management tool X claims exclusive rights to this package". Python tools would always defer this right to the system packager, i.e. a system packager is not obliged to respect a Python tool's claim to a file, but not the other way around. That way, system packaging tools don't need to do anything but mark the installed files as belonging to them. Since most vendors at least *begin* with a "setup.py install", we could provide a way to indicate that the installation is being done on behalf of a system packaging tool, so that it can provide that indication.
For those without the read-only flag, the specification should explicitly say what manipulation is allowed.
Since a distribution isn't really "mutable", I would think that uninstallation and reinstallation would be the only manipulation available. (As distinct from inspection, verification, and other read-only activities.) It's possible, though, that there might also be actions such as restoring or relocating scripts or data in shared locations outside of the sys.path directory. That will get clearer as the spec gets defined.
For those without the read-only flag, the specification should explicitly say what manipulation is allowed.
Since a distribution isn't really "mutable", I would think that uninstallation and reinstallation would be the only manipulation available. (As distinct from inspection, verification, and other read-only activities.)
Sure, but what is precisely the semantics of uninstallation, in terms of changes to the system state? I think any model where uninstallation is merely the removal of files is too limited to be practical. Regards, Martin
At 04:29 PM 3/22/2008 +0100, Martin v. Löwis wrote:
For those without the read-only flag, the specification should explicitly say what manipulation is allowed. Since a distribution isn't really "mutable", I would think that uninstallation and reinstallation would be the only manipulation available. (As distinct from inspection, verification, and other read-only activities.)
Sure, but what is precisely the semantics of uninstallation, in terms of changes to the system state?
I think any model where uninstallation is merely the removal of files is too limited to be practical.
The distutils only support the *addition* of files, so I'm not sure how only removing files is a limit here. Could you explain?
Sure, but what is precisely the semantics of uninstallation, in terms of changes to the system state?
I think any model where uninstallation is merely the removal of files is too limited to be practical.
The distutils only support the *addition* of files, so I'm not sure how only removing files is a limit here. Could you explain?
For files, yes, it only supports addition. But it supports arbitrary other actions, such as: - addition of registry keys - addition of user accounts - creation of database tables in a relational database - updating the shared library loader path - creation and start of a system service - integration of documentation into info - registration of DTDs with the system catalog - ... It's turing-complete, and it has full interface to the operating system, so installation of a distutils package can do *much* more than merely installing files. Uninstallation needs to revert anything installation did, so it is often more than mere removal of files. HTH, Martin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. Löwis wrote:
Sure, but what is precisely the semantics of uninstallation, in terms of changes to the system state?
I think any model where uninstallation is merely the removal of files is too limited to be practical. The distutils only support the *addition* of files, so I'm not sure how only removing files is a limit here. Could you explain?
For files, yes, it only supports addition. But it supports arbitrary other actions, such as: - addition of registry keys - addition of user accounts - creation of database tables in a relational database - updating the shared library loader path - creation and start of a system service - integration of documentation into info - registration of DTDs with the system catalog - ...
It's turing-complete, and it has full interface to the operating system, so installation of a distutils package can do *much* more than merely installing files.
Which is exactly what is wrong with distutils: turing completeness in an installer is an *anti* goal, from the perspective of manageability. I'd be willing to bet that if you asked system packagers to list the dozen or so packages which they *hate* to maintain, the large majority of each list would be packages which acutally use the full power of distutils. (Note: I'm aware that people believe it to be necessary to munge the Windows registry when installing Python packages; I just don't agree with the practice, and don't think we should distort Python's process to coddle it).
Uninstallation needs to revert anything installation did, so it is often more than mere removal of files.
Practically speacking, nobody but the author of the TC-abusing setup.py is *ever* going to be able to do this, and most of them won't get the edge cases right. Maybe we can just punt on such packages, and make a tool which actually works for the huge majority of distributions which don't need to do more than install files. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD4DBQFH6Cdr+gerLs4ltQ4RAlFzAJi3gs8fzb9o8/Dtct1G9P0EJxNSAKCc7V7m uT5MgTzltBDhdrgoNxt8nA== =zgqI -----END PGP SIGNATURE-----
(Note: I'm aware that people believe it to be necessary to munge the Windows registry when installing Python packages; I just don't agree with the practice, and don't think we should distort Python's process to coddle it).
Whoever thinks it necessary is misguided. Installing a package doesn't require munging the registry and none of the popular installation techniques do. Installing Python itself does, and some packages have special requirements (eg, pywin32 registering COM objects), but in general, Python packages on Windows can ignore the registry. Mark
Mark Hammond wrote:
(Note: I'm aware that people believe it to be necessary to munge the Windows registry when installing Python packages; I just don't agree with the practice, and don't think we should distort Python's process to coddle it).
Whoever thinks it necessary is misguided. Installing a package doesn't require munging the registry and none of the popular installation techniques do. Installing Python itself does, and some packages have special requirements (eg, pywin32 registering COM objects), but in general, Python packages on Windows can ignore the registry.
I guess I'll step up and volunteer myself as 'misguided' then. :-) I would very much like to be able to package up my application or library such that it created desktop/start menu/quicklaunch icons for various things like running the app, running examples, opening docs for browsing, etc. I'd also like to be able to associate certain file types with executables/scripts/shortcuts provided by the installation of my project. After all, Windows users in general are not as technically adapt when it comes to command line tools so setting up these nice GUI ways of doing things adds usability significantly. You could argue (like Phillip has) that these operations should be done via an explicitly user-invoked script, and I can buy that for the standard version of the tool. You could also argue that installing applications (which is generally where these kinds of desires come into play) is not the job of the tools we're discussing. But it seems to me that the existing capabilities of setuptools are 80% (or more) of the effort in creating a tool that would allow installation and efficient maintenance of large Python-based applications, such as what my employer delivers to customers. All that being said, I'm fine with the idea that the standard library version of the tool does not enable this. Just so long as nothing is done to actively prevent extensions of that tool to do this sort of thing for those who need or want it. -- Dave
Even installing shortcuts doesn't need to munge the registry! But regardless, you seem to be arguing that setuptools should morph into a full-blown, general purpose installer for python build apps, capable of doing all kinds of platform specific things which any app may desire - and while I don't agree with that, it wasn't what was being discussed at all. Writing and installing a Python extension does not need to munge the registry. Installing a general purpose application, written in Python or anything else, will require, in the general case, anything you could possible imagine. Mark From: Dave Peterson [mailto:dpeterson@enthought.com] Sent: Wednesday, 2 April 2008 3:30 PM Cc: Mark Hammond; distutils-sig Subject: Re: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and beyond Mark Hammond wrote: (Note: I'm aware that people believe it to be necessary to munge the Windows registry when installing Python packages; I just don't agree with the practice, and don't think we should distort Python's process to coddle it). Whoever thinks it necessary is misguided. Installing a package doesn't require munging the registry and none of the popular installation techniques do. Installing Python itself does, and some packages have special requirements (eg, pywin32 registering COM objects), but in general, Python packages on Windows can ignore the registry. I guess I'll step up and volunteer myself as 'misguided' then. :-) I would very much like to be able to package up my application or library such that it created desktop/start menu/quicklaunch icons for various things like running the app, running examples, opening docs for browsing, etc. I'd also like to be able to associate certain file types with executables/scripts/shortcuts provided by the installation of my project. After all, Windows users in general are not as technically adapt when it comes to command line tools so setting up these nice GUI ways of doing things adds usability significantly. You could argue (like Phillip has) that these operations should be done via an explicitly user-invoked script, and I can buy that for the standard version of the tool. You could also argue that installing applications (which is generally where these kinds of desires come into play) is not the job of the tools we're discussing. But it seems to me that the existing capabilities of setuptools are 80% (or more) of the effort in creating a tool that would allow installation and efficient maintenance of large Python-based applications, such as what my employer delivers to customers. All that being said, I'm fine with the idea that the standard library version of the tool does not enable this. Just so long as nothing is done to actively prevent extensions of that tool to do this sort of thing for those who need or want it. -- Dave
Hi Mark, Shortcuts don't, but file associations to those shortcuts do (at least to the best of my understanding,) adding paths does, etc. I agree that a library (extensions or no) doesn't need to do these things. But anything targeted at an end-user on Windows has a reasonable chance of wanting them. Here I thought I was being pretty clear that I was only requesting that things not be done to explicitly make it hard (or impossible) for someone else to extend setuptools (or whatever replaces / refactors it) to do these things. -- Dave Mark Hammond wrote:
Even installing shortcuts doesn't need to munge the registry! But regardless, you seem to be arguing that setuptools should morph into a full-blown, general purpose installer for python build apps, capable of doing all kinds of platform specific things which any app may desire -- and while I don't agree with that, it wasn't what was being discussed at all. Writing and installing a Python extension does not need to munge the registry. Installing a general purpose application, written in Python or anything else, will require, in the general case, anything you could possible imagine.
Mark
*From:* Dave Peterson [mailto:dpeterson@enthought.com] *Sent:* Wednesday, 2 April 2008 3:30 PM *Cc:* Mark Hammond; distutils-sig *Subject:* Re: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and beyond
Mark Hammond wrote:
(Note: I'm aware that people believe it to be necessary to munge the
Windows registry when installing Python packages; I just don't agree
with the practice, and don't think we should distort Python's process
to coddle it).
Whoever thinks it necessary is misguided. Installing a package doesn't require munging the registry and none of the popular installation techniques do. Installing Python itself does, and some packages have special requirements (eg, pywin32 registering COM objects), but in general, Python packages on Windows can ignore the registry.
I guess I'll step up and volunteer myself as 'misguided' then. :-)
I would very much like to be able to package up my application or library such that it created desktop/start menu/quicklaunch icons for various things like running the app, running examples, opening docs for browsing, etc. I'd also like to be able to associate certain file types with executables/scripts/shortcuts provided by the installation of my project. After all, Windows users in general are not as technically adapt when it comes to command line tools so setting up these nice GUI ways of doing things adds usability significantly.
You could argue (like Phillip has) that these operations should be done via an explicitly user-invoked script, and I can buy that for the standard version of the tool. You could also argue that installing applications (which is generally where these kinds of desires come into play) is not the job of the tools we're discussing. But it seems to me that the existing capabilities of setuptools are 80% (or more) of the effort in creating a tool that would allow installation and efficient maintenance of large Python-based applications, such as what my employer delivers to customers.
All that being said, I'm fine with the idea that the standard library version of the tool does not enable this. Just so long as nothing is done to actively prevent extensions of that tool to do this sort of thing for those who need or want it.
-- Dave
On Wed, Apr 02, 2008 at 06:32:13PM -0500, Dave Peterson wrote:
Shortcuts don't, but file associations to those shortcuts do (at least to the best of my understanding,) adding paths does, etc. I agree that a library (extensions or no) doesn't need to do these things. But anything targeted at an end-user on Windows has a reasonable chance of wanting them.
Here I thought I was being pretty clear that I was only requesting that things not be done to explicitly make it hard (or impossible) for someone else to extend setuptools (or whatever replaces / refactors it) to do these things.
I am +1 on this request. I am not a Windows users, but I would still like to see my applications appear in the menus when I ship them. And now free-desktop has a nice specification to do this, so it is technically possible. My 2 cents, Gaël
At 11:19 AM 3/22/2008 -0400, Phillip J. Eby wrote:
Not exactly. More like, "package management tool X claims exclusive rights to this package". Python tools would always defer this right to the system packager, i.e. a system packager is not obliged to respect a Python tool's claim to a file, but not the other way around.
That way, system packaging tools don't need to do anything but mark the installed files as belonging to them.
This probably needs to be refined a little. Exclusive right is too strong, and it goes against Paul Moore's desire for using a single tool. Perhaps instead what it should be is an "uninstall warning" field that must be displayed to a user if an interactive program is doing uninstallation, and that a non-interactive program must refuse to uninstall unless explicitly requested to go ahead. Unfortunately, a warning message might then need to be localized. So this idea still needs some work.
On 22/03/2008, Phillip J. Eby <pje@telecommunity.com> wrote:
This probably needs to be refined a little. Exclusive right is too strong, and it goes against Paul Moore's desire for using a single tool.
Huh? How's that? Don't forget that I'm on Windows, and on Windows there is no "system tool" - just bdist_wininst, bdist_msi and easy_install. The fact that bdist_wininst and bdist_msi link into the system UI for listing and uninstallation doesn't make packages using them "system packages". If easy_install put add/remove program info in place, my "single tool" would be the add/remove list. If something else (for example, the proposed index of installed package, with a suitable UI) is deemed the "single tool", then bdist_wininst and bdist_msi have to be changed to respect that. The only fly in this ointment is bdist_msi, which uses MSI format, which is a lot nearer to a Windows "system packager" than anything else. Whether that means bdist_msi can't be changed to work with a package index rather than (or as well as, I don't care) add/remove, I don't know.
Unfortunately, a warning message might then need to be localized. So this idea still needs some work.
The "one way to do it" uninstaller should be able to uninstall everything. If it needs to call the system uninstaller for a specific package, there's nothing wrong with doing that. But why tell me to run a different command? Just do it for me. I only want one UI, the rest is implementation detail. (Others may have different preferences, so a choice may need to be made. If so, and if it goes against me, fair enough, I have to decide what to do about that for myself. But I'd rather force people to tell me "no", than leave people thinking they satisfied me and wondering why I'm still complaining...) Paul.
Huh? How's that? Don't forget that I'm on Windows, and on Windows there is no "system tool" - just bdist_wininst, bdist_msi and easy_install. The fact that bdist_wininst and bdist_msi link into the system UI for listing and uninstallation doesn't make packages using them "system packages".
In pje's terminology, they do. He uses "system package" as a shorthand for "package in a format defined by the system vendor", not as "package supplied by the system vendor" (IIUC). So .msi files and self-extracting .exe files are all "system packages", as opposed to .eggs (which are in a format that wasn't defined by the system vendor).
The only fly in this ointment is bdist_msi, which uses MSI format, which is a lot nearer to a Windows "system packager" than anything else. Whether that means bdist_msi can't be changed to work with a package index rather than (or as well as, I don't care) add/remove, I don't know.
If the package database is merely a directory with additional files in it, one file per package, then most likely both bdist_wininst and bdist_msi could support that. If the file needs to contain file names specific to the target system, then supporting it in bdist_msi is a bit tricky, as one would have to generate the file at installation time. That's a "custom action"; I'd probably generate a VB script at packaging time which is then run at installation time to edit the package database.
The "one way to do it" uninstaller should be able to uninstall everything. If it needs to call the system uninstaller for a specific package, there's nothing wrong with doing that. But why tell me to run a different command? Just do it for me. I only want one UI, the rest is implementation detail.
The uninstallation procedure of the system installer probably has a separate UI which can't really be suppressed. For example, uninstallation may be rejected as additional applications rely on the package, or uninstallation could cause automatic removal of prerequisite packages that are then no longer used, all requiring user confirmation. Also, on some systems, it's difficult to know what specific tool to run to interact with the system packaging. On some systems, you have a choice of multiple command-line, text mode, and GUI tools, some of which may not be installed, or may fail to work (e.g. when you don't have the windowing system running, and the tool is a windowed one), or may not be the user's preference. Regards, Martin
On Fri, Mar 21, 2008 at 9:31 PM, "Martin v. Löwis" <martin@v.loewis.de> wrote:
The objections to the PEP remain the same as they were then, though: In the requirements, it says "we need", without saying why we need. It then goes on saying "we want" (rephrased) "to duplicate APT and RPM", without saying why we want that, or why that brings us closer to what we need.
IOW, the PEP is lacking a rationale.
It seems to me that this discussion is being undermined by not acknowledging the many use cases up front. There is no rationale because there are too many tacit rationales. Nevertheless, the many use cases are related at some level and would benefit from some form of lowest-common-denominator support in the standard library. As an example, IF I wanted to use a library to support multi-version packages or one to ensure I had all the dependencies, I would need to know what versions of things were currently installed. I can't create a library to provided these functionalities and use it downstream of the package creator without some form of standardization to report package versions. (Or at least without running into the assimilation problem that setuptools has). My personal use case is for multi-version packages. I deploy many small scripts (not applications) that use an evolving set of base libraries. I don't want the older scripts to hold me back from pushing forward with the base libraries, so I peg the scripts to their respective major versions as I release them. Regards, Alex
It seems to me that this discussion is being undermined by not acknowledging the many use cases up front. There is no rationale because there are too many tacit rationales.
I honestly, really, cannot imagine what those are. Explicit is better than implicit.
Nevertheless, the many use cases are related at some level and would benefit from some form of lowest-common-denominator support in the standard library. As an example, IF I wanted to use a library to support multi-version packages or one to ensure I had all the dependencies, I would need to know what versions of things were currently installed.
How would you install multiple versions in the first place? Python supports no such thing, at least not without setting PYTHONPATH, or otherwise changing sys.path.
My personal use case is for multi-version packages. I deploy many small scripts (not applications) that use an evolving set of base libraries. I don't want the older scripts to hold me back from pushing forward with the base libraries, so I peg the scripts to their respective major versions as I release them.
I could not have imagined that as a use case. I never install multiple versions of the same thing on the same system, except for Python itself. I also try to avoid using evolving libraries as much as possible, and quickly abandon libraries if they change in an incompatible manner across releases, at least if porting becomes too much of a burden. Notice that this use case isn't listed in the PEP, and I cannot see how the PEP can help providing that functionality. Regards, Martin
On 22/03/2008, "Martin v. Löwis" <martin@v.loewis.de> wrote:
How would you install multiple versions in the first place? Python supports no such thing, at least not without setting PYTHONPATH, or otherwise changing sys.path.
That's an unrelated feature of setuptools, providing a way to "install" multiple versions of a package and select which version you want at runtime. It does this by sys.path manipulation, I believe. It's a good example of the sort of orthogonal feature which makes setuptools such an issue. It's a genuinely useful feature to some people, but it's unrelated to packaging and installation, so people buy into a packaging solution (eggs) for the non-packaging benefits they provide. Paul.
On Sat, Mar 22, 2008 at 10:02 AM, "Martin v. Löwis" <martin@v.loewis.de> wrote:
It seems to me that this discussion is being undermined by not acknowledging the many use cases up front. There is no rationale because there are too many tacit rationales.
I honestly, really, cannot imagine what those are. Explicit is better than implicit.
Exactly. I was attempting to suggest that we be explicit about "the problem" being solved so it was easier for everyone interested to verify that their "needs" (i.e. use case) will be met and that the proposed solution solves the problem. Since the topic of the thread is how to get rid of eggs in 2.6, the question (to me, at least) is how to enable the functionality provided by setuptools to be provided outside the standard library in a way that does not create pressure for the entire python community to be assimilated into egg-ified zombies. Why do we have the current pressure for egg-ification? The main pressure appears to arise from a strong desire by the python community for a simple installation tool like CPAN that downloads and installs the package of interest *as well as* any dependencies you don't already have installed. Of course, it is the desire for the dependencies to be discovered, downloaded, and installed in a manner that honors the current state of your python environment that creates all of the problems. My participation may be unwanted or unwarranted due to my lack of education/understanding, but I like to live dangerously in the hopes of actually being helpful. With that preamble, here's my attempt at an explicit rationale for a database of installed packages (A.K.A. The New PEP 262): """ Rationale ========= It is often necessary during the course of managing a python installation for an administrator to determine the following things: 1. If the current installation state satisfies the requirements of a new package being considered for installation. 2. If it is safe for the administrator to upgrade or remove a package, and if so, how (e.g. use a system-level package management tool). 3. What files to remove in order to uninstall the package (if a system-level package management tool was not used to install it). 4. If the current installation was modified in-place or custom configured in an way, so that such changes can be noted before upgrading. Furthermore, many administrators want to do as much of this as possible with automated tools, without manually inspecting README files, INSTALL files, or the installed code itself to determine the list of dependencies and installed versions, so there is a desire to be able to make the above determinations programmatically. Current efforts to provide these capabilities without standard library support have resulted in many users being forced to use non-standard package management tools because other users desired these capabilities. This proposal is motivated by a desire to provide the minimum required infrastructure so that both segments of the python community can peacefully coexist without getting in each others way (i.e. the ability to "opt in" to python-based non-system-level package managers). Proposal ======== The proposal is to provide in the standard library the following capabilities: 1. List the installed packages, along with the version and dependency list for each package. 2. Query the ownership of a currently installed package (standard library, system-level package management tool, etc.). 3. List the files installed be specific package. 4. Recall the original message digest for each installed file to determine if the file has been modified. ... Alternatives ============ There are at least three alternatives to providing a database of installed packages that could also potentially enable administrators to accomplish the same ultimate goal of installing new packages and their dependencies without breaking or interfering with the "system" installation. Switchable Environments ----------------------- Eliminate the need to be careful by providing a mechanism for the creation of and installation into what amounts to user-selectable site-packages directories that each "inherit" from what is currently called site-packages (something like an eggless virtualenv with a python comannd-line option to choose the environment). Need to upgrade a package provided by the standard library or your system-level package manager? No Problem! Of course, this won't help you manage your sand-boxed environment, but you can have lots of them for very little cost (it's not a whole new python installation) so just create a new one each time you want to change something. Ubiquitous System Packages -------------------------- Eliminate the need to *not* use your OS's system-level package manager by creating the necessary infrastructure so that python packages can easily be distributed in the major system-level packager formats (RPM, DEB, MSI, MPKG, etc.) for all publicly released python packages. A windows-based developer who releases their pure-python package should be able to create RPMs, DEBs, etc. automatically (using distutils, for instance) without access to a computer running the appropriate OS (python setup.py bdist_all upload). Remote Hosted Installation Database ----------------------------------- Eliminate the requirement that your installation be self-describing by maintaining the information in PyPI necessary for one to figure out what versions of what things are installed in your system (either by recipes or a table of message digest values for the directories) in addition the dependencies and list of installed files for each version of each package thats ever been released. """ Regards, Alex
On Mon, Mar 24, 2008 at 04:57:44PM -0400, Alexander Michael wrote:
With that preamble, here's my attempt at an explicit rationale for a database of installed packages (A.K.A. The New PEP 262):
Nice effort, thanks.
Rationale ========= It is often necessary during the course of managing a python installation for an administrator to determine the following things:
1. If the current installation state satisfies the requirements of a new package being considered for installation. 2. If it is safe for the administrator to upgrade or remove a package, and if so, how (e.g. use a system-level package management tool). 3. What files to remove in order to uninstall the package (if a system-level package management tool was not used to install it). 4. If the current installation was modified in-place or custom configured in an way, so that such changes can be noted before upgrading.
I agree with 1 and 2, but 3 and 4 are optional I think. They are not important for the different tools to interact AIUI. As long as everyone only removes files they own all is fine. By making the database extensible (e.g. X- headers in the RFC822 case) each tool could use their own extensions for recording these specifics as they need them, eliminating required information being spread around the system. Not that I think it would do much harm if they where specified by the PEP, but then I never tried implementing one of these tools so I might not have the best opinion on how much specifying this might unnecessarily restrict implementations. Something that this requirement should probably explicitly describe: does this database only concern modules, i.e. installed in a directory that's on sys.path? My understanding is that it does. However I'm unsure as to how that can be combined with supporting applications that require certain modules. Would every tool need to keep track of the apps it installed separately to make sure that, when it's used to uninstall a module, it doesn't render an app that it installed useless? To tie in another part of this discussion, this could mean that module installation is forced to be a simple list of files while still allowing a Turing complete installer for applications.
Alternatives ============ [...] Ubiquitous System Packages -------------------------- Eliminate the need to *not* use your OS's system-level package manager by creating the necessary infrastructure so that python packages can easily be distributed in the major system-level packager formats (RPM, DEB, MSI, MPKG, etc.) for all publicly released python packages. A windows-based developer who releases their pure-python package should be able to create RPMs, DEBs, etc. automatically (using distutils, for instance) without access to a computer running the appropriate OS (python setup.py bdist_all upload).
This is a bad or even impossible approach. It is very hard or even impossible to create a system package on a different system. It is even harder to keep the releases of *all* python packaging tools in sync with the current policy of a system. It is not a surprise that most of past efforts to create a "bdist_deb" target for setup.py have failed to gain much traction. A system might change policies halfway their release cycle and can't afford the wait for the python world to catch up with that policy change first. This gets even worse when the sytem can't really agree on a policy (which does unfortunately happen). Systems might also have different policies for their different versions (e.g. old-stable, stable and in-development) only making it more impossible for everyone else to try and play catch-up with the system as they would have to supply packages for each supported release. I think it's best to allow systems to create their own packages by giving them a dependable "setup.py install" command --which most systems use in their packaging scripts anyway-- to build on.
Remote Hosted Installation Database ----------------------------------- Eliminate the requirement that your installation be self-describing by maintaining the information in PyPI necessary for one to figure out what versions of what things are installed in your system (either by recipes or a table of message digest values for the directories) in addition the dependencies and list of installed files for each version of each package thats ever been released.
The easiest way to distribute some python stuff is to quickly write a setup.py, even if it's only to share some quick and dirty module with a handful of people. Not everything will want or should be in PyPI. Some machines might also not have net access. So I'm dubious about this option (it also seems technically quite hard). Thanks for writing this up, I think it's very good and useful. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
At 12:33 AM 3/25/2008 +0000, Floris Bruynooghe wrote:
On Mon, Mar 24, 2008 at 04:57:44PM -0400, Alexander Michael wrote:
With that preamble, here's my attempt at an explicit rationale for a database of installed packages (A.K.A. The New PEP 262):
Nice effort, thanks.
Rationale ========= It is often necessary during the course of managing a python installation for an administrator to determine the following things:
1. If the current installation state satisfies the requirements of a new package being considered for installation. 2. If it is safe for the administrator to upgrade or remove a package, and if so, how (e.g. use a system-level package management tool). 3. What files to remove in order to uninstall the package (if a system-level package management tool was not used to install it). 4. If the current installation was modified in-place or custom configured in an way, so that such changes can be noted before upgrading.
I agree with 1 and 2, but 3 and 4 are optional I think. They are not important for the different tools to interact AIUI. As long as everyone only removes files they own all is fine.
Not quite - it's necessary to know which files are owned by a given package or tool in order to do this. If you are installing a package, you need to know who owns any files that you're about to overwrite.
Something that this requirement should probably explicitly describe: does this database only concern modules, i.e. installed in a directory that's on sys.path?
It will probably need to include scripts, data, etc.
My understanding is that it does. However I'm unsure as to how that can be combined with supporting applications that require certain modules. Would every tool need to keep track of the apps it installed separately to make sure that, when it's used to uninstall a module, it doesn't render an app that it installed useless?
To tie in another part of this discussion, this could mean that module installation is forced to be a simple list of files while still allowing a Turing complete installer for applications.
These are currently open issues.
On Mon, Mar 24, 2008 at 10:35:01PM -0400, Phillip J. Eby wrote:
At 12:33 AM 3/25/2008 +0000, Floris Bruynooghe wrote:
1. If the current installation state satisfies the requirements of a new package being considered for installation. 2. If it is safe for the administrator to upgrade or remove a package, and if so, how (e.g. use a system-level package management tool). 3. What files to remove in order to uninstall the package (if a system-level package management tool was not used to install it). 4. If the current installation was modified in-place or custom configured in an way, so that such changes can be noted before upgrading.
I agree with 1 and 2, but 3 and 4 are optional I think. They are not important for the different tools to interact AIUI. As long as everyone only removes files they own all is fine.
Not quite - it's necessary to know which files are owned by a given package or tool in order to do this. If you are installing a package, you need to know who owns any files that you're about to overwrite.
What would the motivation be for requiring to overwrite files created by another tool? That seems rather dangerous to me. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
On Tue, Mar 25, 2008 at 5:16 AM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
On Mon, Mar 24, 2008 at 10:35:01PM -0400, Phillip J. Eby wrote:
At 12:33 AM 3/25/2008 +0000, Floris Bruynooghe wrote:
1. If the current installation state satisfies the requirements of a new package being considered for installation. 2. If it is safe for the administrator to upgrade or remove a package, and if so, how (e.g. use a system-level package management tool). 3. What files to remove in order to uninstall the package (if a system-level package management tool was not used to install it). 4. If the current installation was modified in-place or custom configured in an way, so that such changes can be noted before upgrading.
I agree with 1 and 2, but 3 and 4 are optional I think. They are not important for the different tools to interact AIUI. As long as everyone only removes files they own all is fine.
Not quite - it's necessary to know which files are owned by a given package or tool in order to do this. If you are installing a package, you need to know who owns any files that you're about to overwrite.
What would the motivation be for requiring to overwrite files created by another tool? That seems rather dangerous to me.
As different tools typically shouldn't be allowed to clobber each other, it seems both reasonable and a nice simplification for the proposed db not to require registering of installed files by all package managers/installers if an installer can choose to record its own lists in the db using a simple extension mechanism when desired. Nevertheless, it did seem nice to be able to ask the install db if the current state matches what is on disk by verifying the file lists and checksums against the file system, but this might be taking us too far into the abyss of recreating a system-level package manager.But an additional benefit is that requiring the file lists makes it easier to change tools by ignoring or rewriting the "ownership" information which I would tend to view as purely advisory in nature to begin with (i.e. no enforcement). Of course, this could be accommodated by establishing "standard" but optional auxiliary fields in the db that all python oriented tools would likely use.
At 09:16 AM 3/25/2008 +0000, Floris Bruynooghe wrote:
What would the motivation be for requiring to overwrite files created by another tool? That seems rather dangerous to me.
The point is to NOT overwrite those files.
On Mon, Mar 24, 2008 at 8:33 PM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
Something that this requirement should probably explicitly describe: does this database only concern modules, i.e. installed in a directory that's on sys.path? My understanding is that it does. However I'm unsure as to how that can be combined with supporting applications that require certain modules. Would every tool need to keep track of the apps it installed separately to make sure that, when it's used to uninstall a module, it doesn't render an app that it installed useless?
Has the question of "what does it mean to be installed?" been discussed yet? To me it seems that "python setup.py install" constitutes installing something, but what about "python setup.py install --prefix=~/python-packages" or the various schemes (PYTHONPATH, .pth files) to modify sys.path? A possible way to accommodate the possibly distributed nature of a python installation would be for every directory in sys.path to get its own database so that a first-class environment can be constructed from sys.path manipulations. But what about packages that have simply been placed on sys.path without using distutils? Is it even conceivable that they could be registered? One way to be flexible here is to construct the database by scanning the sys.path for packages and modules with PKG-INFO (ignoring the specifics of how they present it for now), caching in each sys.path directory the database of its own local contents. This way, you could rebuild the database to reflect the current state (including any manually copied packages) when and if desired.
On 22/03/2008, Alexander Michael <lxander.m@gmail.com> wrote:
IOW, the PEP is lacking a rationale.
It seems to me that this discussion is being undermined by not acknowledging the many use cases up front. There is no rationale because there are too many tacit rationales.
Absolutely! It feels like people are trying to design a solution without having written up the problem spec. Maybe that's because there are too many problems for a single solution to work in all cases?
My personal use case is for multi-version packages. I deploy many small scripts (not applications) that use an evolving set of base libraries. I don't want the older scripts to hold me back from pushing forward with the base libraries, so I peg the scripts to their respective major versions as I release them.
My personal use case is for a Windows system with a number of adhoc scripts, which may depend on 3rd party libraries, and a number of "applications" (3rd party or otherwise) written in Python. I want the applications to use a bundled Python and library, via py2exe. There should be no dependency on the "system" Python. Adhoc scripts can depend on libraries installed into the system Python, but generally there will be a "base" set of libraries that I will always have installed and which can be assumed to be present (pywin32, cx_Oracle, py2exe, ...). Scripts can use these without restriction but should be prepared to work with whatever version is present. It should (almost) never be a problem to a script if a library gets upgraded. Additional libraries may be installed in the system Python, for one-off jobs, or for testing and development. I will install and remove these at will, and nothing critical should break ifn I do. There is generally no other Python present on the system. On my development machine, I may have alternative versions installed, and I may have trunk checkouts with a python.exe built, but these will never be used for anything other than specific development tasks, and are treated as "throw-away". It's very rare that any 3rd party library will be installed in these, and special handling when it does happen is entirely acceptable. For the system Python, I need: - a single way to list what's installed (including version) - a single way to uninstall items as needed - a way (or more than one) to install 3rd party software *which ties into the above* The key maintenance task I do on the system Python is to list everything installed, uninstall them all, upgrade the system Python, then reinstall the ones I had installed previously. (Don't bother telling me there are other ways I could do this - that's not the point here, this is how I actually work). The reason setuptools/easy_install breaks this is because it fails the "single way" test. I have to use bdist_wininst for some packages, so setuptools *has* to integrate with that. Also, setuptools doesn't have the list and uninstall features. So that's my problem/requirements. Anything that solves this gets +1 from me. Anything else gets -1 or at best -0. Anything that adds extra features while not solving my problem gets a strong -1, as the extra features will encourage take-up of that solution, exacerbating my current problem. Paul.
On Sat, Mar 22, 2008 at 02:31:34AM +0100, "Martin v. L?wis" wrote:
Tools which will need this data, in order to do their work. Hence, the reason for standardizing the data, instead of the tool(s).
If there was a chance that the infrastructure being developed actually helps these tools, *that* would be a reasonable goal, IMO.
However, I'm extremely skeptical that this can ever succeed to the degree that whoever provides RPMs, .debs, or MSI files will actually use such data, as they will find that the data are incomplete, and they have to redo all of it, anyway.
I've found it extremely useful to have access to dependency information when making Debian packages automagically out of setuptools tarballs. It's not easy or robust to access/use it, but for my simple pure python packages, it's been working wonderfully. -- Brian Sutherland
M.-A. Lemburg wrote:
On 2008-03-21 22:21, Phillip J. Eby wrote:
At 08:06 PM 3/21/2008 +0100, M.-A. Lemburg wrote:
I guess the only way to support all of these variants is to use a filesystem based approach, e.g. by placing a file with a special extension into some dir on sys.path. The "database" logic could then scan sys.path for these files, read the data and provide an interface to it.
All bdist formats would then have to include these files. That's the idea behind the current version of PEP 262, yes, and I think it should be kept.
A separate FILES section also doesn't seem to be necessary - we could just add one or more entries or the format:
CreatesDir abc/ CreatesFile abc/xyz1.py CreatesDir abc/def/ CreatesFile abc/def/xyz2.py CreatesFile abc/def/xyz3.py CreatesFile abc/def/xyz4.ini I actually think the size and hash information is good, in order to be able to tell if you're looking at an original file. I'm not sure how useful the permissions and uid/gid info is. I'm hoping we'll hear from anybody who has a use case for that.
You're heading off in the wrong direction: we should not be trying to rewrite RPM or InnoSetup in Python.
Anything more complicated should be left to tools which are specifically written to manage complex software setups.
We *do* need a way to play nice with all the various platform installers. This would surely be welcomed by the many third parties who now have to package Python for their platforms.
I honestly believe that most people would be happy if we just provide these two things (and no more):
* install a package from a local archive, a URL or PyPI
* uninstall a package in way that doesn't break other installed packages
and whatever the mechanism, avoid making any undercover changes to the Python installation such as adding .pth files, overriding site.py, etc. - these are not needed if the tool keeps to the simple task of installing and uninstalling Python packages.
Examples:
python pypi.py install mypkg-1.0.tgz python pypi.py install http://www.example.com/mypkg-1.0.tgz python pypi.py install mypkg-1.0
python pypi.py uninstall mypkg
If there's a dependency problem, the tool should print the list of other packages it needs. It should not try to install things automagically.
... unless explicitly asked to do so? It seems silly to omit this capability when it's essentially just omitting a recursive call.
If a package needs other modules as well, the package docs can point the user to use e.g.
python pypi.py install mydep1-1.3 mydep2-2.3 mydep4-0.3 mypkg-1.0
instead.
Why would this be better than using --load-dependencies?
Anything more complicated should be left to specialized tools such as RPM, apt, MSI or the other such tools out there - after all the tool should be about Python *package* installation, not application installation.
We *don't* need the tool to:
* support multiple versions of a package (that's just bound to cause problems with pickle, isinstance() etc.)
Another argument for installing apps as separate components with all dependencies. I really don't believe enough consideration has been given as to the essential difference between these two activities.
* provide namespace hacking (is a completely separate issue and can be handled by the packages rather than the install tool)
* support all kinds of funky version numbers (if a package wants to participate in the system, the author better make sure that the version string fits the standard format)
Agreed.
* provide some form of intra-package bus interface (ie. "entry points" as you call them)
* provide support for keeping whole packages in ZIP files (doesn't play well with C extensions, clutters up the sys.path, is read-only, needs special importers, etc. etc. )
It shouldn't require special importers, though. And if the zip file contains compiled code the read-only nature doesn't matter if the zips are version-specific.
* try automatic version matching for required packages
* download things from SourceForge or other sites with special download mechanisms
* scan websites for links
* make coffee, clean the house, send the kids to school :-)
But a package that *would* do this could be immensely popular.
And of course, there are still some issues to be resolved regarding requirements, package name/version stuff, etc. But we can hash those out once we reach a quorum on the Distutils-SIG.
Well, I've probably been killfiled into non-existence on this list by now, but it seems to me that we are in danger of answering the wrong problem yet again. But that's all I have to say on this topic, so you can all heave a sigh a relief and get on with messing it up ;-) regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/
On 22/03/2008, Steve Holden <steve@holdenweb.com> wrote:
Well, I've probably been killfiled into non-existence on this list by now, but it seems to me that we are in danger of answering the wrong problem yet again. But that's all I have to say on this topic, so you can all heave a sigh a relief and get on with messing it up ;-)
You probably have my company in the killfile, but I have a nagging feeling in the same direction. My biggest problem is that I can't express what I believe is the *right* problem, beyond the over-general statement that it seems crucial to me that there should be a single, unified way of managing *all* packages installed in a given Python installation. Whether that's a Python-only solution, or the system packager, doesn't matter. There should be only one way to do it, to reuse a well-known phrase :-) If you know how to state nature of the right problem, that would be useful. All this talk of "playing nicely with the system packager" seems to imply that people are designing a second solution, and trying to manage the interaction, rather than deciding on *one* solution (which by definition has no interaction to worry about). It's reasonable to have multiple solutions for multiple Python installations (system packager for the system python, python packager for a local install, for example) but that's a different matter. Oh, and application installation is (should be) completely different. On Windows, applications should probably be bundled with their own Python interpreter, a la py2exe. On Unix/Linux, I don't know what the standard is, so I'd have to defer to others. Paul.
On Sat, Mar 22, 2008 at 01:46:42PM +0000, Paul Moore wrote:
All this talk of "playing nicely with the system packager" seems to imply that people are designing a second solution, and trying to manage the interaction, rather than deciding on *one* solution (which by definition has no interaction to worry about).
It's reasonable to have multiple solutions for multiple Python installations (system packager for the system python, python packager for a local install, for example) but that's a different matter.
My server runs stable version of Debian, etch currently. It has python installed and most of the modules I need. However not all possible python modules are packaged but I as the local admin might decide a certain module -say pyprocessing- is really required for us and stable enough to use. I now need to install this locally. What I don't want in install an entire new python and hunt *all* modules and then make sure that the correct applications on my server use the correct python.
Oh, and application installation is (should be) completely different. On Windows, applications should probably be bundled with their own Python interpreter, a la py2exe. On Unix/Linux, I don't know what the standard is, so I'd have to defer to others.
An application packaged by the system (Debian in my case) will need to use the system modules. For the same reason we invented shared libraries instead of static libraries. When I have to install an application as local admin then I try to run it with the system python, using as many system modules as possible. If that's not possible (e.g. I need a sandbox) I'll often seriously reconsider installing that application. As a user I have no problems with trying out an app in virualenv or something like it. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
Oh, and application installation is (should be) completely different. On Windows, applications should probably be bundled with their own Python interpreter, a la py2exe. On Unix/Linux, I don't know what the standard is, so I'd have to defer to others.
This I disagree with. I think it's an overall bad thing to have all kinds of applications ship their own copy of Python; see also Aza Raskin's PyCon keynote. On Linux, python typically comes with the system pre-installed; it is not even an option not to have it, except for minimalist installations. So if you write python scripts, you typically expect that #!/usr/bin/env python works; you might put python2.5 there if you don't trust that system one is "good enough". For installing the application, you typically want the choice between a "system installation" (in /usr/bin, or perhaps /usr/local/bin), and a "user installation". As distutils supports both cases, it works fairly well for applications as well. Regards, Martin
On 22/03/2008, "Martin v. Löwis" <martin@v.loewis.de> wrote:
Oh, and application installation is (should be) completely different. On Windows, applications should probably be bundled with their own Python interpreter, a la py2exe. On Unix/Linux, I don't know what the standard is, so I'd have to defer to others.
This I disagree with. I think it's an overall bad thing to have all kinds of applications ship their own copy of Python; see also Aza Raskin's PyCon keynote.
Is this on Windows? It's fairly common practice. Can you give me a pointer to Aza Raskin's keynote? Is it online anywhere? I'd be interested in his point of view. Paul.
Oh, and application installation is (should be) completely different. On Windows, applications should probably be bundled with their own Python interpreter, a la py2exe. On Unix/Linux, I don't know what the standard is, so I'd have to defer to others.
This I disagree with. I think it's an overall bad thing to have all kinds of applications ship their own copy of Python; see also Aza Raskin's PyCon keynote.
Is this on Windows? It's fairly common practice.
Unfortunately so, yes. This can be viewed a burden to the adoption of Python: for a small application, you get this huge download to bundle.
Can you give me a pointer to Aza Raskin's keynote? Is it online anywhere? I'd be interested in his point of view.
Unfortunately no. I was looking for it, but couldn't find it. He mentioned a website with a "call for action", but I couldn't find that, either :-( Regards, Martin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. Löwis wrote:
Oh, and application installation is (should be) completely different. On Windows, applications should probably be bundled with their own Python interpreter, a la py2exe. On Unix/Linux, I don't know what the standard is, so I'd have to defer to others.
This I disagree with. I think it's an overall bad thing to have all kinds of applications ship their own copy of Python; see also Aza Raskin's PyCon keynote.
On Linux, python typically comes with the system pre-installed; it is not even an option not to have it, except for minimalist installations. So if you write python scripts, you typically expect that #!/usr/bin/env python works; you might put python2.5 there if you don't trust that system one is "good enough".
For installing the application, you typically want the choice betaween a "system installation" (in /usr/bin, or perhaps /usr/local/bin), and a "user installation". As distutils supports both cases, it works fairly well for applications as well.
Sharing the system python is hugely problematic on a unix box which actually *uses* python for its own tools: the application is not "safe" from additions / updates / removeals of the packages in /usr/lib/python2.x/site-packages done to support those system tools. The problem gets worse as multiple non-system applications install files there: e.g., the 'twisted' package on Debian boxes depends on an ancient version of 'zope.interface', which can't be used with any currently supported version of Zope. virtualenv makes using the system python on such systems somewhat more tolerable, because each virtualenv can be isolated from the site-packages of the "host" environment (and therefore from the other applications). You still have to live with the choices the system packagers make (UCS4, for instance), but the pain is at least tolerable. For a long-running Python application (as opposed to desktop tools or scripts), installing a custom Python is the "safest" choice: it insulates the application not only from unexpected system-driven site-packages changes, but also allows greater control over other Python configuration choices (the UCS2 / UCS4 knob, etc.). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH6Cqi+gerLs4ltQ4RAlqZAKCxr2lraLSycVsksYAevtf+urALOgCeLzs9 fE2g7IAb+22B+UbSUFPqj4w= =re0h -----END PGP SIGNATURE-----
On Mar 24, 2008, at 3:26 PM, Tres Seaver wrote:
Sharing the system python is hugely problematic on a unix box which actually *uses* python for its own tools: the application is not "safe" from additions / updates / removeals of the packages in /usr/lib/python2.x/site-packages done to support those system tools. The problem gets worse as multiple non-system applications install files there: e.g., the 'twisted' package on Debian boxes depends on an ancient version of 'zope.interface', which can't be used with any currently supported version of Zope.
This is why versioning would be an useful solution. Each package would use the dependent packages that it requires. Foo 1.0 uses Bar 2.3 and Baz 3.2 uses Foo 1.4. An application can use both Foo 1.0 and Baz 3.2 without having to mediate between their requirements. While nobody is really requesting versioning, it seems to be the solution to many problems that plague us. -jeff
On 21/03/2008, Phillip J. Eby <pje@telecommunity.com> wrote:
Questions, comments... volunteers? :)
Sounds good. I won't volunteer as I have neither time nor expertise to contribute much. But I'd like to see this happen, as it sounds like it would address all my issues with setuptools (and just to reiterate, I'm happy if it is implemented *in place of* add/remove programs - I have no vested interest in that facility as such). For uninstall support, you might look at how bdist_wininst installers handle it. In my experience the uninstall is robust, clean, and reliable, and the supporting metadata is pretty simple (just a text file). Paul.
On Fri, Mar 21, 2008 at 09:47:46AM -0400, Phillip J. Eby wrote:
Questions, comments... volunteers? :)
Sounds good, having a PEP626-style install database seems worthwile. Definately if it will enable setuptools to install just like distutils for a "install". Here some notes from my Debian admin/packager point of view (sorry if I miss perspectives of Windows users etc): * I'd like clearly defined module paths for: (a) system, (b) local admin, (c) user. PEP370 seems a step in the right direction. When using "setup.py install" it would be good if the correct prefix would be chosen automatically for (b) and (c). On Windows (a) and (b) would be the same AIUI. * While the installdb can be shared between the sytem and the local admin (/var/lib seems appropriate) the user can't normally write to this. So at separate user installdb might be needed. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
At 10:41 PM 3/21/2008 +0000, Floris Bruynooghe wrote:
On Fri, Mar 21, 2008 at 09:47:46AM -0400, Phillip J. Eby wrote:
Questions, comments... volunteers? :)
Sounds good, having a PEP626-style install database seems worthwile. Definately if it will enable setuptools to install just like distutils for a "install".
Here some notes from my Debian admin/packager point of view (sorry if I miss perspectives of Windows users etc):
* I'd like clearly defined module paths for: (a) system, (b) local admin, (c) user. PEP370 seems a step in the right direction. When using "setup.py install" it would be good if the correct prefix would be chosen automatically for (b) and (c). On Windows (a) and (b) would be the same AIUI.
This seems to be out-of-scope. The "install db" should be on a per-target directory basis, and not get into questions of relative ordering of target directories, or what directory is the default, or any of that. If we go there, we could be talking until Python 4000 is released. :) All I want is the specification for what is recorded within *one* installation directory. IOW, let's please please please keep the PEP 262 replacement orthogonal and separate from any PEP 370 discussions. Any relationship between the two is either an illusion or an insufficiently modular design. :)
* While the installdb can be shared between the sytem and the local admin (/var/lib seems appropriate) the user can't normally write to this. So at separate user installdb might be needed.
Each sys.path directory should have its own installation data; see the "open issues" section of PEP 262.
participants (24)
-
"Martin v. Löwis" -
Alexander Michael -
Brian Sutherland -
Christian Heimes -
Dave Peterson -
Floris Bruynooghe -
Gael Varoquaux -
Jeff Younker -
Joachim König -
Luis Bruno -
M.-A. Lemburg -
Mark Hammond -
Martin Aspeli -
Neal Becker -
Paul Moore -
Phillip J. Eby -
Ronald Oussoren -
skip＠pobox.com -
Stephen J. Turnbull -
Stephen Waterbury -
Steve Holden -
Talin -
Tarek Ziadé -
Tres Seaver