On Thu, Jun 04, 2009 at 11:17:38AM +0200, Tarek Ziad? wrote:

Hello

Here's a status of the current work waiting to be included in Distutils. (target: Python 2.7 and Python 3.2)

I have created PEP 386 for the version comparison work, and gathered in it all the work related to version comparison, I am not an Fedora, Ubuntu, [put your os here] specialist and the PEP is in its early stage, so it needs your feedback of you see false statements, incomplete info or errrors.

STATUS:

- new PEP 386 | waiting for your feedback

- http://svn.python.org/projects/peps/trunk/pep-0386.txt

... < V('1.0.dev456') ... < V('1.0') ... < V('1.0.dev456post623') Looks like a typo or very un-intuitive. It doesn't seem right that a "dev" version sorts after a full release.

- PEP 376 | status : waiting for Phillip complementary feedback (and anyone else of course)

- up-to-date PEP proposal : http://bitbucket.org/tarek/pep376/src/tip/docs/pep-0376

Will there be a way to tell distutils that it is not responsible for a package and should error when trying to uninstall it? Perhaps not installing the RECORD file at all in this case. I can imagine distutils uninstalling files previously installed by dpkg as a shortcut to breaking a machine. Though I'm not sure what will actually happen in practice.

- up-to-date prototype code for pkgutil : http://bitbucket.org/tarek/pep376/src/tip/pkgutil.py

- PEP 345 | status : waiting for PEP 386 validation

Regards Tarek

-- Tarek Ziadé | http://ziade.org _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- Brian Sutherland

2009/6/4 Tarek Ziadé <ziade.tarek@gmail.com>:

...

...

- PEP 376 | status : waiting for Phillip complementary feedback (and anyone else of course)

I can imagine distutils uninstalling files previously installed by dpkg as a shortcut to breaking a machine. Though I'm not sure what will actually happen in practice.

Distutils defines a standard for an EGG-INFO structure, and provides a API for the uninstallation that is more likely to be a reference implementation.

Although, It is already providing an install feature.

If uninstalling a package with Distutils, while it was installed by dpkg breaks,

I can imagine that in the very same system, you can also break it if you *install* packages with Distutils, easy_install, pip because you shortcut dpkg as well.

With Windows, if you install using bdist_wininst and then uninstall using the (currently nonexistent) distutils uninstall, I'd expect that it wouldn't remove the Add/Remove programs support items in the registry, and the Removexxx.exe and xxx-wininst.log files installed by bdist_wininst. The system would then still think that the package was installed, even though it isn't. (And the system uninstall may break - I'm not sure). Given that Windows doesn't do dependency management, this is a relatively minor form of breakage, but on a system that does, having the system metadata wrong could cause incorrect dependencies to be installed at a later date.

I think this is the responsability of such a system to make sure it handles all installation and uninstallation.

It's surely not the responsibility of the OS to handle you "going behind its back" - if I did "rm -rf /usr" on a Unix system, I wouldn't think that it's the responsibility of the system to handle this (other than by breaking!) I'd say that it's distutils' responsibility not to offer to uninstall anything it didn't install. Paul.

On Thu, Jun 04, 2009 at 12:57:12PM +0200, Tarek Ziadé wrote:

On Thu, Jun 4, 2009 at 11:52 AM, Brian Sutherland <brian@vanguardistas.net> wrote:

...

...

- http://svn.python.org/projects/peps/trunk/pep-0386.txt

... < V('1.0.dev456') ... < V('1.0') ... < V('1.0.dev456post623')

Looks like a typo or very un-intuitive. It doesn't seem right that a "dev" version sorts after a full release.

This is a dev version of a post-release version. Which is an edge case submitted by Phillip.

How would you write it ?

1.0.post623dev456 is what feels intuitive to me, here's my version of the last few lines: ... < V('1.0') ... < V('1.0.post456dev623') ... < V('1.0.post456')) It would even feel more consistent if it was 1.0.port456.dev623 Frankly I don't really understand the post-release requirement (and the PEP text doesn't help me out there, no explenation of what it is). If it's like a pre-relase version (alpha, beta, release candidate aka a, b and c) then I imagine it's a preview release of a bugfix release. So if you start with 1.0, then discover bugs so need a 1.0.1 but want to do a preview of that the PEP seems to suggest 1.0.post0, however I would call it 1.0.1a. I guess both will sort correctly according to the PEP so the PEP doesn't care which method you use. Still find it confusing tough. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

On Thu, Jun 4, 2009 at 6:02 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:

On Thu, Jun 4, 2009 at 2:26 PM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:

...

...

This is a dev version of a post-release version. Which is an edge case submitted by Phillip.

How would you write it ?

1.0.post623dev456 is what feels intuitive to me, here's my version of the last few lines:

... < V('1.0') ... < V('1.0.post456dev623') ... < V('1.0.post456'))

It would even feel more consistent if it was 1.0.port456.dev623

Sounds reasonable, I'll try something in the prototype, and submit it here

I don't want to whack a dead horse, but when Philip brought up the "dev release of a post release" I asked for a concrete real-world example. I didn't get a response to that email. Do other people see a need for this? My understanding from the PyCon discussions was that the "postNNN" marker was the equivalent of the "devNNN" marker, but for projects that don't immediately increment the leading version numbers after a release. Can people point to some examples of projects using post-release tags, and that would require the use of a "dev release of a post release"? If there are real examples, those would also help in explaining the motivation for this in the versioning scheme in the PEP. Trent -- Trent Mick trentm@gmail.com

Brian Sutherland wrote:

2009/6/4 Tarek Ziadé <ziade.tarek@gmail.com>:

...

Oups messed up sorry -> resending my answer

On Thu, Jun 4, 2009 at 11:52 AM, Brian Sutherland <brian@vanguardistas.net> wrote:

...

...

- http://svn.python.org/projects/peps/trunk/pep-0386.txt

... < V('1.0.dev456') ... < V('1.0') ... < V('1.0.dev456post623')

Looks like a typo or very un-intuitive. It doesn't seem right that a "dev" version sorts after a full release.

This is a dev version of a post-release version. Which is an edge case submitted by Phillip.

How would you write it ?

... < V('1.0.dev456') ... < V('1.0.dev456post123') # post-release of a dev version ... < V('1.0') ... < V('1.0.post456dev623') # dev version of a post-release ... < V('1.0.post456') # post release

i.e. that a postfix of "dev" means "before" and a postfix of "post" means after.

...

...

...

- PEP 376 | status : waiting for Phillip complementary feedback (and anyone else of course)

I can imagine distutils uninstalling files previously installed by dpkg as a shortcut to breaking a machine. Though I'm not sure what will actually happen in practice.

Distutils defines a standard for an EGG-INFO structure, and provides a API for the uninstallation that is more likely to be a reference implementation.

Although, It is already providing an install feature.

If uninstalling a package with Distutils, while it was installed by dpkg breaks,

I can imagine that in the very same system, you can also break it if you *install* packages with Distutils, easy_install, pip because you shortcut dpkg as well.

Yep. Though I think that nowdays dpkg installs to a different directory than Distutils' default. So users have to specify extra options to break their systems.

That's unfortunately not true: by default (wo any --prefix option), distutils install into /usr. So if a user just uses sudo with a distutils-based package, he is very likely to mess up his system. I think this should be considered as a distutils bug - by default, most source distributions install into /usr/local (autoconf-based projects certainly do).

How should such a system stop distutils (acting as root) from doing an uninstallation?

Ideally, distutils should detect whether it installed the package itself or not. IMHO, uninstall is beyond the scope of distutils; it is very difficult to get right, with so many ways to screw up a system badly. cheers, David

Tarek Ziadé wrote:

The idea of the uninstall API is to provide a reference implementation that can be used in package managers that will rely on the other APIs, rather than a complete system.

For package managers which manage everything (pretty much every native binary installer, be it msi/wininst/rpm/deb), uninstall feature is not needed anyway. They handle all this by themselves once they know what they install.

For instance, Distutils doesn't handle dependencies installation/uninstallation, while PEP 345 introduces them in the metadata.

I think dependencies is the easy part for uninstallation - it is a non-issue IMHO (once a decision on how to deal with dependencies is done :)). The hard problem is how to uninstall one package at the right location (/usr vs $HOME vs something else on Unix, for example), reliably. I am generally very much in favor of delegating many things to 3rd party, using metadata instead of tools, but I think uninstall/registration of packages can only be implemented by python itself reliably, if only because there can't be several systems at the same time. If it is handled by a 3rd party system, I see the following problems: - how to know exactly which package to remove (if I have several of them installed) ? - how to get access to the list of installed packages - how to deal with doc, generated .pyc/.pyo, configuration files, rollback, etc... I don't see how this kind of infrastructure can be provided by a 3rd party, unless it is guaranteed there will be only one. David

Paul

I'd say that it's distutils' responsibility not to offer to uninstall anything it didn't install.

Brian

Yep. Though I think that nowdays dpkg installs to a different directory than Distutils' default. So users have to specify extra options to break their systems.

But how does those third-party install the projects ? Don't they use Distutils' install command under the hood ? If so, the new PEP 376 APIs are ignoring previous versions of egg-infos and work only with the ones that complies with the new standard (it's a quick control right now, look at the "is_egg_info" API in http://bitbucket.org/tarek/pep376/src/tip/pkgutil.py) So, in any case the uninstall command will not work with project that are filtered out by is_egg_info(). So maybe we could add a "INSTALLER" file with a unique md5 key provided by the project that installed the package, and ask for the key when calling this API ? If not provided, it would use Distutils's md5 key Tarek

On Thu, Jun 4, 2009 at 3:23 PM, Brian Sutherland <brian@vanguardistas.net> wrote:

So, with setuptools I was running this while building a package:

   python2.X setup,py install --single-version-externally-managed --root=debian/$(package) --install-data=usr/lib/$(package)

So, now I would need to run this:

   python2.X setup,py install --root=debian/$(package) --install-data=usr/lib/$(package) --installer 7e2fccc88b1f74aeee3d37340e8183ed

And that would cause distutils to refuse to uninstall that package by default?

something like that, but it would be even simpler : the installer option would be automatically added by setuptools when the 'install' command is called, with setuptools md5 marker. (since setuptools has its own install command) That said, maybe setuptools would decide to rely on distutils for uninstallation if its installation turns to be compatible with distutils'one , and not provide its own marker when you manually call 'install' like you showed. And easy_install would have its own marker maybe, if the way it installs stuff slighlty differs

2009/6/4 Tarek Ziadé <ziade.tarek@gmail.com>:

On Thu, Jun 4, 2009 at 3:23 PM, Brian Sutherland <brian@vanguardistas.net> wrote:

...

So, with setuptools I was running this while building a package:

   python2.X setup,py install --single-version-externally-managed --root=debian/$(package) --install-data=usr/lib/$(package)

So, now I would need to run this:

   python2.X setup,py install --root=debian/$(package) --install-data=usr/lib/$(package) --installer 7e2fccc88b1f74aeee3d37340e8183ed

And that would cause distutils to refuse to uninstall that package by default?

something like that,

but it would be even simpler :  the installer option would be automatically added by setuptools when the 'install' command is called, with setuptools md5 marker. (since setuptools has its own install command)

That said, maybe setuptools would decide to rely on distutils for uninstallation if its installation turns to be compatible with distutils'one , and not provide its own marker when you manually call 'install' like you showed.

Ok, so then in this situation the --single-version-externally-managed setuptools option could tell distutils to use a non-distutils marker while the default distutils marker is used when this option is not present.

And easy_install would have its own marker maybe, if the way it installs stuff slighlty differs

ok, seems reasonable:) Perhaps the --installer option take a string (like "dpkg") and convert that to an md5 sum?

2009/6/4 Tarek Ziadé <ziade.tarek@gmail.com>:

On Thu, Jun 4, 2009 at 3:41 PM, Brian Sutherland <brian@vanguardistas.net> wrote:

...

...

And easy_install would have its own marker maybe, if the way it installs stuff slighlty differs

ok, seems reasonable:)

I'll work on that. I guess it's time to code the uninstall prototype,

...

Perhaps the --installer option take a string (like "dpkg") and convert that to an md5 sum?

Mmm, or maybe drop the md5 completely, and go for a human readable name.

(distutils, dpkg, etc..)

The api would raise an error if the name is not in [A-Za-z0-9_\-]

I just remembered that setuptools implicitly sets --single-version-externally-managed when the --root option is set. Unfortunately my packaging-fu is not good enough to tell if it's an option in this case as well, i.e. to set --installer to "external". http://peak.telecommunity.com/DevCenter/setuptools#install-run-easy-install-...

On Thu, Jun 4, 2009 at 3:59 PM, Paul Moore <p.f.moore@gmail.com> wrote:

Also, you need to remember that bdist_wininst is not a "third-party" tool. It's part of distutils (although it's probably written in such a way that it could easily be extracted as a 3rd party addin - I honestly don't know).

I think that would be great to externalize such command to decouple its release cycles from Python (As mentioned during the Langage summit) The only requirement would be to find someone that would lead its development, If I find a windows developer that wants to handle this <hint> <hint> , we could probably create a separate project

I'd seriously recommend looking at bdist_wininst as an example - it's distributed with Python, so the source is easy to find, but it installs extra system-level metadata above and beyond what setup.py install does, so you can see what sorts of issues you have to deal with. (Some of the Unix-level packagers like bdist_rpm may be equally illuminating, but I've no experience with those).

I will, thanks for the tip. I guess bdist_msi should be also our target for this,

On Thu, 4 Jun 2009 16:15:11 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:

I think that would be great to externalize such command to decouple its release cycles from Python (As mentioned during the Langage summit)

The only requirement would be to find someone that would lead its development,

If I find a windows developer that wants to handle this <hint> <hint> , we could probably create a separate project

It's a lot of work for anybody doing any seperate project. What are you expectations for this? David

On Sun, Jun 7, 2009 at 4:36 PM, Ronald Oussoren<ronaldoussoren@mac.com> wrote:

...

So maybe we could add a "INSTALLER" file with a unique md5 key provided by the project that installed the package, and ask for the key when calling this API ? If not provided, it would use Distutils's md5 key

Why should the the INSTALLER file contain a md5 value? I'd use either the name of the packager (dpkg,rpm,...), or even that plus the name of the external package:

       rpm,py26-zlib

This would make it possible for a PEP376-based uninstaller to give more meaningful error messages when you try to uninstall a project that was installed by a system package. That is, it could say "this project was installed using the 'py26-zlib' system-package, use 'rpm' to uninstall" rather than "go away I didn't install this".

Yes that's what I said later, and I changed the PEP accordingly,

BTW. I'd say that the INSTALLER should default to "distutils", that is: don't install the INSTALLER file unless someone explicitly asks for it by specifying a different value for the installer. A tool like bdist_rpm could of course automaticly set that different value.

That's the case besides the fact that the INSTALLER file is created even with the default distutils value. I don't see any good reason not to create it nevertheless.