[Distutils] formencode as .egg in Debian ??

[Ian Bicking]

Josselin Mouette wrote:
> Le dimanche 27 novembre 2005 à 16:26 -0600, Ian Bicking a écrit :
> 
>>>>No one is forcing Debian to package any of these.  
>>>
>>>Of course you are forcing Debian to package these. As long as your
>>>projects have enough users, someone will want to build a Debian package.
>>>The whole point of this thread is to make this package not suck.
>>
>>easy_install works, right now, for these packages.  There are some
>>outstanding issues, and those issues can probably be resolved in
>>easy_install, without any intervention by Debian.  Maybe the simplest
>>resolution is adding an option like:
>>
>>  easy_install --already-provided=1.2.6 ElementTree
> 
> 
> We're not going to tell our users to use easy_install for some kind of
> packages and some other thing for another package, and so on. They
> expect to be able to install anything, just by typing "apt-get install
> foo".

That's okay; I'll just tell my users to use it when Debian packages are 
not available (which is the case for the majority of Python libraries, 
and probably always will be unless libraries are automatically 
packaged).  easy_install provides a much better and safer experience 
than downloading tarballs and using the normal "setup.py install" 
routine; it is an improvement on the status quo.  The other option would 
be to allow people to easily create debs, in the same way they can 
easily create RPMs right now; but I get the strong impression you don't 
think that's a good idea.

>>>I'm happy you see stable releases as "counterproductive". However many
>>>of our users happen to like them, as they don't like to upgrade their
>>>software every other day. Again, eggs are a useful tool only for
>>>developers, not for regular users.
>>
>>These are developer tools, regular users *are* developers.
>>
>>If someone reports a bug, and I fix it, and they can't get access to
>>that because they can't install a new version of the package, then that
>>is counterproductive.  It discourages feedback.
> 
> 
> Stable releases are not meant for developers like you, but there are
> developers who want a stable development platform, even if it has some
> bugs. No bug fixing also means no regressions. The fact you don't
> understand those people's needs - and forget them when designing your
> tools - doesn't make all stable releases "counterproductive".

I'm speaking in terms of the open source ecosystem -- gaining 
contributors is far more valuable than gaining users.  So from my 
perspective -- which isn't out of line from Debian's, I think -- cutting 
off feedback is damaging.

> Developers who want a bleeding edge development platform should be using
> Debian testing or unstable, where the new versions become available as
> soon as possible.

I don't think those really provide the same experience.  If you are 
really serious about the bleeding edge, you should be using a checkout 
from the upstream repository.  Which, incidentally, setuptools and 
easy_install handle nicely.  But again, I say that from the perspective 
of someone who wants to gain contributors, not just users.

>>*Or*, we develop various ad hoc techniques to allow
>>people to install other versions of packages *in spite* of the Debian
>>packages.
> 
> 
> If your packages can't be installed in /usr/local, where they take
> precedence over the version in /usr, your packaging system is flawed.

They can be, with no problems; easy_install uses the standard 
distutils.cfg configuration, which can indicate prefix=/usr/local.  Even 
without it, easy_install/setuptools is much less likely to clobber 
Debian installed packages, compared to normal distutils.

>>I just don't think it is an option to say that only one version of a
>>package can be installed.  In /usr/lib/pythonX.Y/site-packages, sure,
>>and there can potentially be other restrictions.  But you are limiting
>>your users too much if there can never be multiple versions installed.
> 
> 
> Stable users will want to keep the same version, and allowing them to
> install several versions provides a way to easily break their systems.
> It also cuts them off the security fixes.

New versions don't overwrite old versions.  I believe that if you have a 
library in /usr/lib, and install a new version to /usr/local/lib, that 
the new version won't ever be accessible until you specifically 
pkg_resources.require() it (or implicitly require it due to 
dependencies).  *However*, to make that work nicely an easy_install'ed 
package should see metadata about Debian installed packages; otherwise 
duplicate versions of libraries have to be installed, because it's not 
reasonable to discover at runtime what Debian packages are installed (if 
only because the Debian package database is much too slow).  Though if 
you really wanted, setuptools/easy_install could be patched to read the 
debian package database on installation, maybe even writing a .egg-info 
directory then (putting it in /usr/local/lib, but it can refer back to a 
library that was installed in /usr/lib).  That mostly depends on the 
enthusiasm of the setuptools packager.

Anyway, in spite of all the disagreement, I think we actually have 
reached a sort of conclusion...

For the packages in question (ultimately leading to packaging 
TurboGears) there's only one current Debian package of consequence: 
ElementTree.  I think it's fine to remove that package in particular 
from the egg dependencies.  So the timescale for changes to Debian 
aren't that much of an issue because these are new packages.  Most of 
the libraries are developed with setuptools upstream... I think all of 
them, in fact (setuptools itself, TurboGears, Kid, SQLObject, 
FormEncode, probably nose as well).

There are some questions about zip files -- simplest answer: don't 
install them as zips.  Some of them can't be installed that way anyway.

There are some questions about the performance implications of 
installing them in subdirectories of site-packages, and extending 
sys.path for each installed package.  You can revert to normal 
installation with an .egg-info directory, and Phillip will (or already 
has?) added support for putting the version number in those directory 
names, so that will all work fine.  I believe if you require a version 
of a package from /usr/local/lib, it will do the Right Thing and change 
sys.path appropriately.

As I think about it, not putting Debian-packaged libraries entirely in 
subdirectories (the typical egg installation) seems best -- keeping 
libraries in subdirectories makes package management easier (by 
easy_install, nest, or whatever), but in this case we definitely *don't* 
want those tools to try to manage those packages.  Maybe we could even 
add a flag to indicate to setuptools/easy_install when a library is 
managed by some external tool?  I think that would add some important 
safety.  I would suggest something like a 
Package.egg-info/managed_by.txt file, whose existance indicates the 
package is externally managed, and whose content explains what manages 
it (so setuptools could provide a nice error message that directs the 
user to apt-get on Debian, or some other native tool on other platforms).

A distutils.cfg file (bug 338572) will allow people to use easy_install 
on Debian systems, and still follow Debian policy.  But I assume that 
such a change will take some time to propogate even if it is applied 
immediately to sid.  But even without it, easy_install won't overwrite 
any Debian package files without that configuration file.  I think 
easy_install also gives some scary-looking warnings in that case, but 
that might be possible to handle in documentation, especially if we all 
agree on what steps users should be following.


-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org