Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
At 10:04 AM 10/7/2008 +0100, Paul Moore wrote:
2008/10/7 Phillip J. Eby <pje@telecommunity.com>:
In the case of BUILDS, I propose to do the same: define a standard whose cost/benefit ratios are ideally balanced for each participant. This does not, by the way, mean that everybody ends up with the same cost/benefit ratio; it simply means that the cost/benefit ratios are best for those people whose participation is most required for the standard to be widely adopted.
You can see that this is also what I did in the design of easy_install and setuptools, except that in that effort I only considered developers and users, not system packagers.
I'd argue (you may differ) that the most significant area where you missed the mark on user benefits with easy_install and setuptools is the lack of easy *uninstall* and easy *list* options.
Well, I'd certainly agree that those features are desirable. But I didn't "miss the mark", in that those features were not part of my mark. ;-) The goal was to get widespread adoption by developers, and thus the primary target audience was developers. Any features that attracted users were there only insofar as the benefit to users would specifically drive adoption by developers. And that means that features that only matter *after* you have things installed (i.e., easy uninstall and list) were of relatively little importance in the initial feature set. It was only necessary to provide the *possibility* of uninstall and list features, and then allow others to scratch the resulting itches. Now, for BUILDS, the situation is different. Just as setuptools' competitor was distutils, BUILDS' competitor is setuptools. So, BUILDS has to offer developers a switching benefit. And, unlike the situation before, now developers are much more likely to have a lot of packages they'd like to list or uninstall, so such features have considerably more value now, than they did then, which makes them worth putting some effort into. Also, since that switching benefit is largely going to come from having better management tools, the management tools need to be easy to build (so that people will build them). However, since system packagers are experiencing pain right now wrt Python packages, there is at least some benefit to their participating in the process. Back when setuptools was developed, people from the distros weren't hanging out here in the distutils-sig... or at least, they never answered any of my calls for interested parties.
Most of the issues I hear from users about setuptools (filtered by my prejudices, admittedly) is that there's no management options (which brings in the system packagers, and their concerns).
Can I suggest that this be included somehow in the new spec, so that metadata is available to make wtiting uninstallers and listers as easy as writing installers?
Since the single most important part of BUILDS (and the part that should be done first) is an installation manifest for packaging tools to know what files are of what kind and where, it should be easy for uninstallers to work. Regarding listers, there are already a number of them available in the Cheeseshop, and if you're working with Python 2.5+ (and not using an older distro that deletes .egg-info files), they should also work with distutils-installed packages. The idea behind the installation manifest and metadata (the "IL" and "DS" of BUILDS) is that it should be possible for anyone to write their own management tools for specific scenarios, and not have to rely on a bunch of implementation-specific and not-very-well-documented stuff. (It should also be possible, given the "IL" part, to make a distutils and/or setuptools command that generates this data from a sufficiently well-behaved setup.py.)
2008/10/7 Phillip J. Eby <pje@telecommunity.com>:
At 10:04 AM 10/7/2008 +0100, Paul Moore wrote:
2008/10/7 Phillip J. Eby <pje@telecommunity.com>:
You can see that this is also what I did in the design of easy_install and setuptools, except that in that effort I only considered developers and users, not system packagers.
I'd argue (you may differ) that the most significant area where you missed the mark on user benefits with easy_install and setuptools is the lack of easy *uninstall* and easy *list* options.
Well, I'd certainly agree that those features are desirable. But I didn't "miss the mark", in that those features were not part of my mark. ;-) The goal was to get widespread adoption by developers, and thus the primary target audience was developers. Any features that attracted users were there only insofar as the benefit to users would specifically drive adoption by developers.
OK, I was assuming that when you said "I only considered developers and users", you were implying that you were aiming to provide for users benefits (and hence, the user benefit of package management - uninstall and list - was important). As you say, though, your key aim was to drive adoption, so making *installation* easy for users is important, but making *uninstall* easy isn't, as it's not something related to adoption - rather the opposite :-) Of course, as a user, my perspectives are different :-) And as a user who is increasingly resistant to setuptools-enabled packages precisely because they don't address user-level issues, I suspect that catering for user level requirements (in addition to system packagers', and the already covered developers) is more relevant this time round. My feeling, by the way, is that "system packagers" are the more relevant group on Linux/Unix (where most users install Python modules via system packages, or else they are developers) whereas on Windows, users are the more relevant group (as there isn't really a significant system packager role in the first place). Whether this, coupled with the notoriously difficult task of getting Windows and Linux/Unix users to understand each other's perspectives, is an important fact, only time will tell. Thanks for taking the time to clarify for me. Paul.
Paul Moore wrote:
My feeling, by the way, is that "system packagers" are the more relevant group on Linux/Unix (where most users install Python modules via system packages, or else they are developers)
I think this is part of why I don't understand the system packager perspective. Developers shouldn't use system packages, it just doesn't make any sense to have that intermediation. Users don't use Python modules, they use applications. Users only care that their applications work, that they can install applications without unnecessary conflicts, that the applications don't break based on unintentional environment changes (e.g., the value of PYTHONPATH). Packagers seem to care a great deal about having applications share libraries on the packaging level, but this is for their own accounting, there's no reason for users to care (except for the too-small-to-matter issue of disk space). Also, packagers seem to jump the gun on this library sharing, as they are concerned about libraries when one (or often zero!) applications depend on the library. Some widely used libraries seem reasonable, but for every widely used library there are a dozen or more niche libraries. Users also don't care about /usr/share or /usr/lib -- the only thing *I* ever care about is /usr/share/doc, /usr/bin, /etc, and maybe a man page. -- Ian Bicking : ianb@colorstudy.com : http://blog.ianbicking.org
2008/10/10 Ian Bicking <ianb@colorstudy.com>:
Paul Moore wrote:
My feeling, by the way, is that "system packagers" are the more relevant group on Linux/Unix (where most users install Python modules via system packages, or else they are developers)
I think this is part of why I don't understand the system packager perspective. Developers shouldn't use system packages, it just doesn't make any sense to have that intermediation. Users don't use Python modules, they use applications. Users only care that their applications work, that they can install applications without unnecessary conflicts, that the applications don't break based on unintentional environment changes (e.g., the value of PYTHONPATH).
From my point of view, as a Windows developer, I want to use "system
Interesting viewpoint (from my Windows developer/user POV :-)) packages" (bdist_wininst installers) because then I get the benefits of having a listing of what I have installed, being able to quickly, easily and cleanly install and uninstall packages to "try them out" etc. I have the same environment for my one-off scripts as for my development environment, which again I feel is good as I'm using a consistent, familiar, toolset. As a user, I want my applications packaged with py2exe, and independent of the "system Python" - which for me is the development environment, and therefore not stable enough to rely on for apps! And as a developer, py2exe cleanly picks out those bits of my development environment which are needed in the delivered application, so I don't have to worry about installing "too much" before I package my application up. This probably reflects the fact that per-user, or any other sort of private, installations of Python, are the exception rather than the norm on Windows, whereas on Unix, building a custom Python to do your development with (either from scratch, or with things like virtualenv) is easy and sensible. I guess the more we see how "the other half" lives (without the flamewars :-)), the better we'll be able to compromise or provide options to help everyone. (Skipped the comments on system packaging, as you've confirmed what I thought, which is that it's very much a Unix phenomenon, and I'm not qualified to have an opinion...) Paul.
On Fri, Oct 10, 2008 at 02:34:30PM -0400, Ian Bicking wrote:
Paul Moore wrote:
My feeling, by the way, is that "system packagers" are the more relevant group on Linux/Unix (where most users install Python modules via system packages, or else they are developers)
I think this is part of why I don't understand the system packager perspective. Developers shouldn't use system packages, it just doesn't make any sense to have that intermediation.
This only applies to packages you're developing, not those you're using to develop your app/library/whatever. Why I, as a developer, prefer system packages: * easier installation * easier upgrades * I don't have to keep track of security updates myself In fact those are the same reasons why I prefer system packages as a user. There are downsides (e.g. you usually don't get the latest version, and sometimes distributions apply broken patches). Marius Gedminas -- You'll find creativity working hand in hand with engineering. It will feel strange and you might feel like things are out of control. Relax - they are. -- Richard Gabriel on software http://www.dreamsongs.net/LessonsFromNothing.html
On Fri, Oct 10, 2008 at 10:56:02PM +0300, Marius Gedminas wrote:
On Fri, Oct 10, 2008 at 02:34:30PM -0400, Ian Bicking wrote:
Paul Moore wrote:
My feeling, by the way, is that "system packagers" are the more relevant group on Linux/Unix (where most users install Python modules via system packages, or else they are developers)
I think this is part of why I don't understand the system packager perspective. Developers shouldn't use system packages, it just doesn't make any sense to have that intermediation.
This only applies to packages you're developing, not those you're using to develop your app/library/whatever.
Why I, as a developer, prefer system packages:
* easier installation * easier upgrades * I don't have to keep track of security updates myself
I'd like to add to this: making myself develop against system packages will make sure my work will be easy to package up and hence deploy. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
Ian Bicking wrote:
I think this is part of why I don't understand the system packager perspective. Developers shouldn't use system packages, it just doesn't make any sense to have that intermediation.
I can't see why you would think that. Of course it makes sense to have that intermediation, for packages which are not directly relevant to you. I am more than happy to use packages myself for almost everything but the things I am directly working on (and maybe the layer just below).
Users don't use Python modules, they use applications.
What is an application ? I mean, people who use numpy/scipy, they use a module. People who are scons users, they use a software which is just one module + some scripts. The application / library difference is blurry in python.
Users only care that their applications work, that they can install applications without unnecessary conflicts, that the applications don't break based on unintentional environment changes (e.g., the value of PYTHONPATH).
Yes, of course. But this is obviously linked to deployment issues. Then, the argument is that when deploying things, multiple / concurrent modules is a headache. Api versioning and stability is not a linux packagers crazy idea
Packagers seem to care a great deal about having applications share libraries on the packaging level, but this is for their own accounting, there's no reason for users to care (except for the too-small-to-matter issue of disk space).
Did you read carefully what was written by the packagers ? They almost never mention disk space (which does matter on some systems BTW; not everybody uses a desktop or a server). They mention security, they mention maintanability. Both those are made much more complicated when you enable multiple / concurrent versions.
Also, packagers seem to jump the gun on this library sharing, as they are concerned about libraries when one (or often zero!) applications depend on the library. Some widely used libraries seem reasonable, but for every widely used library there are a dozen or more niche libraries.
Those niche libraries are not always packaged. If a library is used by only one package, it is quite likely that it is included in the package itself. cheers, David
On Fri, Oct 10, 2008 at 02:34:30PM -0400, Ian Bicking wrote:
Developers shouldn't use system packages, it just doesn't make any sense to have that intermediation.
I don't agree. I work as a developer/scientist in a lab. I develop algorithms that sometimes will never be used on another box than mine (that's when they are bad). We happen to be standardised on a Linux distro that is a bit old, and with little packages. I can tell you it really annoys me to have to install manually a package each time I want to use it. The pure-Python ones, with no other dependencies are not hard, easy-install does the trick, but these actually represent a small fraction of the important ones (wxpython, numpy, pytables, matplotlib, elementtree, pyobjects, cython, ets, pyvtk). So I lose a lot of time installing packages on my box. But the real nightmare begins when I want to share the algorithms and applications I developed. Across our institute we have NFS-shared drives on which we have to compile all the dependencies, for all three platforms we support. And that's only for distributing software in the institute, when we want to distribute outside we end up shipping big binary blobs with everything, from our own version of Python, to numpy, wxPython, ATLAS (that's bad). If only we could rely on package managers to provide us with base packages (and API-stability of these packages to be sure they work).
Users don't use Python modules, they use applications. Users only care that their applications work, that they can install applications without unnecessary conflicts, that the applications don't break based on unintentional environment changes (e.g., the value of PYTHONPATH).
That is true, as long as you ship "the world", ie everything you'll ever need for your extensible application, including the c libraries that go behind. This means putting a huge burden on your development team, and if you think about out, this is exactly the work of a distribution. So you are duplicating this effort. I addition, one of the great things about Python, is that it is easy to read and to code. Advanced users will want to write plugins, extend there programs, and programs should be written for this: "Hey, I have this cool application that does brain segmentation, and visualization of the tissues, and I have this cool other app that has an algorithm to identify tumorous tissues, I want to plug them together, and because the APIs are well-written, it should be trivial... Darn, the two apps have confined Pythons, and they can't import from each-others. No big deal, I'll unbundle the whole thing... Darn, one has been build with UCS-2, the other one with UCS-4, and there C libraries are incompatible... So now the easy task has become a long effort of building the two apps, and I can't easily share my work." I can hardly believe this happens only in the scientific computing world. I have fought (and I still fighting) to deploy a Django-based application. It didn't get along with the server's apache version. Gaël
Ian Bicking wrote:
Paul Moore wrote:
My feeling, by the way, is that "system packagers" are the more relevant group on Linux/Unix (where most users install Python modules via system packages, or else they are developers)
I think this is part of why I don't understand the system packager perspective. Developers shouldn't use system packages, it just doesn't make any sense to have that intermediation. Users don't use Python modules, they use applications. Users only care that their applications work, that they can install applications without unnecessary conflicts, that the applications don't break based on unintentional environment changes (e.g., the value of PYTHONPATH).
Packagers seem to care a great deal about having applications share libraries on the packaging level, but this is for their own accounting, there's no reason for users to care (except for the too-small-to-matter issue of disk space). Also, packagers seem to jump the gun on this library sharing, as they are concerned about libraries when one (or often zero!) applications depend on the library. Some widely used libraries seem reasonable, but for every widely used library there are a dozen or more niche libraries.
There is a great deal of incentive for everything to use system libraries. take formencode-1.0 vs formencode-1.0.1 for instance. If every TurboGears application and whatnot that used formencode were using their own copy of formencode-1.0 and then the discovery that formencode-1.0 wasn't doing chained validators correctly came out, the end users have to go about finding every copy of formencode and updating it. If there is only a single formencode library on the system and all the TurboGears apps are using it, that's a single package that needs to be updated.
Users also don't care about /usr/share or /usr/lib -- the only thing *I* ever care about is /usr/share/doc, /usr/bin, /etc, and maybe a man page.
They do when user == system administrator. Ask someone who manages Linux systems at a large University if they care about the FHS and the answer will be yes. The various distinctions in the FHS between writable vs nonwritable, architecture dependent vs architecture independent, etc allow system administrators to maintain a large number of systems with less pain. -Toshio
participants (8)
-
David Cournapeau -
Floris Bruynooghe -
Gael Varoquaux -
Ian Bicking -
Marius Gedminas -
Paul Moore -
Phillip J. Eby -
Toshio Kuratomi